NDI output causing dropped frames and audio glitches
Windows 10 21H1
Wirecast 14.3.1
Intel i7, 1080ti graphics
6x NDI input, Magewell pro capture quad (only 1 input in use)
Canvas 1080p30, Encoder for stream and local MP4: NVENC 1080p30 7.5Mbps keyframe every 30 frames, VBR
Streamed a 3 hour show today and around hour 2 noticed that the audio output to both the stream and local record developed a glitch and at the same time the dropped frames started climbing by roughly 4-5 per second. Stopped and restarted the stream, issue persisted. Stopped and restarted the record, issue persisted. Stopped and restarted NDI output, issue resolved temporarily. In 5 or 10 minutes it would come back and I would have to stop/start NDI output. At the time I had 5 NDI inputs from Zoom Rooms active in one shot, but no changes had been made for about 20 minutes. For the rest of the day the issue would recur even if the NDI inputs were inactive (no video being sent to them, and not in the active shot or preview)
At about 30 seconds into the attached recording I stopped NDI output and restarted it. The NDI sources were in the document but inactive when the recording was made.
-
Bump. Same issue on a brand new (different) windows machine.
AMD ThreadRipper pro 3995WX, Gigabyte WRX80 motherboard, GTX 1660 Super Graphics, 256MB RAM, 1TB SSD
Windows 21H1, wirecast 14.3.1
I was not streaming or outputting anything aside from the NDI output this time.
-
CraigS Any insight?
-
Billy Davidson Might be an NDI bandwidth issue.
Do double-check CPU use and make sure it's no higher than 60%.
Test using a single source and see if the issue happens.
If that's good then the next steps are to look at the number of sources and their impact.Also, Timestamps might be an issue but first, test a single source and see if it happens. If it does, change Timestamps
Another single-source test is doing an ISO recording and comparing it to a local recording since ISO is preprocessing and Local Recording is post-processing (and is also impacted by the Wirecast encoder).
-
CraigS
CPU usage was below 40%
Single source will do it too. Single local media file as source.
I've been setting all timestamps to generated by wirecsast.
I'll try and set up a single source ISO test later.
-
-
Billy Davidson said:
I'll try and set up a single source ISO test later.Thanks. That would be an important test.
That will tell us if it's incoming.
Billy Davidson said:
Single source will do it too. Single local media file as source.To be sure. Test in a new document so there are no other NDI sources in the Wirecast Asset Manager to make sure Wirecast isn't "polling" the other sources impacting network bandwidth.
-
CraigS New document, 1080p30 canvas and record via default nvenc 1080p30 4Mbps encoder, a single video capture from Black Magic capture card with audio embedded. NDI output active, multiview output on to external monitor, but no other outputs. Within 20 minutes it started dropping frames. Started local record and ISO once I saw frames dropping, record had the same glitches, video seemed choppy as well, ISO was clean.
-
-
Billy Davidson said:
video seemed choppy as well, ISO was clean.ISO was clean means input is likely good.
Next consider the encoder settings.
Please test a local recording (Recording to Disk) only using 1080p30 x264 preset (don't change any of the settings).-
CraigS My mistake, that record I just did was on the default x264 1080p30 setting
-
Billy Davidson
Confirming the tech points so I can discuss them with the developers.- Single 1080p NDI source in a new document.
- ISO Recording is good.
- Local recording using 1080p30 x264 preset exhibits the issue.
- CPU% is well under 60%
- Restarting NDI output fixes the issue temporarily.
- Timestamp set Wirecast (developed will ask to set to Source to see if it behaves differently so please test that now).
- When you testing Local Recording did you have NDI Out enabled? If so, please test with NDI Out disabled to confirm it is the source of the problem.
- Please test with no NDI sources in (new document) and only one Magewell source and NDI Out enabled.
If I've made a mistake with any of these please correct it.
Developers will want to set up the simplest scenario matching your workflow that can reproduce the issue. If they can't they will have questions on variables so simplicity can expedite a solution.
-
CraigS
The source on todays test is 1080i@59.97, this has been observed in other documents with local media at 1080p30 and other capture sources at 1080p30 and p60 as well as NDI sources at various framerates and a Scarlett USB audio interface.
CPU is well under 10% on this new machine
I will test with timestamps set to wirecast.
when NDI out is disabled it instantly fixes the issue. You can hear it in the original video I uploaded above at roughly 30 seconds in. The issue does not happen when NDI output is disabled.
-
Billy Davidson Thanks for confirming that NDI Out is specific to the cause impacting the encoder whether streaming or recording.
-
CraigS Confirmed still happens with timestamps set to source.
There are no NDI sources in the document I'm using today.
-
Billy Davidson Thanks for that.
-
Billy Davidson Investigating WIRE-19409
Please go to Wirecast Help > Send Support Information and select the Save Report button and please attach the report here to a forum post. If you can, do this immediately after you replicate the issue. -
CraigS This report was within seconds of a period of dropped frames. On the previous machine they tended to keep going continually once they started dropping, on the new one they tend to start for brief periods and stop at random.
-
Billy Davidson Thanks for that. That may give us some important clues.
-
-
Experienced this on a new machine today. Mac M1 on MacOS 11.6, Wirecast 14.3.2.
This machine hasn't had this issue before. I realized I accidentally had 2 different encoders today, so the CPU use was hovering around 50-60% when its usually 30%. But once the dropped frames issue started the CPU was pegged according to wirecast. Here's a log from this morning, sorry its a while after the issue, but Wirecast wasn't restarted. The stream and record ran fine for ~25 minutes before acting up.
-
Billy Davidson said:
The stream and record ran fine for ~25 minutes before acting up.Please clarify if this was an NDI In or NDI Out issue.
If NDI what were the sources?
Wirecast can certainly drop frames if the CPU goes over 80% so this may not be the same issue.
-
CraigS NDI out. Yes, I know CPU usage can be an issue, but the fact that it spiked and began the same issue on different hardware seemed like a clue.
-
Billy Davidson Can you replicate the issue keeping CPU to 60% or lower?
-
CraigS This is the only time I've seen it happen on a mac. We've only run it for 1.5 hour streams every sunday for the past 6 weeks or so, so not a huge sample size. I'll try to replicate it, but when the issue was not occuring yesterday my usage was bouncing around 60% with the misconfigured encoder.
To add to the above, this is the only Mac based system I use that has NDI out as part of its regular workflow. I have 3 other older intel macbook pro machines at my usual job, but we aren't usually sending NDI from them.
-
-
Billy Davidson said:
Yes, I know CPU usage can be an issue, but the fact that it spiked and began the same issue on different hardware seemed like a clue.We'll investigate this on Mac as well.
We have a Windows fix confirmed for a future release but now investigating Mac as well given your report. -
Billy Davidson said:
I'll try to replicate it, but when the issue was not occuring yesterday my usage was bouncing around 60% with the misconfigured encoder.Then the issue may simply be high CPU use if it's good at 60%. That's why we'd need confirmation at 60% and below.
-
You may be aware, but this is not fixed on Windows in Wirecast 14.3.3.
Log from this morning. Not sure how your logs are structured, but this is an overnight run that I was stability testing without NDI output, but this morning I enabled NDI output and at the end of the log it was dropping frames.
-
Billy Davidson It may be a bit before the fix is released since there's a bunch of related work that needs to be done along with it.
-
-
CraigS How's this coming along?
-
Billy Davidson It should be fixed in an upcoming release although I can't yet give a date for that.
-
-
The issue may be fixed in our public Beta
Download Wirecast 15 Beta Here.
https://telestreamforum.forumbee.com/t/y4hjdwp/15-0-beta-dec-23-2021-social-media-comment-moderation-performance-lots-more
Please test the issue you reported and post in the Beta Discussion forum only.
https://telestreamforum.forumbee.com/category/wirecast-beta-discussion
