Still dropping frames and now crashing
I've posted before about problems I've had this sports season (starting Sept) on my Gear 110. I was getting a lot of dropped frames when doing graphic overlays and instant replays. Lately, the computer has started to completely lock up and require a hard re-boot (3 times in one game a few weeks ago). CPU usage is usually in the 20s and bandwidth is way beyond adequate so it doesn't seem that resources are being over-taxed.
The Gear had worked fine last year, so I was wondering what might have caused the change:
1) I added a Sportzcast scorebot, using New Blue Sports Titler for scoreboards. I did a test a few weeks ago without using the scorebot, and had no problems, so I thought I had the problem solved. But then when I did an actual game without scorebot, the same problems with overlays and replays popped up. Although it never locked up completely, I did have some menu freeze-ups.
2) Over the summer, the warranty on the Gear expired, so I opened it up and added a second SSD for storage. This is where I now put my graphics and save my replays to. Right now, I'm not able to test the Gear adequately, and probably won't be able to until the next actual game. Is it possible that this could be causing the problems? I've had other computers where I've saved graphics and replays to a regular Hard drive and didn't have any problems, so I can't really see that a second SSD would be a bottleneck -- but I can't think of anything else that's been changed.
The Gear has a 'Reserved' Drive D on it, but no files show up. Is this the same as a 'Restore' partition that would let bring the system back to original condition? If so, would that be worth trying?
Got some info from the Support team, and they acknowledge that the on-board graphics on the Gear is very limited and is more than likely causing the problems. I did a test with my 3-year old tower computer and checked GPU usage in comparison with the Gear. The tower would be around 10-15% with Sportzcast AND another graphic overlay playing on top of the main video. For the gear, that was getting 65-70% (and this was with using a standard monitor rather than the 2K monitor that I use for webcasts.)
Very disappointing to say the least. I've wanted a scoreboard hookup since I first started streaming, and Sportzcast is working great. Now I have to decide whether I junk a $5,000 computer or the $1,000 I spent for the Sportzcast setup.Reply
On Friday I did a basketball double-header using my old mid-tower unit. Not a single dropped frame or freeze-up.
I doubt I'm the only person to run in to this problem with the Gear. I would think that anyone who is going to spend from $5000 to $8000 on a streaming computer with 4 camera inputs is probably going to do more than a simple point-and-shoot webcast. They're more than likely intending on doing lots of graphics, multiple cameras, live scoreboards, multiple streams, etc. To foist off a setup on high-end users that is incapable of doing even 2 overlays isn't going to sit well with a lot of people.
The reply I got from Tech support mentioned that they were aware of the problem and were trying to come up with a solution. Since the GPU can't be upgraded, I doubt I'll see any significant changes that will allow me to keep using the 110. Right now my most viable option seems to be to move the 4-port Magewell to my tower, put a single-port capture card in the 110 and hope some beginner level streamer might want to buy it.Reply
Last month I purchased a 110 and Scorebot to do the same thing you are. Before I go into detail, I’m courious if you have your Gear connected to Ethernet when broadcasting? That is the only remedy I have received so far from support for the massive frame drop rate I experience by doing just a single cam and no overlay setup. We’re still in the process of getting the Scorebot connected at our scoreing table so I haven’t been able to test it out. Very disappointed to say the least.Reply
Yes, I am using an Ethernet cable for my connection.
The first year I had the Gear, it worked just fine -- I used 4 cameras and plenty of graphics (although only 1 graphic at a time).
The problems started with the Sportzcast scorebot. Just having the Scorebot overlay worked fine, but running certain graphic overlays on-top of the Scorebot caused lots of dropped frames, and instant replays looked terrible because of the dropped frames. I suppose I could quit using the scorebot, but it is so handy that I think I'll just put the Gear away and use my old computer, which can handle the graphics just fine.Reply
Until I get IT to hookup my Scorebot and run a line for the Gear, I've been using WIFI and Scoreboard OCR on another computer to make the NDI stream for the Gear. Reseting the Gear to factory solved my BSOD issues I had the first broadcast but not the dropped frames. I'm hoping being hardwired solves that problem. Reading your story, I'm now debating if I want to drop $500 on the Scorebot program license that I have yet to purchase. Sounds like selling the Gear and building my own system will be cheaper, unless Support finds a fix. It's hard to believe they didn't put a dedicated GPU in a $5,000 machine.Reply
Jerel Peterson said:
The Gear has a 'Reserved' Drive D on it, but no files show up. Is this the same as a 'Restore' partition that would let bring the system back to original condition?
All Gear units ship with a OS 250GB M.2 SSD. This is the C:\ drive. Additional storage depends on model, with the 110 having one additional 2.5" SSD storage drive D:\. The 230 model has two additional storage drives, D:\ and E:\. This is for storage, and is not an restore partition. It is advisable to record and store replays to the storage drive, rather than the OS C:\ drive from a performance optimization standpoint.
Jerel Peterson said:
I opened it up and added a second SSD for storage. This is where I now put my graphics and save my replays to. [...] Is it possible that this could be causing the problems?
Unlikely that this is having any affect, but a I/O test on the drive can rule it out. It would not be unheard of for a drive to exhibit read access issues, which could cause CPU/lockup issues.
Jerel Peterson said:
I added a Sportzcast scorebot, using New Blue Sports Titler for scoreboards.
The scorebot itself is unlikely to have caused a big performance hit, but NewBlue can certainly be the culprit.
It is important to understand the workflow to be able to answer this one way or the other however. Live rendering of 3d titles can be very computationally expensive, but it also depends on how complicated the file being rendered is.
It is also worth considering running NewBlue on a secondary machine over NDI. This is actually the recommended workflow in most any broadcast situation. 3D rendering can be expensive, and pairing it with a live switching and encoding solution on the same machine can quickly allow for over usage of resources if care is not taken. This is why in a typical live scenario, titles are ingested from a secondary machine.
There are also other inefficiencies possibly in play, e.g. not using Quick-sync for your encoding and opting for the more expensive x264 option, using a canvas frame rate of 60 when you are only encoding 30, ingesting your video sources in interlace instead of progressive which turns on CPU de-interlacing, Ingesting your sources as 1080p when you are only encoding 720, using the multi-viewer feature that was introduced in a later version of Wirecast in conjunction with other tools such as NewBlue, other background applications utilizing system resources, etc etc.
There are also other highly common no-nos to consider as well. You should never be watching the broadcast in a web page on the same machine as the Wirecast instance. Decoding video is not free, and doing so can be the difference between dropped frames and no dropped frames. Are you streaming to multiple places, or doing a program recording to disk? If recording to disk, unless you absolutely need mp4 for size reasons, MOV is better for editing, and is disk intensive rather than CPU intensive. This can absolutely help.
Given that you cited low CPU usage with lockups, I would venture a guess that moving NewBlue to a second machine over NDI would help some, but unfortunately there is not enough info here to say one way or the other. I can say that Gear is more than capable of multiple streams with multiple overlays, but with enough inefficiencies or computationally expensive operations introduced, you can make any system drop frames.
Please try looking into your settings and seeing if you can improve efficiency here and there. If you notice you are dropping frames on one particular file type addition such as MOV or MP4, try re-encoding it as a different file type in handbrake or similar.Reply
I'm not watching the stream on the same computer. I'm only streaming one instance (to Youtube), I'm not recording to disk.
I'm not actually employed at the school where I stream, so carrying in 2 computers for each game to have NDI isn't going to be the answer. My mid-tower handles the streaming without a hitch. It's not as portable as the 110, but it's more portable than the 110 and a secondary computer for Titler.Reply
Jerel Peterson Right there with ya. The whole point of getting the 110 was so everything could run on one computer. A 5 year old Asus ROG laptop that a friend let me run tests on can handle everything I throw at it, why can't the 110?!!! I'm was hoping to test the Scorebot out on the 110 tonight to see if I got similar results as you but my school's IT department is swamped. Maybe by Saturday. I'll keep everyone posted.Reply
It sounds like you guys have narrowed the issue down to the number of NewBlue Graphics you are using at a time when being served from the same machine.
Sounds like scorebot is mission critical, but the NewBlue active overlays are to add polish to the production.
Since the traditional broadcasting paradigm of the CG or 3D title generator being a secondary machine is not workable for your uses, at this point I think your only option left relating to Gear is to minimize your usage of NewBlue titles.
Are you live encoding those additional overlays (making a change to the title itself while you are actively streaming)? If you are, and do not need to be, exporting them to MOV, or MP4 could improve.
Additional information -
It is important to note that the dropped frames are due to latency. A frame must be delivered within 33.3ms(30 FPS) or 18.6ms(60FPS) to not get dropped.
Any additional software running outside Wirecast has potential to increase the latency. The GPU usage itself can point towards increased latency, however is not the same measurement.
A machine with increased specs, or an added discrete GPU can certainly reduce the latency in question, but if the result was a decrease from 35ms to 30ms, a background windows task could suddenly cause dropped frames. Therefore, it is against best practices to run additional 3D or encoding software on the switching and encoding machine. It opens more opportunities to reduce stream quality and is why it is avoided whenever possible.
I do certainly understand the assertion that it works on this machine or that machine, but not Gear.
Unfortunately, determining if there is a bug or looking for optimization opportunities is likely the only way forward currently. Gear is absolutely used in many high value productions across the world, but certain combinations of computationally expensive features can bring all but the most elite machines to their knees.
In this case, the 3D engine of Wirecast, the 3D engine of NewBlue, stream encode, streambot, and multiple encodes/decodes for a few 3D titles seems to be a bit of an overload for Gear with the specific variables you have in play.
Will a purpose-built gaming machine (ASUS ROG) have better 3D performance? In most cases, yes. Most streaming workflows do not require a heavy 3D load, and when following best practices, would not be an issue within these workflows.
There is such a vast array of streaming workflow requirements, that no one machine can possibly address all of them without being prohibitively expensive to most users.
4K streaming requires ultra-high-end equipment and great care to optimizations even on that equipment. One could easily assert that they are only grabbing one 4K source which is the equivalent to four 1080p sources and streaming it, and that their i7x gaming machine with a 1080ti from 2 years ago can do it, why can't Gear. However, this is outside of most workflows and not what Gear was designed for.
My apologies that there is not a clear and easily solution that would enable you to utilize Gear to meet your new workflow requirements.Reply
Bryce Stejskal Thanks for taking the time to provide all that great info. What would an ideal workflow be for the gear? Do you have any insight into what cameras, number of cameras, switches, settings, number of overlays, etc... that “high value productions across the world” are using with their gear?Reply
So if it's New Blue itself that is causing the problem, would using the Scorebot with the SportzCast Live CG be any better? I've never used it, but with LiveCG you apparently create your own custom scoreboard as a PNG file and then have the data fill in to your own designated areas.Reply
Orion Lemler said:
Bryce Stejskal Thanks for taking the time to provide all that great info. What would an ideal workflow be for the gear? Do you have any insight into what cameras, number of cameras, switches, settings, number of overlays, etc... that “high value productions across the world” are using with their gear?
Sure thing, happy to provide insight when I can. Apologies it is not the greatest news.
That is a pretty broad question. Gear is fairly adept at most workflows. Exceptions being workflows that require more I/O than Gear has(capture/output), highly demanding 3D workflows(extensive multi-viewer use, other 3D render engines running, game streaming with game running on Gear, etc), and 4K video. I think any of those are outside of an ideal workflow for Gear.
Types of cameras do not really matter too much, though that could be picked apart at a highly technical level(USB cameras may send encoded video, or RAW, which has performance impacts though small, networked video sources require decoding in most cases). PCIe bus bandwidth with 4x1080P60 sources is more than 4x720p60 which can increase latency, so your configuration will determine if the decoding hit of say a network source is more impactful to you than someone else.
We have customers doing 4x1080p60 with record to disk in MOV at 1080p60 but streaming 720p60 via Quicksync, with any varying number of overlays and replays/ISOs. We have customers with just about any variant of that you can imagine. If they added multi-viewer output in 17slot config, they would probably start dropping frames. Another user without the multiple simultaneous MOV or MP4 overlays may use multi-viewer with no ill effects, and another might have to consider dropping to a 30fps workflow simply because they also need multiple chroma keyed sources along with the MOV and MP4 overlays. It all depends. If you have a pure 720p workflow, many of those considerations become much less of a concern. If you add a live NewBlue Titler instance to the same machine, you may have more considerations to remember.
That makes me think a bit, some sort of cost estimator, or points system could go a long way towards improving that experience. Likely difficult to do something like that with any level of accuracy however.
Jerel Peterson said:
So if it's New Blue itself that is causing the problem, would using the Scorebot with the SportzCast Live CG be any better? I've never used it, but with LiveCG you apparently create your own custom scoreboard as a PNG file and then have the data fill in to your own designated areas.
If it is a static PNG with just the text data updating, and that would remove the necessity for NewBlue to be active, that could certainly make a world of difference in your workflow. I also have not personally used LiveCG for that, so I can not say with certainty that will resolve it, but from a technical standpoint, removing the NewBlue render/encode from the system would free up much more resources than a PNG would use. Though I must caveat, high refresh rate of a png file with flattened text to serve as data display might not fly. I imagine you would be adding the PNG in question, and then pointing our text source to a text file that is being updated at a high rate. We do actually have an improvement related to the text refresh rate slated for WC12 btw.Reply
It now appears as though my problems with the Gear 110 had absolutely nothing to do with the Scorebot and/or TitlerLive Sport.
Just before the last game of the season, my mid-tower computer quit working, so I had to go back to the Gear for the final game. I did not connect to scorebot/scoreboard at all, and didn't have TitlerLive running at all, either.
Yet I still had significant dropped frames during instant replays, and the computer locked up twice during a single game. The second lockup happened when running a graphic overlay, I'm not sure when the first one happened as I was running the camera and not looking at the computer monitor when it locked up.
So at this point, I'm at a total loss as to knowing what to check for next. As I mentioned earlier, the Gear worked flawlessly last year (with 4 camcorders and plenty of graphic overlays). Why would it crash and drop frames now when I was only using 2 camcorders? Last year I was even streaming to two destinations--The Cube and Youtube. This year it has only been going to Youtube.Reply
Jerel Peterson Are you willing to run a test stream?
- Reset Wirecast Preferences. Wirecast > Help > Send Support Information but click on the lower left Reset Preferences.
- Restart Wirecast.
- Start with a New Document (just in case a previously created document is carrying the issue).
- Check the drives you are using and make sure they have ample free space. You should have at least 20% free space even after recording an even otherwise that might impact performance.
- For this test make sure you aren't using Multi-Viewer at all.
- With nothing else running on the Gear, use two cameras and Replay (don't use any graphics or imported files) and stream to YouTube.
- Tell me what happens with dropped frames and what point they start occurring if they do.
This is maddening...no consistent results. I did 4 tests with WC 11 and then upgraded to WC 12 for 3 more:
I first reset preferences. I have over 50% of my drives free. Didn't use Multi-viewer. 2 HDMI camcorders.
Test 1: Intel QuickSync, 1080p30, 6000kbps
'plain' video was usually at 25-40% CPU, slightly higher with replay. No dropped frames with replay.
Test 2: X264, 1080p60, 6.8 mbps. 50-75% CPU, many dropped frames
Test 3: X264, 1080p30, 4.5 mbps 38-50% CPU
120 dropped frames for a 10-second IR
I put in a bug overlay and also a stat graphic overlay. There were no dropped frames but CPU went up to 60% (this wasn't with replay, just the 2 overlays)
Test 4: Went back to QuickSync 1080p30, 6000 kbps (same as Test 1)
CPU was 17-34%, went up to 40% with stat overlays. Even though the settings were the same as Test 1, I was now getting over 100 dropped frames on Replays
Installed Wirecast 12
Test 5: Quicksync 1080p30, 6000 kbps
15-30% CPU plain video, up to 40% using a stat graphic
Instant replays would have 20-30 dropped frames for a 10-second replay
Test 6: X264, 1080p30 4.5 mbps
38-50% CPU for plain video, 50-65% with stat graphics
Instant replay had 0 dropped frames, CPU stayed at 50-65%
Test 7: X264 1080p30 6 mbps
CPU 45-55% plain video, up to 50% with stat graphic
Instant Replay had only 2 dropped frames with CPU in the low 60s%
There just doesn't seem to be any rhyme or reason for these results. Why were Test 1 and Test 4 to different when I used the same settings?, etc.
CPU gets pretty high with graphics, especially considering I wasn't using multi-display.
I didn't have any crashes, but I only ran each test for a few minutes each. During games, it usually takes about 1/2 hour or so before I would get a crash.Reply