0

Wirecast Bug: Bitrate Spike - Apple Encoding #00618449

Team.

I have been battling trying to fix problems associated with my streaming. One problem I had previously believed to be related to my ISP. Let me explain the problem.

Regularly throughout my streams, I find times where the stream is "buffering" in youtube, and the stream icon goes RED in Wirecast. After last week's stream, I did a review. It came as a surprise to me that every time I had a sudden transition, this problem happened. For example - going from a virtual studio shot to a radically different full-screen shot on screen.

I did some testing today, to narrow down the problem. For the sake of simplicity, I setup a 1-shot document. This document has only one video source, and corresponding audio coming from Dante VIA. (I tested without audio - and got the same results).

My desktop background for the VLC window is a black background. No content. Multiple times, I started playing a video on VLC - and I keep getting problems with the stream.

So. I took this early data, and did more testing. I did a RTMP stream to a RTMP proxy on the same computer. No network in between. But I still used the same profile from the youtube stream - which is 720p30, 1250kbps using the Apple encoder. For some reason Wirecast decides to send 11 Mbps. (See graph below).

I re-tested with X.264, and that behaved a lot more as expected. Totally reasonable bandwidth.

So - here is my problem. If I use X.264, I don't use the GPU - and CPU load is too high to stream successfully. If I use Apple, I use the GPU - but the stream keeps crapping out when this happens.

What really annoys me more - is the fact that I have been blaming my internet company, for something that it turns out is related to Wirecast.

 

14replies Oldest first
  • Oldest first
  • Newest first
  • Active threads
  • Popular
  • Since it seems specific to Apple H.264 do include OS version as well.
    You mention a "sudden transition." Please explain. Cuts? Have you tried different content for outgoing and incoming shots? It's possible that Apple H.264 doesn't have any constraints on peaks.

    Like
  • CraigS - The OS here was High Sierra. I have not yet updated to Mojave, because some apps that I require do not fully support it yet.

    Let me qualify cuts - During my stream, I was cutting from one shot to another, and immediately after - pressing the play. button on VLC which fed to a screen capture. For the actual testing where I replicated this, I was able to get the same effect without doing any shot transition in Wirecast. Just pressing the play button in VLC would cause this effect.

    On your point of apple H.264 not having constraints on peaks - that may be true.. but how does Wirecast handle that anomaly. If it pushes packets as fast as possible, that would be a bad choice. Alternatively, it could keep the packets queue'd up and send them out as fast as permitted in the bandwidth settings in the encode setup.

    I don't have any evidence yet... but I am assuming it sends as fast as possible... because that causes my outbound link to get saturated, which causes packet loss, which causes TCP retransmission, which causes youtube streams to die... and I am certainly seeing that.

    The hard thing here - is measuring the actual packet flow. Using SNMP and similar tools - it is measured at intervals, and the returned data is an average over x minutes each time it is read... so no really good data there.

    The only way to confirm this would be to use Wireshark or a similar tool to capture network traffic, and generate a report.

    Like
  • Greg Kuhnert said:
    Let me qualify cuts - During my stream, I was cutting from one shot to another, and immediately after - pressing the play. button on VLC which fed to a screen capture.

     Can you create other scenarios where this occurs? Playing a source inside Wirecast for example? Seeing if it correlates with rapid change over time (which can impact encoders). Transition (cuts vs dissolves too with the same shots) would be interesting as well.

    Greg Kuhnert said:
    The only way to confirm this would be to use Wireshark or a similar tool to capture network traffic, and generate a report.

     Might be a good idea.

    Like
  • I've done a little reading since my earlier post. I understand that X.264 has a peak bitrate setting. Not sure what it has as a default, but that might explain the difference with software versus hardware encoding if it has a sensible default. I am assuming that Wirecast calls to the Apple encoder are not passing a peak bitrate?

    Regarding the other topic - I can test this too - but I vaguely recall seeing some bad performance doing cuts and media inside wirecast like this a long time ago. But I did not analyse in detail at the time.

    GK

    Like
  • Greg Kuhnert I believe hardware assisted encoding doesn't have a peak control so it's possible that presents a challenge.

    Like
  • CraigS said:
    I believe hardware assisted encoding doesn't have a peak control so it's possible that presents a challenge.

    I can confirm that I have tested this using Wireshark. Please find attached below a bandwidth graph from Wireshark, that confirms the graphs in Wirecast are correct, and that the peak throughput is an order of magnitude bigger than what was configured when using an apple encoder. I had a video clip set to LOOP - The baseline is at 1.5 megabits per second, and the peaks are around 7 megabits per second. 

    Regarding your other comment. I had read elsewhere where apple did indeed support peak throughput. There was a bug in Apple's implementation where if you change the peak throughput once a stream has started, that change was ignored. However, it was indeed available.

    Therefore, one of two things is happening here. Either:

    1. Wirecast is not setting peak throughput to a sensible value, or
    2. Wirecast is setting the peak throughput after the stream first establishes, and this setting is ignored by the operating system, due to the apple related issue I found previously.

    I tried to find that reference to the Apple bug again, but I was unable to find it... but the peak throughput setting is where the developers need to look for this particular issue.

    GK

    Like
  • Greg Kuhnert said:
    I tried to find that reference to the Apple bug again

     If you can find it that would be good.

    Like
  • CraigS said:
     If you can find it that would be good.

     Yeah. I did try, but was unsuccessful. But here is a similar reference:

    https://forums.developer.apple.com/thread/87047

    Not the exact post, but the same basic problem. Its referring to High Sierra in the post. I am unsure if it is still a problem in Mojave - I can't upgrade, due to other apps not being compatible... but it would be interesting to know if its fixed in Mojave.

    Like
  • Greg Kuhnert Thanks for that.

    Like
  • Greg Kuhnert With investigation we've confirmed the issue is with the codec itself so it's in Apple's providence.

    Like
  • Hi Craig. 

    CraigS said:
    Greg Kuhnert With investigation we've confirmed the issue is with the codec itself so it's in Apple's providence.

    I agree 100% that there is an Apple problem. The question is, if we don't know how/when a fix will be provided, what are the options that will reduce or minimise the impact?

    1. In the thread I found, it talks about a change to the peak bandwidth being made after the stream is established. Now I have not investigated the parameters at the API level... but can I ask your dev's to confirm - is the peak setting being made DURING the establishment, or AFTER the initial establishment? If it is after, this is the known apple bug. A simple change to do it DURING establishment will eliminate the known problem.

    2. It would be possible to pipeline packets in a Wirecast owned buffer before throwing them out the ethernet port, to "smooth out" this known problem. That would require significant effort to code compared to #1, but it would solve the problem.

    3. I don't yet run Mojave. Theoretically, this could be fixed on that OS. Did your tests show this still present on Mojave? If it is fixed there, I will push the other application vendor that I use to upgrade their code sooner, to bring forward an upgrade.

    Conclusion: If we know its fixed in Mojave, that is a happy answer. If not and Apple do not publish an ETA, we need to look at #1 or #2 above. Otherwise, this will continue to have throughput impacts for all users of Wirecast on OSX - which will be particularly noticeable in countries with poorer upload speeds.

    GK

    Like
  • I have been having a look back at this problem again. In reading my old data, it was a HEVC encoding bug, not H.264. So that known apple bug is not the cause of the problem.

    So. What can we do? I had a look at the libraries for the Apple Video Toolbox, and I came across the exact item that I have been asking for.

    The toolbox allows you to set a hard limit on data rate in addition to the existing soft limit that is configured by Wirecast. I have previously submitted a request for "bandwidth smoothing" or some other buffering within Wirecast, but that is not needed. This can all be fixed with one line of code, to set a sensible hard limit.

    As a quick fix, Wirecast could for example configure a hard limit that is say 25% higher than the soft limit, and that would prevent people using Apple hardware encoders from having dropouts related to this bug.

    Check out the official Apple API where it publishes details about the DataRateLimits property for more information.

    Like
  • Greg Kuhnert Should this be closed and start a new topic since the Apple bug is not the correct reference?

    Like
  • CraigS Absolutely not. The title is accurate

    Bitrate Spike - Apple Encoding #00618449

    That is still 100% accurate. There is a bit rate spike when using apple encoding, and the ticket number is listed. We thought in the discussion it may be a bug on apple side, but my recent analysis indicates it was not an apple bug. This is all about how to set data rate limits and my assumption that this is not currently set.

    GK

    Like
Like Follow
  • 3 yrs agoLast active
  • 14Replies
  • 304Views
  • 2 Following