Video file playback audio beginning to go out of sync as program progresses
We have recently noticed that in a particular program that over time (a few hours), playing back media files (h.264 5000mbps encoded mp4s), slowly starts to lose audio sync (audio lags behind he video picture). Sometimes I can almost hear the moment when things shift, like a slight audio artifact-ing (glitch occurs). The only way to seemingly fix this is restart Wirecast.
Windows 10
CPU i5-7600K
Ram 32gb
GTX 1070
Wirecast 14.3.4
System CPU: 25%
App CPU: 23%
Encoding streaming at 720p, 3500kbps (however, I can replicate this without even streaming, in which case the CPU load is down by about 15% on average).
I noticed that when I am building our my program, the process of building shots/adding media etc.. sends wirecast into a completely fubared state for actual live. I can never not "restart" prior to starting a show.
Our clients are starting to become unsatisfied with these little bugs that start to show up when we are producing events. Its becoming painful not being able to explain where we "screwed" up when we aren't changing anything in our process (and really havn't for at least a year). Is this the computer? or the media? what am I missing, its driving me nuts lol. Again this is JUST when playing back video media. Sync issues with live sources we can always adjust for in software or hardware.
-
Sorry to hear that.
How was the mp4 created?
I assume you're hearing this monitoring Wirecast and not just the stream but please confirm that.
Would you be able to link to a file we can test?
At what rate is the drift? For example how far out is it at "x" minutes? We may need this if we test your file.
Do you have the wherewithal to convert one of the MP4 files to Apple ProRes to see if the sync drift still occurs? If that fixes it, it might indicate an issue with how Wirecast handles the source codec.
Cody Ferreira said:
I noticed that when I am building our my program, the process of building shots/adding media etc.. sends wirecast into a completely fubared state for actual live. I can never not "restart" prior to starting a show.We'd need repeatable steps and what the actual failure is?
-
Hey CraigS thanks for the response.
The files were created by Vimeo's transcoder (we downloaded them from the 720p rendition). Unfortunately I cannot send a sample as the content is something that is not public facing (its for paid users etc..).
The drift appears to happen at random and random amounts. There is no catchup. Its as if the demuxer in wirecast literally starts to take a dump and never corrects it self. As I move from video to video, nothing ever resets, nothing ever gets better...eventually only gets worse (until I restart wirecast).The drift can be seen both in the program viewer (live program, the right window), and can also be seen in the encoded live stream as its going out (what I see, the end users see).
I can certainly change the fils to a ProRes format and try things out and see how it runs.
The repeatable steps are something along these lines:
1. Play a 1 minute video (on loop) for 5 minutes or more
2. Play a different 2 minute video until it ends
3. Play another 1 minute video until it ends
4. Play a more long form video (10 to 60 minutes), until it ends
5. repeat from steps 2 through 4.
Anything else you can advise while I look into a prores encoded test against this flow?
-
Cody Ferreira Do single files play without issue?
Are you using these files in a playlist?
Does sync issue show in the first file you play or only when you move to the second file?
I tested one 19 minute file playing it straight through and it remained in sync the entire time. Looping it again to see if it drifts. -
CraigS Its not something that just happens on a single play through, its something where we see if begin to happen after going through the flow I described and playing videos consecutively. No there is no playlist, this is all manually cued. I can play a video once without any issue, its a matter of moving from shot to shot and then over time all of a sudden we starting drifting more and more out of sync as time goes on.
-
-
Thanks for the above info.
Cody Ferreira said:
I can certainly change the fils to a ProRes format and try things out and see how it runs.Please do. That's an important test.
-
CraigS So I was finally able to get to this test. The ProRes doesn't seem to really do much of anything different other than bloat the crap out of the small file. It takes roughly the same amount of system power to playback the file versus a 3 to 5 mbps mpeg file. Also if the system is in disarray the prores playback will also be in disarray (out of sync).
I also tried the different "user" which didn't seem to really show any difference. I actually was having a hard time replicating the issue easily as i had to run about 3 hours of video before I saw it.
I was able to get wirecast to go out of sync by some brute force which gave interesting results. I was encoding a video file in Adobe Media encoder and decided to also push a video to program in wirecast (was not streaming or recording). I could see the lad and hear some skips in the audio (while the system CPU remained VERY low). Obviously I would never do this while actually broadcasting but I was just trying to get things to not behave. Once things started to go out of sync, there was no coming back from it with ANY video being played in wirecast without restarting wirecast. This seems odd to me. Why would an obviously disrupted operation like encoding a video in adobe media encoder while trying to run wirecast cause wirecast to PERMANENTLY lose audio and video sync on any video being played.
I also did some testing with the hard drive "sleep/turn off" option and it seems to have no affect on things. I tried turning it off, setting it to super low times (like 2 minutes) and could never get any repeatable action to occur.Lastly I am doing some stability testing on my system and the GPU so far after many cycles I haven't been able to find any issues. I am still digging however.
I have yet to fiddle with the beta as I haven't had time.
-
-
Cody Ferreira said:
Unfortunately I cannot send a sample as the content is something that is not public facingWe won't disclose the contents. You can use our forum private message system and link to a secure file transfer system such as Dropbox, etc. We could download a random file but if we can't reproduce it that would impact resolving/fixing.
Investigation issue WIRE-20142 -
Cody Ferreira said:
all of a suddenat the point of switching a shot or somewhere in the middle of playback.
It's possible that there's some background or drive activity that interrupts playback. -
It appears to happen in the middle of playback, it could be something where it also happens when switching the shot, but its impossible to tell because the shots when switching dont have audio vs. video to really tell if thats when it happens. I know for sure I have seen in just happen while playing back, again to reference my original message... i can sometimes even hear an audio "glitch/artifact" occur and it throws things out of sync a few frames.
Also Im doubtful of background activity, because we run it on a system that doesn't have anything running application wise except for wirecast and windows 10. So if it is a background activity it would mean wirecast is super sensitive and essentially broken to something with windows 10. As we are running it without any applications that do not require actual user initiated function (like a dropbox, adobe cloud, slack etc.. etc..)-
Cody Ferreira Yet it's odd for a file that plays fine on its own to suddenly develop an issue independent of any other activity.
No virus checker that might be checking the drive?
Might it be drive throughput having a momentary issue?
Are you playing the files on the system hard drive (if so 7200 or 5400) or is it an SSD drive and do the drives have more than 20% free space?
Currently I don't see any other users reporting a file playback sync issue with your file types (h.264 with 48kHz audio). -
CraigS ya it is odd, but its something very repeatable as we had two back to back days of the issues with a wide range of media files from the client ranging in length but the broadcast flow was the same.
the media is located on an ssd, with 50% or more of free space always as well as the os runs off an mvme drive. No antivirus installed only stuff that runs is mandatory to windows.
-
Cody Ferreira I'm testing with your files on a 2013 Laptop with i7 4th generation with only integrated GPU and 16GB RAM also off system SSD and can't duplicate the issue. So my laptop is considerably less resources than yours.
I don't think it's related to the files.
Try creating a new user account and testing.
-
CraigS Crap, that is not good, that means there is something wrong with the machine potentially which will be impossible to diagnose im sure.
A new user account? Like a windows user?
-
-
Cody Ferreira said:
A new user account? Like a windows user?Yes. That will determine if there's an odd account-based issue.
-
I will give that a shot as soon as I can.
Thanks!
-
Cody Ferreira BTW the developers say that it may be possible the drive goes to sleep while playing the file and then wakes causing the glitch. They also noted playback improvements in Wirecast 15 so you may want to test 15 Public Beta 2 to see if that helps.
-
CraigS that is good feedback, I will also check on that behavior, I wasn't aware windows would apply power options like that to an SSD. I will give it a look.
Also would be happy to check the beta, were can i download?
-
-
Take a look at the sample rates. Most sync issues are due to different sample rates 44.1 vs 48KHz. Or try some other options like the free OBS and see if it's more a hardware or software wirecast vs. OBS. Vimeo has a production software for free as well from the aquisition of livestream.
-
Cody Ferreira said:
I actually was having a hard time replicating the issue easily as i had to run about 3 hours of video before I saw it.I can't help but think this is a system-related issue since we have no other reports like this.
Cody Ferreira said:
The ProRes doesn't seem to really do much of anything different other than bloat the crap out of the small file.ProRes shouldn't drift so it really does seem like an odd system problem. Again no other reports like this.
Cody Ferreira said:
I was able to get wirecast to go out of sync by some brute forceThat seems like an extreme circumstance.
Cody Ferreira said:
I was encoding a video file in Adobe Media encoder and decided to also push a video to program in wirecast (was not streaming or recording). I could see the lad and hear some skips in the audio (while the system CPU remained VERY low). Obviously I would never do this while actually broadcasting but I was just trying to get things to not behave.It sounds like Wirecast would behave under normal use.
Cody Ferreira said:
I have yet to fiddle with the beta as I haven't had time.Please do as the file playback has likely improved. The developers are waiting for this to move forward.
-
CraigS sure thing..... I just wish I could pin point a system issue as the system runs SPOTLESS. Even stress testing the GPU, CPU, Memory... not a single error or issue. Only issues appear in wirecast.
-
Cody Ferreira said:
just wish I could pin point a system issue as the system runs SPOTLESS.Odd. It may not be a resource issue. Something is causing a momentary "burp" it would seem. Especially since it seems you can sometimes hear the point it occurs.
Cody Ferreira said:
Only issues appear in wirecast.Live playback and switching might be more sensitive to this kind of issue.
Usually background task or drive throughput would be the suspects. RAM, especially if the computer were resorting to using temporary drive space would be a possibility. But your system seems to have specs that wouldn't cause such an issue if you weren't running any other programs and using an internal SSD for file playback and had plenty of drive free space.
-
Cody Ferreira BTW it's not clear if you tested a New User Admin account. Please test that as well.
-
CraigS I did test another user account, I didn't see the issue when I was testing another user; but then again I wasn't able to run a 3+ hour test.
-
-
Cody Ferreira said:
I did test another user account, I didn't see the issue when I was testing another user; but then again I wasn't able to run a 3+ hour test.That may well mean a user account-specific issue. The test should match the circumstances. It may well indicate some background task-specific to the user account interfering.
-
Yea the hardest part of reproducing the issue is trying to find a free entire day to replicate the whole broadcast. I need something like at least 3 or 4 hours to even get to the point where I was seeing failures. Its the struggle buss to get there. As I was saying earlier, I could even get the system to a failure point successfully prior to that stretch of time on the normal user account. Also I have no background tasks running beyond whats native to windows and whatever resources are needed to run wirecast. Given the state of the machine, its build, time using wirecast (for years and years) its really hard to point the issue at the machine unless its something SO tiny that wirecast just has a bug with that "tiny tiny little task".
One thing I realized, as it didn't occur to me until just now, is that we were running a rendezvous instance. We only had two users connected and were only being used in the program for about an hour of the 8 hour productions. Im wondering how much of an effect Rendezvous could be factoring. For example, we do run the live program playback for the rendezvous users and it does tax the system an additional 10-15% CPU (which seems really strange). Again never going above 30%-35% when "playing media and running rendezvous". I will do some more testing as I find the time later this week to see whats going on, will also try to test the beta.Is there a way to install the beta in parallel to the current stable version?
-
Cody Ferreira said:
I need something like at least 3 or 4 hours to even get to the point where I was seeing failures.I can understand. Yet if it were a file playback issue it would likely be easier to reproduce. I don't doubt there are other users doing longer programs but so far no reports of a similar issue though so it's hard to find the cause. If it's your personal time perhaps setting up a playlist and letting it run can duplicate the issue.
Cody Ferreira said:
we were running a rendezvous instance. We only had two users connected and were only being used in the program for about an hour of the 8 hour productions. Im wondering how much of an effect Rendezvous could be factoring.I'm not sure if that would impact file playback. I could understand if Rendezvous itself were going out of sync if there were an issue with the server or a peer to peer connection.
Cody Ferreira said:
have no background tasks running beyond whats native to windowsIt could certainly be something native to Windows but again we'd probably hear from other users if it were common.
Cody Ferreira said:
Is there a way to install the beta in parallel to the current stable version?Not on Windows, unfortunately (it's easy to do on Mac).
-
I have finally made time to run the beta and the issue i am running into is audio and video still at random, drifting out of sync. However in the beta, the playback of the "live program" is out of sync, but the actual stream encoding (as I am seeing it playing back in a video player like vimeo) is in sync. Really frustrating.....after production this week I am going to completely re-install windows and wipe everything.
-
Another update. It seems pretty consistent that in the beta when viewing the "live" window that video and audio are out of sync, yet.....the encoded video that is actually being pushed to our RTMP endpoint is perfectly in sync.
We also had a complete crash when enabling the live output of the video in the rendezvous dashboard (for the guests to see the liveouput of the program). I haven't been able to reproduce this issue, but essentially, I just turned on the live output source like I normally would and the whole app froze to the point of it needing to be force-terminated.
-
