| Printable Version of Topic
Click here to view this topic in its original format |
| Unofficial VirtualDub Support Forums > Testing / Bug Reports > Vp6f Mp3 .flv To Avi - Very Slow On 2nd Job |
| Posted by: 4N4 May 20 2012, 06:39 PM |
| I have been using VirutalDub to convert some VP6F+96kbpsMP3 .FLV files to XviD+128kbpsMP3 .AVI files, with resize from 960x544 to 720x408. The first job completes quickly, but subsequent jobs complete much, much slower. If I restart VirtualDub, the first job completes quickly again, but then subsequent jobs complete painfully slow. Not sure if this is a VirtualDub issue or codec issue, but it's reproducible every time. Definitely happens with 1.10.1 and 1.10.2, and am just double-checking with 1.9.11 |
| Posted by: 4N4 May 20 2012, 10:18 PM |
| I am using Xvid 1.3.2, Jawor's patched build (http://jawormat.republika.pl/xvid.html), although the problem also occurred with the vanilla 1.3.2 XviD codec. Confirmed issue also exists with 1.9.11 stable. For example, first job completed in 47 minutes, but a second different job has only reached 72% after 2 hours 48 minutes. On a quad core, CPU usage for VirtualDub was approx. 70% for 1st job, while 2nd job is only approx 31%. The slowdown makes the batch processing feature unusable. I have also noticed a slowdown for all dubs if I launch multiple instances of VirtualDub - one for each job. For example, jobs that usually take no longer than approx 1 hour, end up taking 6+ hours with four instances of VirtualDub. |
| Posted by: phaeron May 27 2012, 09:10 PM |
| What happens if you run the jobs one-by-one? You might try running one job, clearing the video codec setting, and then running the next job. If you can reproduce it on demand on the second job then we can try some interactive debugging, like the on-screen profiler. Oh, also... any difference in the Performance tab of the status window? You can enable this during a batch run even if it isn't enabled when the job starts. In particular, there are counters displayed there that indicate whether the I/O and/or the processing pipelines are backed up. For a job like this I would expect the audio/video buffer pipes to be full and the processing thread to be constantly busy. |
| Posted by: 4N4 May 30 2012, 12:56 PM | ||
How do you clear the video codec setting after the first job has run ? |
| Posted by: 4N4 May 30 2012, 07:09 PM | ||
Perf counters show Audio and video buffers are full for both jobs Audio buffer 64/64KB Video buffer 31 or 32/32 Video frame requests = 0 or 1 (mainly 0) I/O thread activity: 1 or 2% Processing thread activity: 100% CPU usage in Task manager for VirtualDub.exe starts at approx. 40% at beginning of first job, which drops to approx. 28-30% at end. CPU usage for second job seems to stay constant at around 30% (although the 2nd job is currently running now and is at 34% complete). |
| Posted by: phaeron Jun 2 2012, 09:37 PM |
| Yeah, like I thought it isn't I/O or audio. You can clear the video codec by choosing [Uncompressed] or [No Recompression] from the Video | Compression menu option. This unloads the last video codec used, which might clear any carry-over effects. |
| Posted by: 4N4 Jun 4 2012, 06:14 PM | ||
I'm running the second job now after clearing the video codec (all I did was choose Uncompressed from Compression dialog and clicked OK, then reloaded processing settings), and it looks like the second job is going to take much longer than the first job. The first job starts off at approx. 40% CPU usage, while 2nd job starts at approx. 30%. I/O counter for 1st job starts at approx. 5%, while 2nd job is approx. 1-2%. Prefs-Threading option are at default settings, although I tried setting Video compression threads to 1 but the problem still existed. |
| Posted by: 4N4 Jun 4 2012, 06:15 PM |
| I am running tests on latest 1.10.2 x86 on Win 7 x64 |
| Posted by: 4N4 Jun 4 2012, 11:56 PM | ||
1st job took 1 hour 11 mins 2nd job took 4 hours 29 mins Both jobs were using same input file, same processing settings, but obviously different output filenames. |
| Posted by: 4N4 Jun 5 2012, 12:49 AM | ||
| I have tried to reproduce on shorter clips (eg. 30MB 3 minute, as opposed to 800MB 1 hour 20 minute), but am unable to. If you want to try to reproduce the problem, you can get the VP6F .FLV files via RTMP: protocol from http://premiershiprugby.tv, although you will probably need to be in the UK or use a UK proxy. The problem files are the full games, rather than highlights. Processing settings being used (as listed in .vdscript) Edited: Feel free to truncate the VirtualDub.video.SetCompData line once you have the data, as it's interfering with forum layout in this topic by not word-wrapping.
|
| Posted by: -vdub- Jun 11 2012, 12:45 PM |
| Instead of using a Code box, does a Quote box work better ! |
| Posted by: 4N4 Jun 12 2012, 12:03 AM |
| The reason for using a code box over a quote box, is that a code box will display the text verbatim, whereas text in a quote box can have characters stripped out (eg. if they appear to be part of a tag). At least that's the way I understand how it works. |
| Posted by: 4N4 Jun 12 2012, 12:06 AM |
| Just tested with a quote box and although it doesn't strip out any of the characters (at least in my sample .vdscript), it still doesn't format properly so I've switched back to code box. |
| Posted by: -vdub- Jun 14 2012, 12:35 PM |
| I just noticed this forum is different for this than other forums. Where if place any oversize text in a quote box it won't truncate also. Instead the quote box will have windows guides so the forum doesn't expand as happened here. A forum back end setting needs changing, if there is a setting for it with the Invision Power Board v1.1.2 settings ! |
| Posted by: phaeron Jun 17 2012, 10:06 PM |
| I'll try hacking the forum stylesheets to see if I can force a wrap. It's probably using a PRE tag for CODE and there are a series of CSS3 and browser specific force wrap modes that might work for this. Anyway, back to the topic at hand.... I think at this point we might need more instrumentation within VirtualDub itself to track this down. In the meantime, if you're adventurous and still willing to track this down, the real-time profiler is probably the best bet. It takes a bit getting used to, but it will show you exactly which parts of the pipeline are taking time. Mouse wheel, arrow keys, and Alt+left-drag are used to navigate it. |
| Posted by: 4N4 Jun 17 2012, 11:41 PM |
| Whatever you do to the forum worked as it's now wrapping. Re. slowdown, I can't promise anything, but if I get some free time I'll give the real-time profiler a go. |
| Posted by: -vdub- Jun 20 2012, 12:39 AM |
| The fix phaeron has done for this now making other forums look poor in comparison |
| Posted by: 4N4 Jun 22 2012, 03:57 PM | ||
If you start a job, the Real-time profiler menu option is not available, and the keyboard shortcut doesn't work either. Is it possible to bring up the profiler window once a job is running ? Can you give a bit more info on what I should be looking for ? I see two lines in the profiler. They both start with a ~0.10 sec Audio block, then the top line has a ~0.05s Audio block every ~0.10 secs (gap varies slightly). The second line consists of multiple V-Decode->small gap->resize->V-BltOut->V-Compress blocks. Should I just make a note of average time taken for each stage of the pipeline in line 2, and then see which stage is taking longer than usual in the second job (ie. this stage is causing the longer processing time) ? BTW, forum e-mail notification doesn't appear to be working (at least for me) - I'm subscribed to this topic but don't receive an e-mail when someone replies. Are they working for you ? |
| Posted by: phaeron Jun 24 2012, 03:45 AM |
| Spam blocking has been a frequent issue with the forum notifications -- make sure your email address is correct and add forums@virtualdub.org to your spam whitelist if you have one. What you are looking for in the profiler is the bottleneck, i.e. which blocks are the longest. The length of each block represents time, so a longer block means that portion of the pipeline is taking longer. In particular, what we're looking for here is a block which is longer in the bad case than in the good case. Of the blocks you've indicated, V-Decode is video decode, resize is the resize filter, V-BltOut is an output frame conversion, and V-Compress is video compression. The different lines correspond to different threads, so what you're seeing is the audio thread doing a little work here and there and the video thread being the throttling factor. If you have threading enabled in VirtualDub for the various functions, you'll see some blocks moved out to their own lines and running at the same times as others, which indicates multithreading. This will only show multithreading within VirtualDub itself, as it doesn't have visibility into any threading that filters, plugins, or codecs may be doing. Unfortunately, you can't bring up the profiler after a job has started, as it's too late for the profiler to init then. |
| Posted by: 4N4 Jul 14 2012, 08:15 PM |
| Finally had some free time to look at this and, on the second slow job, it's the V-Compress that's taking longer (maybe 3 or 4 times longer, although the amount varies per frame). There also appears to be a bug in the profiler as while zooming in/out and dragging left/right the profiler has hung several times with the title saying Not responding and CPU usage higher than normal. The window goes blank (ie. white) when this happens. The profiler hanging does not stop the encode from progresing though. |
| Posted by: phaeron Jul 14 2012, 09:55 PM | ||
| Okay, try this: Close VirtualDub, then manually edit the VirtualDub.jobs file. Find every video.SetCompression() statement, and add a new one before it so you have something like this:
What this will do is force VirtualDub to unload the video codec before reinitializing it for the next job. Ordinarily this isn't a good idea as it can foul two-pass processing, but in this case it may be the only way to work around the problem you are seeing. If this works, then I can add an option to do this. In the meantime, I'll see if I can track down the issues with the profiling UI. |
| Posted by: 4N4 Aug 8 2012, 09:45 PM |
Unfortunately that didn't work either.  The first job took ~40 minutes (with an initial video rendering rate of ~75 fps, although it fluctuates between 50-80fps) and the second job has an ETA of over 4 hours (with an initial video rendering rate of ~10 fps). The only sure-fire way to get full-speed is to restart VDub.I'm wondering if I'm the only one experiencing this slowdown when converting to XviD and MP3, or whether it's happened to other people but they just haven't noticed. On an unrelated note - While saving a job file, the save dialog is set to only show .jobs files, but if I only put the filename without a .jobs extension and save, the .jobs extension is not automatically added. |
| Posted by: dloneranger Aug 9 2012, 07:51 AM |
| You've had 370+ views so far, and nobody's added "me too" I don't have any problems like yours either So here's a few random thoughts You could do a double test, queue the same job twice and see if the take the same time - they should be pretty close Does it happen with just .flv's? Is it the encoding or decoding that's the problem? You've mentioned you tried encoding with a few different codecs and are having the same problem with them all? You can test that the decoding's not causing the problem by setting the encoder to 'uncompressed' and queuing a few 'run video analysis pass' If the second one's slow doing that, then it's a decoding problem And lastly have you checked it's not something like cpu throttling because the pc's getting too hot after an hours encoding? HWMonitor (and others) can show your cpu temperatures, and the min/max values reached You can start the monitor before you start encoding and leave it running while the encoding is going on, then you can see the difference between idle and maximum http://www.cpuid.com/softwares/hwmonitor/versions-history.html |
| Posted by: 4N4 Aug 9 2012, 04:59 PM | ||||
I thought you might be onto something with the CPU being throttled, but that's not the case.
Second job is much much slower with the same job (eg. 1st job averages 65-70 fps, 2nd job starts and averages 11 fps). I'll try some of your other suggestions and post back here, but I'm pretty sure it's a problem with the video encoding part. |
| Posted by: dloneranger Aug 9 2012, 05:39 PM |
| There's no encoding going on if you've set the compressor to uncompressed, and run a video analysis pass All that's happening then, is reading from the disk, decoding the video and then virtualdub throws the decoded frame away So I'd think that kind of narrows it down to decoding or temperature Are they all flv's or does it happen with every type of file Does it happen with two really short clips? eg 30 seconds long If it does then heats not a problem as it shouldn't have time to get all that hot Just out of curiosity, what temps did you get? Here I go from 28C at idle to 40C at full load (And this pc is normally encoding 24/7) One other thing, and I'm sorry if it sounds like a dumb question, but it'd be silly not to double check The first and second videos are not first-pass and second-pass in the codecs settings are they? |
| Posted by: 4N4 Aug 9 2012, 06:17 PM |
| It's not just the second job that can be slow - on my latest test, the first job started fast (between 60-85 fps), but after 34-35 minutes (81% complete) it suddenly drops to about 11-13 fps and stays there for the rest of the encode. However, a previous test with the same job stayed between 60-85 fps for the entire encode. The latest test is with an .flv (AVC1 640x356x24bits + AAC Stereo 44100 Hz) I will try encoding to a different video (and audio, if necessary) format to see if that changes anything. |
| Posted by: 4N4 Aug 9 2012, 06:23 PM |
| I accidentally quoted your entire post rather than just the bits I had tested - fixed now. I haven't tried the uncompressed + video analysis yet, but I'll do that next. Max temps I got was 75C, but this is overclocked CPU but that's within expectations as it's overclocked, and I verified via CPUID that it wasn't throttled. Re. jobs, they both use exactly the same processing settings, although as I've just mentioned in my previous post, I have now noticed a slow-down near the end of the first job. |
| Posted by: dloneranger Aug 9 2012, 06:34 PM |
| That really does sound like a heat problem, there's no reason it should slow down in the middle of an encode that I can think of apart from that You could check the core speeds with cpuz http://www.cpuid.com/softwares/cpu-z/versions-history.html It shows the core speed and multiplier in real time If these change (more than a few mhz either way) if would be a sign of throttling Although, there's some reports of intel chips slowing themselves down in a way that doesn't show up like that |
| Posted by: 4N4 Aug 9 2012, 06:57 PM |
| I meant to say CPUZ rather than CPUID - it didn't show any sign of throttling during the encode. Plus, if I let the CPU cool down a bit before running the same job again, the second job will usually start encoding at ~12 fps, so I'm pretty sure throttling is not the issue. Two video analysis jobs both completed in the same time with average of 250 fps. I will now try encoding using different video codec. |
| Posted by: 4N4 Aug 10 2012, 01:14 AM |
| It looks like the XviD encoder is the problem as with a quick test using ffdshow's FFV1 encoder I got approx. 55 fps throughout job 1 and 55 fps at the start of job 2 (I only let it run for a short time). I will try with DivX next. |
| Posted by: 4N4 Aug 10 2012, 02:16 AM |
| No problems using DivX either. XviD seems to be the problem. I guess there is a bug in XviD which results in it losing performance for some reason. I may try with some older builds of XviD to see if they also suffer from this problem. |
| Posted by: dloneranger Aug 10 2012, 09:44 AM |
| Glad you've found the problem I use the Jawor version that has some extra profiles and options (packed bitstream profiles etc etc) It's on this page, and I haven't had any speed problems with it http://jawormat.republika.pl/xvid.html There is one 'gotcha' about changing versions of codecs The data that's saved in the jobs list for the codecs settings is not always loadable by a different version of a the same codec So it's best to swap versions after finishing or clearing your jobs list |
| Posted by: 4N4 Aug 10 2012, 11:58 PM |
| I was already using Jawod's 1.3.2 build which was exhibitng the problem. However, I've just switched to Jawod's 1.2.2 build and in my first test no slowdown occurred, with both jobs completing in the same time. |
| Posted by: 4N4 Aug 11 2012, 06:59 PM |
| I spoke too soon, as with the second test the first job slowed down near the end. I'm going to play around with the various XviD settings to see if one of them triggers this problem. |
| Posted by: -vdub- Aug 12 2012, 11:26 PM | ||
Did you try also with the official xvid v1.3.2 and the previous version is the problem with those also ? |