Unofficial VirtualDub Support Forums -> Virtualdub 1.9.x Test Thread

Welcome Guest ( Log In | Register )

Unofficial VirtualDub Support ForumsťVirtualDubťTesting / Bug Reports

Important

The forums will be closing permanently the weekend of March 15th. Please see the notice in the announcements forum for details.

Pages: (16) [1] 2 3 ... Last » ( Go to first unread post )

Virtualdub 1.9.x Test Thread, Very experimental build

« Next Oldest | Next Newest »

Track this topic | Email this topic | Print this topic

phaeron

Posted: Aug 3 2009, 12:49 AM

Virtualdub Developer

Group: Administrator
Posts: 7773
Member No.: 61
Joined: 30-July 02

I'm going to start a new test thread in parallel to the 1.9.9 thread that's going on:

http://www.virtualdub.org/beta/VirtualDub-....9.X-test32.zip
http://www.virtualdub.org/beta/VirtualDub-...est32-AMD64.zip
http://www.virtualdub.org/beta/VirtualDub-...X-test32-src.7z

This is from a parallel branch where I'm doing feature work and more experimental stuff in preparation for 1.9.6+. As usual, I can't provide any guarantees about what's going in or when it'll be done, but I can say that I'm starting to go through the feature backlog. Do not re-request features that you have already asked for, because they're already in the backlog. I'm still running down the list and figuring out which ones are easier to implement.

The main reason I'm starting up this thread, however, is that there is a very experimental change in this branch, namely the ability to run video filters multithreaded. Unlike other recent changes to the video filter system, such as 3D acceleration, filters do not need to be rewritten to take advantage of this feature -- the VirtualDub filter API is such that existing filters can be run in parallel. Others have in fact already implemented this before, but I'm a bit late in doing so because it wasn't until after the major filter system rewrites I did for multi-sourcing and 3D acceleration that this was possible in VirtualDub's implementation. Notes on the current implementation:

Threaded filtering is not supposed to affect the actual video frames output in any way.
You have to enable threaded filtering in Options > Preferences > Threading.
Only different filter instances can be run in parallel. In other words, if you only have one "resize" filter entry, VirtualDub can only use one thread to run the video filter chain because it can only run that filter instance on one frame at a time. Two "resize" filters in the chain, however, can run in parallel, because they're separate instances even though they're the same filter type.
Running the filter chain requires more memory because more frames have to be in flight to take advantage of the threading. This is not likely to be a problem unless you are working with HD size frames or are trying to run with ridiculously high parallelism (say, 8 cores). If you're working with 1080p frames on a 4-core system with half a dozen filters, however, you're likely to see VirtualDub.exe hit the 100-200M range. The "process ahead" setting allows you to tweak how many output frames the rendering engine allows in the queue at a time. Lowering it reduces memory usage, while raising it reduces the chance of bubbles in the pipeline.
There is a difference between disabling multithreaded filtering and setting it to "1 thread." If it is set to one thread, video filters can run in parallel with the video codecs.
Color format conversion and VDXA (3D acceleration) currently don't take advantage of threaded filtering. I expect to improve these situations before release.
Slow video codecs can affect parallelism. The reason is that the way this is implemented, only the main processing thread can schedule frames, even though all of the threads in the thread pool can execute them. This means that you may have to increase the process ahead value or turn on multithreaded compression to improve parallelism.
Some video filters may be incompatible with the new threading, if they are doing something that violates the threading rules for VirtualDub filters. None of the internal video filters should have this problem, but I'm interested in knowing if there are filters out there that crash.

Change list for this build:

(test-32) Fixed filename detection regression.
(test-31) Incorporated mouse wheel fixes for filter preview dialog and curve control.
(test-31) Fixed ASF pseudo-handler so that it doesn't override filename-only detection.
(test-30) Fixed focus bug in filter list dialog.
(test-29) Synced to 1.9.10 final.
(test-27) Fixed audio display regression.
(test-26) Fix for scene detector crash; integrated latest fixes from stable branch.
(test-25) Added sorting to external encoder dialog.
(test-24) Fix for duplicated capture device items.
(test-23) Multiplexer errors are reported properly again.
(test-23) Fixed I/O error at end when outputting raw audio.
(test-22) Added %(samplingratekhz) macro.
(test-22) External encoder sets can now only have an audio encoder.
(test-22) Added setting to force deletion of temporary file prior to running external encoder.
(test-22) %(tempvideofile) and %(tempaudiofile) can now be forced to %(outputfile) in the encoder set when there is no multiplexer.
(test-22) Fixed resizing regression in D3DFX display driver.
(test-22) Input format setting in external encoder profile now saves properly.
(test-21) Added %(hostdir), %(programdir), and %(systemdir) macros for external encoders.
(test-21) Fixed crash when exporting without an audio encoder.
(test-20) Fixed "Foo" in capture driver list.
(test-20) Fixed incorrect biSizeImage field when producing I8 format.
(test-19) Fixed frame mapping in normal/fast recompress mode when filters are present.
(test-18) Threaded video compression fix, rev 2.
(test-18) List scrolling fix in video compressor dialog.
(test-17) Fix for some of the B-frame threaded compression issues.
(test-17) "Run video analysis pass" now runs the video codec again.
(test-16) Synchronized to 1.9.9/stable.
(test-16) 2+ thread support for key frame only video compression.
(test-16) IVTC frame drop mode.
(test-14) Filters: Added deblurring mode and preview to IVTC filter.
(test-13) Added copy source/output frame number command.
(test-11) Added import/export for external encoder profiles.
(test-10) "Process partial output" setting saves.
(test-9) Win98 and NT4.0 fixes for external encoder encoding.
(test-2) Render: Fixed hang at end with B-frames and with smart rendering.
(test-1) Holding the Ctrl key during drag-and-drop appends instead of replaces.
(test-1) TGA RLE compression can now be disabled.
(test-1) "Scan for errors" command is now scriptable (VirtualDub.video.ScanForErrors).
(test-1) Save Animated GIF command is now scriptable (VirtualDub.video.SaveAnimatedGIF).
(test-1) Render: Added option to show the status window for batch operations.
(test-1) Filters: Added multithreading support.
(test-1) JobControl: Auto-shutdown now works over remote desktop and records a planned shutdown on server versions of Windows.

This post has been edited by phaeron on Oct 30 2010, 11:06 PM

jpsdr

Posted: Aug 4 2009, 08:44 PM

Advanced Member

Group: Members
Posts: 335
Member No.: 20490
Joined: 23-December 06

Hello.

Test with some of my own filters and internal :
Input : An avisynth script of DGDecode, output YV12.
Color deapth : In YV12, out YUY2.
Thread : 1, filter both on auto.
My own IVTC automatic IVTC, produce actualy an output of IVTC 2.
My own YV12 -> YUY2 convert.
IVTC internal
My own saturation.
resize internal
output huffyuv.

VDub is at 0% CPU, 161Mo of mermory used, and 100% done but
still in progress since half an hour !

To try to eventualy reproduce it, i'll send you by email the job list
and all the necessary.

BTW, do you intend to make IVTC and Deinterlace work on YV12 ?
If no, where can i find the yadif algorithm ?

EDIT : I was expecting more than 4-5 threads in the task manager...

pintcat

Posted: Aug 4 2009, 10:38 PM

Advanced Member

Group: Members
Posts: 142
Member No.: 18182
Joined: 19-February 06

Same for me. When VDub reaches 100% the job isn't done. It just keeps going and going... (tested with DV video 720x576 4:3, resized to 720x540 1:1; video compressed with XviD 2-pass, but the 1st pass could not be finished).

phaeron

Posted: Aug 5 2009, 04:52 AM

Virtualdub Developer

Group: Administrator
Posts: 7773
Member No.: 61
Joined: 30-July 02

I did say it was very experimental. Anyone actually seeing any speedup?

And as for threading, by default VirtualDub clamps the thread count to 4 when set to Auto. This is because it's pretty hard to actually get any real parallelism above that level unless you have an insane set of filters, and it could consume a tremendous amount of memory if it tried to thread too far.

jpsdr

Posted: Aug 5 2009, 07:19 AM

Advanced Member

Group: Members
Posts: 335
Member No.: 20490
Joined: 23-December 06

Hello.

I know it was experimental, so i've experiment it.
I thought you wanted problem to be reported.

But, as i'm having an i7 965, i'll try this evening the following :
Same chain filter (around 6-7 filters) between 1.9.5-test4 and 1.9.X with threads put to 8, as it's
the number of "core" i'm having.

I'll tell you if i see speedup.
Nevertheless, without the ending time (as i'm doing other stuff on my normal PC while the
dedicated video PC is running), it may be a little hard...

jpsdr

Posted: Aug 5 2009, 07:45 PM

Advanced Member

Group: Members
Posts: 335
Member No.: 20490
Joined: 23-December 06

Ok, test done.
I've done the folowing :
Create a batch process of around 6 filters with 1.9.X. version.
(The one i've send to you).
Copy the job file into the directory of the 1.9.5-test4 version.
This way, in theory, as there was no filter using/producing random data,
each version should do exactly the same thing, and should produce
exactly the same file.

Version 1.9.X configured with threads at "1", filter thread at "8",
and frame on "auto".
Version 1.9.5 configured with threads at "1".

Version 1.9.5 : Process time around 14mn.
Version 1.9.X : Process time around 9-10mn.

Version 1.9.X produce a 33205 frames file, 1.9.5 produce
a 33206 frames file. I assume the difference of 1 is because
the 1.9.X is stalled on the last frame, and don't finalise the batch.

BUT....!!!!!! I've discover something absolutely abnormal.
The file size produced by the 1.9.5 is 16Go, and the file
size produced by the 1.9.X is 8Go !!!!!!!
I realy doubt the last frame has 8Go size.
And, for a 720x576 huffyuv file of 33206 frame, the expected size
is around 8Go.
So, i don't know wich it is, but there is a problem in the 1.9.5,
there isn't in the 1.9.X...
In the package i've send to you, i think you have all you need to try
to investigate/reproduce the problem.

phaeron

Posted: Aug 6 2009, 04:47 AM

Virtualdub Developer

Group: Administrator
Posts: 7773
Member No.: 61
Joined: 30-July 02

I fixed a couple of bugs that caused hangs if smart rendering or B-frames were enabled. Test-2 is now up.

I did a test with a subset of the filters that I could run from your example, and I got bit exact video between 1.9.5 and 1.9.X. There were several major bugs in the new video processing code in 1.9.X test-1, though, so who knows. About the size difference, the only thing I could think of is that something really bad happened like the entire half of the video was blanked out.

Incidentally, you have good taste in anime (Crest of the Stars).

jpsdr

Posted: Aug 6 2009, 08:38 AM

Advanced Member

Group: Members
Posts: 335
Member No.: 20490
Joined: 23-December 06

Yes, Crest/Banner has been a great discovery form me.

I've try to make my own fansub DVD, unfortunately,
even on the japanese DVDs, video is a real mess
(in therm of telecine), so, i'll not be able to do what
i'm usualy do : IVTC and accelerate to 25fps, to have
like a pure PAL telecine. Of course, the only interest
on that it's to NOT have to deinterlace, and having
in final a true progressive picture, and finaly a better
result than a standard NTSC-PAL transcoding, wich
produce interlaced frames. And of course, upscaling
480->576 is better when done on progressive frames.

jpsdr

Posted: Aug 6 2009, 06:14 PM

Advanced Member

Group: Members
Posts: 335
Member No.: 20490
Joined: 23-December 06

Ok, tested the test2 version, with same job.

1.9.X : Process time around 9mn, and end correctly.
1.9.5 : Process time around 12mn.

Both produced a 8.01Go file, wich is the size expected, don't know what
happened yesterday.
What is the "Video filter process-ahead" option for ? (Was on auto during test).
What does it do ? If i increase it, will it increase the CPU usage ?
I mean, during my test i'm only around 25% CPU usage, so i thought maybe
there is way to increase CPU usage, and speed up things.

Keep the good work.

phaeron

Posted: Aug 7 2009, 06:10 AM

Virtualdub Developer

Group: Administrator
Posts: 7773
Member No.: 61
Joined: 30-July 02

The process ahead option is needed since having more than one frame in flight is the only way to run most chains in parallel.

Let's say that you had a filter chain that contained filters A, B, and C. Each filter is a 1:1 filter like blur, where one frame in produces one frame out. If I request frame 2 from C, it requests frame 2 from B, which requests frame 2 from A. Filters A, B, and C then process frame 2, in that order. There is no room for parallelism here, because only one filter can run on the frame as it goes through the chain.

If you have three threads and three frames, though, then you can run the filters in parallel on different frames. For instance, you could have C working on 3, B working on 4, and A working on 5. Now the filters can work like an assembly line on frames flowing through. This is how VirtualDub runs the filter chain on multiple threads without requiring the filters to be rewritten for multithreading.

The moral of the story is that you will typically want the process-ahead setting set to the number of threads in use. This is what the Auto setting does.

Note that this only helps if you have filters that are somewhat balanced. If you have a bunch of quick filters and one really, really slow filter, you're going to be completely bottlenecked by the slow filter. The only way to get around that would be to run that one filter in parallel, and that's not something that I support right now (and isn't possible for all filter algorithms).

jpsdr

Posted: Aug 7 2009, 12:22 PM

Advanced Member

Group: Members
Posts: 335
Member No.: 20490
Joined: 23-December 06

Found out something.
Put display of input and output.
Put swap input/output.
Choose color depth input in high deep color, chek with the 3 last.
color depth output RGB24.
Put a simple filter (Brightness).
Pane display is messed up.

phaeron

Posted: Aug 8 2009, 04:26 AM

Virtualdub Developer

Group: Administrator
Posts: 7773
Member No.: 61
Joined: 30-July 02

Can't reproduce. Are all of those steps needed? I can't imagine that the swap input/output makes a difference. Also, what source are you using? There aren't many codecs that support those last three formats, so you'd have to be using an Avisynth script or some other case where VirtualDub sees an uncompressed input (in this case, it will allow any conversion path).