|
|
| Craig Hall |
 Posted: Oct 20 2002, 07:55 PM |
 |
|
Unregistered
|
Hi!
My First Post, just allow me to say that VDub Rocks and I'm amazed it's not a commercial product.
I do a lot of Encoding Jobs using XVid and Divx 5, and they take a bloody long time! (Even on my AthlonXP 1700  )
In 3D Computer Graphics (My Profession), there exists a technique that uses the CPU power of many machines to make a rendered frame faster.
A Server controls the Jobs and Sends them out to the Clients on the Renderfarm, and they are sent back when they are done (usually over a 100Mbps LAN / Ethernet)
Is there anyway to do this in Vdub?
In Rendering, a frame it's self can be split into several jobs, I am willing to bet you cant use more than one computer per job, but I would like to know whether there is an automated way to assign encoding jobs to different computers on a LAN.
I am at University all day and so are my housemates, so if I set the jobs up in the morning while everyone doesnt need their computers it will be done when we get in later.
I'm guessing that if any part of VDub can do this it will be the frameserver.
Ideas?
Regards
Craig. |
|
| Neo Neko |
| Posted: Oct 21 2002, 05:02 AM |
|
|
VirtualdubMod Alpha tester
Group: Vdubmod Alpha Testing Team
Posts: 474
Member No.: 24
Joined: 11-July 02
|
This sort of thing has been discussed several times in several places. There are lots of problems with this when it comes to video compression. It is possible, and has been done with multiple CPUs on the same system. The trick is to get this synchronised across a network. And with MPEG based codecs such a thing would be self defeating and quite possibly slower. With pretty much any codec frames have to be encoded linearly unlike 3d images. Each frame depends upon the frame immediatly preceeding itself and possibly more. So besides transmitting just the frame to be compressed you will also have to transmit the frame that cam before etc. This would congest the network and slow the process down quite a bit.
Now a job could be broken up into segments. But there are also problems with this. The start of each segment would have to be some sort of key frame. The more segments the more keyframes the larger the file. This can greatly decrease the efficiency of the codec used. Although it could be faster. That is why you don't see encoder farms as opposed to render farms. |
|
| porcazzo |
| Posted: Oct 22 2002, 10:58 PM |
|
|
Unregistered
|
i know this could sound stupid, but the easiest and best (and only) way to distribute the video rendering process is
1)to split each uncompressed video in 2 or 3 parts
2) let clients access the source avi from the server
3) join the results
in this way you have no excess of keyframes
 |
|
| Neo Neko |
| Posted: Oct 22 2002, 11:23 PM |
|
|
VirtualdubMod Alpha tester
Group: Vdubmod Alpha Testing Team
Posts: 474
Member No.: 24
Joined: 11-July 02
|
| QUOTE (porcazzo @ Oct 22 2002, 04:58 PM) | i know this could sound stupid, but the easiest and best (and only) way to distribute the video rendering process is
1)to split each uncompressed video in 2 or 3 parts
2) let clients access the source avi from the server
3) join the results
in this way you have no excess of keyframes
|
But because the process must be linear the encoding job will not be that efficient and you will still be limited by your slowest PC. The more PCs you use the faster it goes but the less efficient it is. So as I said before it would work. Not well. And only up to a point. Nowhere near what you would expect coming from a 3d render farm.
Here is a more viable option for 2-pass though. Run the first stats pass on a single machine(still slow, I know). The stats could be used to devide the video by where the codec would place keyframes. Each one of those segments could then be fed to a machine along with it's stats segment. It would be slightly faster and retain all the efficiency ince it already knows all it needs to know about the video. |
|
| paulie |
| Posted: Oct 23 2002, 08:03 AM |
|
|
Unregistered
|
Hello.
------------------------------------------------------------------------------
Maybe parallel processing over LAN can be done, in some special cases.
It could be usefull at least for:
- applying filters (well, not all types of filters, maybe resizing)
- rendering to final video using MJPEG codecs (every frame is a keyframe)
In this case parallel processing is possible, althoug difficult to code.
------------------------------------------------------------------------------
Say we have 1 server (to store source video, synchronize "client stations"
and put final frames together to final video.)
Source Video can be streamed from server to clients, or stored locally on each client, depends on size (maybe we have large HDDs on clients and slow network)
server would do all frameserving, just like virtualdub's frameserving,
but every frame will be rendered (and maybe compressed) by other client.
------------------------------------------------------------------------------
> Now a job could be broken up into segments.
> But there are also problems with this.
> The start of each segment would have to be some sort of key frame
What about this:
in the beginning source file is scanned for keyframes, then the job can be broken
into segments by detected keyframes, and any client will do his segment.
After segment is completed, client then waits for server to download data and assign another segment.
------------------------------------------------------------------------------
I know i have VERY VERY simplified this problem, but may be someone has some ideas...
Paulie.
P.S.: English is not my native language... so excuse me please... :-)
|
|
| Neo Neko |
| Posted: Oct 24 2002, 05:07 AM |
|
|
VirtualdubMod Alpha tester
Group: Vdubmod Alpha Testing Team
Posts: 474
Member No.: 24
Joined: 11-July 02
|
| QUOTE (paulie @ Oct 23 2002, 02:03 AM) | Hello.
------------------------------------------------------------------------------
Maybe parallel processing over LAN can be done, in some special cases.
It could be usefull at least for:
- applying filters (well, not all types of filters, maybe resizing)
- rendering to final video using MJPEG codecs (every frame is a keyframe)
In this case parallel processing is possible, althoug difficult to code. |
Yes that should be quite possible. I had the same idea. But since it does not really relate to just speeding up simple encoding in general I did not mention it. But you are correct.
| QUOTE (paulie @ Oct 23 2002, 02:03 AM) | Hello.
------------------------------------------------------------------------------
Say we have 1 server (to store source video, synchronize "client stations"
and put final frames together to final video.)
Source Video can be streamed from server to clients, or stored locally on each client, depends on size (maybe we have large HDDs on clients and slow network)
server would do all frameserving, just like virtualdub's frameserving,
but every frame will be rendered (and maybe compressed) by other client. |
You have to be careful there. Codecs often require past and future frames with which to encode and that makes this networked frame serving scheem infinantly more tricky.
| QUOTE (paulie @ Oct 23 2002, 02:03 AM) | Hello.
------------------------------------------------------------------------------
> Now a job could be broken up into segments.
> But there are also problems with this.
> The start of each segment would have to be some sort of key frame
What about this:
in the beginning source file is scanned for keyframes, then the job can be broken into segments by detected keyframes, and any client will do his segment. After segment is completed, client then waits for server to download data and assign another segment. |
The key is the keyframes in the encoded file and not the file being encoded. Where they keyframes are in the source file is of marginal significance. But the dificulty I was reffering to was the encoding of keyframes in a file only where nessecary. Breaking a file into segments inserts keyframes at the begining of each segment chewing up valuable bandwidth that would otherwise not likely be used.
|
|
| paulie |
| Posted: Oct 24 2002, 10:20 AM |
|
|
Unregistered
|
| QUOTE | | The key is the keyframes in the encoded file and not the file being encoded. Where they keyframes are in the source file is of marginal significance. |
| QUOTE | | Breaking a file into segments inserts keyframes at the begining of each segment chewing up valuable bandwidth |
I mean, if source file is split to segments BY KEYFRAMES (every segment begins with an keyframe), then no new keyframes are inserted.
If no new keyframes are produced during encoding (no scene change detection ) then keyframe count of all segments together were the same as in the source file.
I think encoder doesn't need to know frames before first frame of segment (keyframe).
Paulie |
|
| Neo Neko |
| Posted: Oct 24 2002, 06:49 PM |
|
|
VirtualdubMod Alpha tester
Group: Vdubmod Alpha Testing Team
Posts: 474
Member No.: 24
Joined: 11-July 02
|
| QUOTE (paulie @ Oct 24 2002, 04:20 AM) | | QUOTE | | The key is the keyframes in the encoded file and not the file being encoded. Where they keyframes are in the source file is of marginal significance. |
| QUOTE | | Breaking a file into segments inserts keyframes at the begining of each segment chewing up valuable bandwidth |
I mean, if source file is split to segments BY KEYFRAMES (every segment begins with an keyframe), then no new keyframes are inserted.
If no new keyframes are produced during encoding (no scene change detection ) then keyframe count of all segments together were the same as in the source file.
I think encoder doesn't need to know frames before first frame of segment (keyframe).
Paulie |
Codecs put keyframes in different spots. My MJPEG or HuffYUV capture are all keyframes. On the other had most of my MPEG4 encodes have a keyframe about once every 15 frames with no more than 300 frames between keyframes. Even encoding from MPEG4 to MPEG4 will not always put keyframes in the same spot. And therein is the big trick. Where and how would you break up the original file so you don't incur any extra keyframes in the finished file is what is important. Simply breaking the file into segments by keyframes will not cover all the angles. |
|
| paulie |
| Posted: Oct 25 2002, 06:59 AM |
|
|
Unregistered
|
Is there any way to disable automatic KF insertion in codec during encoding ? I mean, if this feature is not accesible through codec settings.
Are there some tutorials on the Internet on wfv and wdm programming ? I think I should learn more on this, before I start to ask more stupid questions  |
|
| Neo Neko |
| Posted: Oct 25 2002, 07:10 PM |
|
|
VirtualdubMod Alpha tester
Group: Vdubmod Alpha Testing Team
Posts: 474
Member No.: 24
Joined: 11-July 02
|
| QUOTE (paulie @ Oct 25 2002, 12:59 AM) | Is there any way to disable automatic KF insertion in codec during encoding ? I mean, if this feature is not accesible through codec settings.
Are there some tutorials on the Internet on wfv and wdm programming ? I think I should learn more on this, before I start to ask more stupid questions |
You don't want to disable the keyframe insertion in a codec. It is very important. All video segments have to start with a key frame. Whether or not that frame needs to be a key frame.
As for VFW and WDM learn them if you wish. But you are not far off the mark here.  This is more a problem with video codecs themselves. |
|
| Christian |
| Posted: Nov 13 2002, 10:03 AM |
|
|
Unregistered
|
Hi Craig,
have a look at Vidomi, afaik this program could do distributed encoding via LAN.
Christian
|
|
| Sarreq Teryx |
| Posted: Nov 13 2002, 11:31 AM |
|
|
VirtualdubMod Alpha tester
Group: Vdubmod Alpha Testing Team
Posts: 175
Member No.: 41
Joined: 16-July 02
|
maybe the best way to use clusters with mpeg or wavelet (or anything else using non-keyframes) codecs, (barring xvid's filesize predicator) may be concurrent 2-pass encoding. have one system doing the first pass, then when done enough of a chunk of data, pass it on to another system to do the second pass, and while that's happening have the first do another chunk and pass it on, and before all that, as was stated before, have a third system do the filtering, and then store it all on yet another system. I'm guessing encoding itself would be limited to a 2 system cluster, but there might be a way to use more that no one's thought of yet. possibly doing multiprocessor idct or wavelet calculations across the cluster, but I could just be being silly too.
however it might be accomplished, I'd think there'd have to be 1 system controlling everything, reassembling and distributing the data needed as neccessary, keeping the flow going properly.
--------------------
And as I walk through the Valley of the Shadow of Death, Lord, thy balls and shaft, they comfort me, you annoint my head with oil, some salt, a dash of pepper, a sprigg of parsley......Lord?............Lord??? What dost thou intend to do with that fork??? |
|
| Thomas-Matern |
| Posted: Nov 22 2002, 05:36 PM |
|
|
Unregistered
|
Hi
| QUOTE (porcazzo @ Oct 22 2002, 04:58 PM) | i know this could sound stupid, but the easiest and best (and only) way to distribute the video rendering process is
1)to split each uncompressed video in 2 or 3 parts
2) let clients access the source avi from the server
3) join the results
|
I added exactly this feature to VD and Nandub some month ago.
Please have a look at
http://www.matern-parkett.de/dub
There is no english documentation but many pictures ;-)
If *YOU* are interested in translating the homepage - please contact
me via email.
Thx, bye
Thomas
|
|
| jcsston |
| Posted: Nov 23 2002, 01:54 AM |
|
|
Matroska Dev
Group: Moderators
Posts: 553
Member No.: 652
Joined: 3-November 02
|
I tried this and I got the NanDub version to work with my system and an K6 300Mhz.
Cool
The VDub comes up with an error on the slave 'FastWrite Stream Error'
Well I'll try adding the Pentium 166Mhz for 3 way processing.
--------------------
Use the Matroska file format |
|
| Thomas-Matern |
| Posted: Nov 23 2002, 02:30 PM |
|
|
Unregistered
|
| QUOTE (jcsston @ Nov 22 2002, 07:54 PM) | The VDub comes up with an error on the slave 'FastWrite Stream Error'
|
Do you use VDub with:
- a 2-pass codec (DivX-4/5)
- the "direct-stream-copy" option?
The 1st pass is a problem because all hosts are writing to the same stats-file.
Nandub has not such a problem, because it handles the statsfile itself, while in VDub the codec handles the statsfile...
Bye
Thomas
|
|
