|
|
| Serge |
| Posted: Jul 18 2010, 11:12 PM |
 |
|
Member
Group: Members
Posts: 12
Member No.: 28048
Joined: 18-July 10
|
Hello Everybody,
I implemented a new feature for VirtualDub that can be merged into the source tree now. It is available as a source tree patch, you can download it from here -
http://www.codeproject.com/KB/miscctrl/dyn...patch-hdfix.zip
I also built VirtualDub binaries with the patch -
http://www.codeproject.com/KB/miscctrl/dyn...tched-hdfix.zip
Here is a testing example -
http://strakhov.longmusic.com/work/Roof.zip
UPD: Please make sure you have "Video->Full processing mode" selected in your VirtualDub menu. No other options (Direct stream copy / Fast recompress / Normal recompress ) should be used.
Would be great to hear your opinion. Below are some technical details (you can find them in the README file inside of the patch source archive).
LICENSE AND WARRANTIES
======================
The patch is distributed under the GNU general public license (just like VirtualDub).
In case, if it is going to be merged into the official source tree of VirtualDub,
some personal credits are expected to appear in the About box. Otherwise, use it
at your own risk with no warranty of any kind.
STORY
=====
One day I came across a video (http://www.youtube.com/watch?v=A23SPewFZhU)
that was a dub of an old 8mm film we used to make as kids. The dub was made
unprofessionally with no special equipment or even a tripod, so it suffered from a
number of problems. The worst was sporadic non-compensated motion of the dubbed
screen area. I tried to find software that would allow me to crop the moving areas
dynamically but had no luck. There was no free software with such a feature, and all
commercial software was way too expensive. Then I came to an idea of writing such
software by myself. As a base, I chose VirtualDub (http://www.virtualdub.org/)
because it already provided some static cropping functionality. All I needed to do
was to make it dynamic, e.g. with placement of the cropped area defined as a
function of time. I decided to make it as flexible as possible, so one could edit,
save & restore the dynamic cropping settings, and to design it as an extendable
feature.
It was done and here is the first result -
http://www.youtube.com/watch?v=qlUCC5bH5kc
DESIGN
======
The "work horse" of the solution is a new IVDDynClippingStorage interface defined
in ClippingControl.h :
class VDINTERFACE IVDDynClippingStorage : public vdrefcounted<IVDRefCount> {
public:
// Initialize clipping map
virtual void Init(int sourceW, int sourceH) = 0;
// Get source width
virtual int GetWidth() = 0;
// Get source height
virtual int GetHeight() = 0;
// Load clipping map from a file
virtual VDStringW Load(wchar_t *filename, HWND parent) = 0;
// Save clipping map to a file
virtual VDStringW Dump(wchar_t *filename, HWND parent) = 0;
// Display configuration interface
virtual void Configure(HWND parent) = 0;
// Set clipping bounds for position
virtual int SetClipBounds(sint64 pos, const vdrect32& r) = 0;
// Get clipping bounds for position
virtual int GetClipBounds(sint64 pos, vdrect32& r) = 0;
// Get position for clipping map index
virtual sint64 GetClipPos(int idx) = 0;
// Get clipping map size
virtual int GetTotalClipPos() = 0;
// Get the resulted crop size
virtual void GetCrop(vdrect32& r) = 0;
// Delete clipping map entry by index
virtual void DelClipBounds(int idx) = 0;
// Crop pixmap for given position
virtual bool DoCrop(VDPixmap& dst, sint64 pos, vdrect32& crop_area) = 0;
// Add/drop range to/from clipping map
virtual void ChangeTimeline(sint64 pos, sint64 nframes, bool add) = 0;
};
It defines a set of methods to manage the clipping map and to perform the actual
dynamic cropping. The VDDynClippingStorage class defined in ClippingControl.cpp
is implementing the interface. An instance of the class can be created per
FilterInstance (however, a need in having more than one filter with dynamic
cropping in the chain of filters is highly doubtful).
There are some changes made to the basic functionality of the cropping control
as well. They are not strictly related to dynamic cropping, but are very handy:
1) From now on you will be able not only stretch the cropped area by dragging
its sides and corners with left mouse button, but also move the complete
cropped area using the right mouse button! Try it and you will see how useful
it is for dynamic cropping when you have to go frame-by-frame adjusting the
cropped area placement.
2) A new "lock aspect ratio" checkbox allows the cropped area to have the aspect
ratio of the original video. It is suggested to always check it with dynamic
cropping enabled.
The following VirtualDub components are affected with the patch:
- Cropping control (ClippingControl.h/.cpp, FiltDlg.cpp, resources),
- Filter instance class (FilterInstance.h/.cpp),
- Saving (Job.cpp) and loading (Script.cpp) of the processing settings.
The following project files are modified:
src\virtualdub\h\clippingcontrol.h
src\virtualdub\h\filterinstance.h
src\virtualdub\res\resource.h
src\virtualdub\res\virtualdub.rc
src\virtualdub\source\clippingcontrol.cpp
src\virtualdub\source\filtdlg.cpp
src\virtualdub\source\filterinstance.cpp
src\virtualdub\source\job.cpp
src\virtualdub\source\script.cpp
BUILDING
========
You should patch the source tree of VirtualDub and re-build it.
There are two ways of patching the source tree, you can either
1) unzip the archive into the root of VirtualDub 1.9.9 source tree overwriting
its files, or
2) you can use the provided dyncropping.patch file that contains diff info
produced with GNU diff (http://gnuwin32.sourceforge.net/). Run it with the Win32
version of patch from the root of VirtualDub source tree as follows -
C:\VirtualDub-1.9.9-src> patch -p1 -i dyncropping.patch
, this may work well with versions of VirtualDub other than 1.9.9 too.
After that run the building process as usual.
RUNNING
=======
Load a video, add any video filter, select "Cropping..." and check the "Dynamic
cropping" checkbox in the "Filter input cropping" control. Check also "Lock
aspect ratio" for better results. Walk the video frame-by-frame inside of the
cropping control and adjust the sizes and position of the cropped area
accordingly to your needs. All position and size changes will be saved in the
map for rendering. If you rewind the video back to the beginning and go again
frame-by-frame you will notice how the cropped area is moving to follow your
adjustments! You can click "Options..." and select the desired settings. Once
done, click "OK" to close "Filter input cropping" and once again "OK" to close
"Filters". Run "Preview filtered..." and watch what is happening on the left
and right UI panels. Use "Save as AVI..." to save the final rendering.
You can also use the provided testing sample Roof.zip to evaluate the
functionality of the patch. Just open the Roof.avi file, load its processing
settings from Roof.vcf. It will load the dynamic cropping map from Roof.dcrp.
Run "Preview filtered..." and watch what is happening on the left and right UI
panels. Open "Cropping..." of the Warp Sharp filter and play with the new
dynamic cropping control. Walk the clipping map vertex-by-vertex with "<" and
">" buttons. If you go frame-by-frame with the position control you will
notice how the cropped area is moving around. You can adjust it at any position
modifying its size and placement. All changes will be saved in the cropping map
and used for rendering.
CONCLUSION
==========
In general, this feature can be useful for manual "motion post-compensation" in
the following cases:
- Old videos made with camcorders with no hardware motion compensation
available,
- Videos made in complex motion situations (in car, boat, on the run),
- Videos made by kids or impaired people,
- "Noisy", dark, low-contrast videos where automatic algorithms usually fail or
produce bad results.
Also, it can be handy for some special video effects.
It is very basic, but it is doing what it is supposed to do. Here is a couple
of more things I wanted to share.
- Currently only linear interpolation between vertexes of the clipping map is
implemented. One can try to implement a cubic spline-based interpolation to
see if it improves the rendering results and reduces shaking. Also, one can
try to implement a real motion-compensation algorithm inside of
IVDDynClippingStorage::DoCrop that will cache and analyze frames.
- There is also a "no interpolation" mode available that can be used for videos
that contain sets of scenes in which static cropping is required - this avoids
cutting / cropping / gluing them piece-by-piece.
- With linear interpolation you usually achieve the best results with two
cropping adjustment passes.
- The resampling selection has three settings: original frame size, maximal
cropping size and custom. In any case, the result will be resampled to the
rounded sizes divisible by 16 for general compatibility with compression
algorithms. An attempt to force custom sizes to values non-divisible by 16
or bigger than those of the source will result in an error.
- There are two resampling algorithms built into VirtualDub - "bilinear" and
"nearest". The "bicubic" is not implemented in the core of VirtualDub but
instead as a plugin and therefore is not supported.
- It appears to be useful to have a line grid optionally appearing on the
cropped area when you slide it with the right mouse button (maybe it can
appear when you press and hold the right mouse button and then the left one).
It would allow to position the cropped area more precisely. I did not
implement it, but left it as an idea for others.
- The IVDDynClippingStorage::ChangeTimeline method is defined and implemented
in VDDynClippingStorage but it is not been used for anything - it can be a
future extension, just in case.
- All clipping map vertex indexes are 1-based. That is, what index you see on
the interface is what you have in the map.
ABOUT THE AUTHOR
================
I am a Software Engineer with 20+ years of experience in different fields of
programming. Presently I live in the Toronto area. |
|
| rjisinspired |
| Posted: Jul 19 2010, 12:54 AM |
|
|
Advanced Member
Group: Members
Posts: 1256
Member No.: 20008
Joined: 12-October 06
|
WOW! This is really neat!
I see the effect of the cropper in action in the output window though only by stepping in frame-by-frame. The video preview doesn't show it and it doesn't save out to the cropped version. i take it that this won't be the case once everything is all put together?
This gave me an idea: Couldn't this cropping action also work as a video stabilizer? What I mean that instead of motion vectors, have the initial crop area be set for the first frame, for the positioning of the crop area ,and then any movement would make the crop area move in relation to the motion? |
|
| Serge |
| Posted: Jul 19 2010, 01:41 AM |
|
|
Member
Group: Members
Posts: 12
Member No.: 28048
Joined: 18-July 10
|
| QUOTE (rjisinspired @ Jul 19 2010, 12:54 AM) | WOW! This is really neat!
I see the effect of the cropper in action in the output window though only by stepping in frame-by-frame. The video preview doesn't show it and it doesn't save out to the cropped version. i take it that this won't be the case once everything is all put together?
This gave me an idea: Couldn't this cropping action also work as a video stabilizer? What I mean that instead of motion vectors, have the initial crop area be set for the first frame, for the positioning of the crop area ,and then any movement would make the crop area move in relation to the motion? |
Thanks, rjisinspired.
"File->Preview filtered..." and "File->Save to AVI..." work for me though. "Preview input..." would not show anything in the output widow, maybe this is what you are using?
And it does work as a manual video stabilizer too, that was one of the ideas behind it.
|
|
| rjisinspired |
| Posted: Jul 19 2010, 03:52 AM |
|
|
Advanced Member
Group: Members
Posts: 1256
Member No.: 20008
Joined: 12-October 06
|
I can see the effect when Vdub is idle at any point in the clip or when seeking manually through the time-line. Using "preview filtered" didn't work for me.
I don't know why this isn't working on my end. I'm eager to see this thing in action! |
|
| Serge |
| Posted: Jul 19 2010, 05:50 AM |
|
|
Member
Group: Members
Posts: 12
Member No.: 28048
Joined: 18-July 10
|
| QUOTE (rjisinspired @ Jul 19 2010, 03:52 AM) | I can see the effect when Vdub is idle at any point in the clip or when seeking manually through the time-line. Using "preview filtered" didn't work for me.
I don't know why this isn't working on my end. I'm eager to see this thing in action! |
OK, it seems I could figure it out!
Please make sure you have "Video->Full processing mode" selected in your VirtualDub menu. No other options (Direct stream copy / Fast recompress / Normal recompress ) should be used. It will work right away.
|
|
| fredgiblet |
| Posted: Jul 19 2010, 06:10 AM |
|
|
Advanced Member
Group: Members
Posts: 356
Member No.: 22601
Joined: 13-December 07
|
| QUOTE (Serge @ Jul 18 2010, 11:12 PM) | | I implemented a new feature for VirtualDub that can be merged into the source tree now. |
Just an FYI, I'm fairly certain that phaeron doesn't do this (else I'd expect that fccHandler's stuff would have been merged long ago). |
|
| Jam One |
| Posted: Jul 19 2010, 09:04 AM |
|
|
Advanced Member
Group: Members
Posts: 580
Member No.: 25518
Joined: 8-May 09
|
| QUOTE | ... the result will be resampled to the
rounded sizes divisible by 16 for general compatibility with compression
algorithms. An attempt to force custom sizes to values non-divisible by 16
or bigger than those of the source will result in an error. |
Well, HD standard 1080 is not divisible by 16, you know...
The whole idea is fine, congratulations!
|
|
| Serge |
| Posted: Jul 20 2010, 05:18 AM |
|
|
Member
Group: Members
Posts: 12
Member No.: 28048
Joined: 18-July 10
|
| QUOTE (fredgiblet @ Jul 19 2010, 06:10 AM) | | QUOTE (Serge @ Jul 18 2010, 11:12 PM) | | I implemented a new feature for VirtualDub that can be merged into the source tree now. |
Just an FYI, I'm fairly certain that phaeron doesn't do this (else I'd expect that fccHandler's stuff would have been merged long ago). |
Is there any Perforce access for contributors? I mean, if this is an open source project, what is a point of opening them without giving people a chance to contribute? |
|
| phaeron |
| Posted: Jul 20 2010, 05:19 AM |
|
|
Virtualdub Developer
Group: Administrator
Posts: 7773
Member No.: 61
Joined: 30-July 02
|
fredgiblet is correct -- for various reasons I generally don't accept patches from third parties. Sorry about that. However, this is still an interesting patch.
The main question I have here is: how much of this could be done as a video filter, and what additional features to the filter API are needed to make that work well? From what I can tell, the actual cropping operation itself can be done, so it's the UI that's mainly in question.
I have a couple of reasons for wanting it to work that way. One reason is just to decouple it from the main code base, as I update at the rate of a slug. Another reason, though, is that the amount of logic that is in FilterInstance is already a problem, and I'm trying to pull out as much as I can. The cropping logic was simple enough to deal with early on, but once I added features like multithreading, frame buffer caching, YCbCr format support, etc. it quickly became a problem.
I haven't looked at the code, but there are a couple of potential filter API issues involved here, too, such as how this interacts with filters that request SSE aligned scanlines or in-place buffer allocation, both of which aren't very amenable to the crop rectangle changing on a per-frame basis.
The right-click drag, though, is something I think I'm going to have to steal.
|
|
| Serge |
| Posted: Jul 20 2010, 05:27 AM |
|
|
Member
Group: Members
Posts: 12
Member No.: 28048
Joined: 18-July 10
|
| QUOTE (Jam One @ Jul 19 2010, 09:04 AM) | | QUOTE | ... the result will be resampled to the
rounded sizes divisible by 16 for general compatibility with compression
algorithms. An attempt to force custom sizes to values non-divisible by 16
or bigger than those of the source will result in an error. |
Well, HD standard 1080 is not divisible by 16, you know...
The whole idea is fine, congratulations! |
Thanks!
MPEG2 HD is using 16x16 macroblocks, each of them is split on four 8x8 blocks, so that 1080 is divisible by 8. DivX in turn does not allow to split its 16x16 macroblocks. I guess, it is a point of discussion - whether to enforce 16- or 8-divisible size formats or just leave it free in this solution. |
|
| Serge |
| Posted: Jul 20 2010, 05:47 AM |
|
|
Member
Group: Members
Posts: 12
Member No.: 28048
Joined: 18-July 10
|
| QUOTE (phaeron @ Jul 20 2010, 05:19 AM) | fredgiblet is correct -- for various reasons I generally don't accept patches from third parties. Sorry about that. However, this is still an interesting patch.
The main question I have here is: how much of this could be done as a video filter, and what additional features to the filter API are needed to make that work well? From what I can tell, the actual cropping operation itself can be done, so it's the UI that's mainly in question.
I have a couple of reasons for wanting it to work that way. One reason is just to decouple it from the main code base, as I update at the rate of a slug. Another reason, though, is that the amount of logic that is in FilterInstance is already a problem, and I'm trying to pull out as much as I can. The cropping logic was simple enough to deal with early on, but once I added features like multithreading, frame buffer caching, YCbCr format support, etc. it quickly became a problem.
I haven't looked at the code, but there are a couple of potential filter API issues involved here, too, such as how this interacts with filters that request SSE aligned scanlines or in-place buffer allocation, both of which aren't very amenable to the crop rectangle changing on a per-frame basis.
The right-click drag, though, is something I think I'm going to have to steal. |
Thanks! I see.
This was implemented as a core patch mostly because of the existing cropping support tighten up with FilterInstance. One can possibly implement it entirely as a video filter. At least, the IVDDynClippingStorage implementation can be used there only with some minor adjustments, as it seems. Need to have another look at it maybe in a month.
Oh, BTW- I tried to upload my own avatar here, but it is not displayed for some reason. Any possibility to fix it? |
|
| rjisinspired |
| Posted: Jul 20 2010, 05:55 AM |
|
|
Advanced Member
Group: Members
Posts: 1256
Member No.: 20008
Joined: 12-October 06
|
I have the mode on full processing and it still isn't working.. The effect shows in the output pane when the seek line is idle anywhere in the clip. I am really confused. |
|
| rjisinspired |
| Posted: Jul 20 2010, 06:04 AM |
|
|
Advanced Member
Group: Members
Posts: 1256
Member No.: 20008
Joined: 12-October 06
|
Hold on. I just noticed the VCF file and I loaded this and it works.
I feel kind of stupid right now. I had, initially, manually went in to add null transform and dynamic crop then loaded the map file.
In your VCF you have warp sharp and cropping set for 400 X 304. This seems to be the only difference. Looks like I missed an extra step somewhere by doing things the other way.
|
|
| fredgiblet |
| Posted: Jul 20 2010, 06:36 AM |
|
|
Advanced Member
Group: Members
Posts: 356
Member No.: 22601
Joined: 13-December 07
|
"Is there any Perforce access for contributors?"
I don't know what that means, sorry.
"I mean, if this is an open source project, what is a point of opening them without giving people a chance to contribute?"
So that other people can make their own modifications. Before phaeron implements the plugins system FccHandler created VirtualDub-MPEG2, which had his MPEG-2 decoding patch in it (now with the plugin capabilities FccHandler just makes the plugins), there's a fork called VirtualDubMod which added tons of abilities that's sadly been dead for almost half a decade now, there was a version called nanDub which offered better compression with DivX 3.11 and I think I remember a version called ASFdub or something that was supposed to re-add support for ASF. |
|
| Serge |
| Posted: Jul 20 2010, 06:42 AM |
|
|
Member
Group: Members
Posts: 12
Member No.: 28048
Joined: 18-July 10
|
| QUOTE (rjisinspired @ Jul 20 2010, 06:04 AM) | Hold on. I just noticed the VCF file and I loaded this and it works.
I feel kind of stupid right now. I had, initially, manually went in to add null transform and dynamic crop then loaded the map file.
In your VCF you have warp sharp and cropping set for 400 X 304. This seems to be the only difference. Looks like I missed an extra step somewhere by doing things the other way. |
Yes, bingo! So, now you can try to create your own dynamic cropping if you load a video, add a filter (can be "null-transform"), go into "Cropping..." and enable Dynamic Cropping there. |
|
|
