|
|
| phaeron |
| Posted: Dec 23 2011, 07:37 AM |
 |
|
Virtualdub Developer
Group: Administrator
Posts: 7773
Member No.: 61
Joined: 30-July 02
|
http://virtualdub.org/downloads/VDPluginSDK-1.2.zip
- Synchronized to 1.10.1 headers
- Includes documentation for input plugin API V4, video filter API V16
This was a pain to make because I ported the whole thing over from my old HTML preprocessor to a newer XML/XSLT based one, and so I had to fix up all of the places where I had violated the XML spec... but at least I can use XPath and sorting now, I don't have to do a full build to preview a topic, and I made it check links so I fixed up a bunch of broken ones.
|
|
| fccHandler |
| Posted: Dec 24 2011, 04:39 AM |
|
|
Administrator n00b
Group: Moderators
Posts: 3961
Member No.: 280
Joined: 13-September 02
|
Thank you very much for this. Today I replaced the SDK headers in the WMV plugin and it was no problem at all. Now I just have to work on supporting all the new colorspaces...
I should mention one minor flaw in the documentation. On the page which defines the VDXInputDriverDefinition structure, kFlagNoOptions is described, but it is missing from the enum in the definition at the top of the page.
The date is "Copyright © 2007-2012 Avery Lee." So you live in the future now? Always thinking ahead. 
What is the meaning of the suffix "_FR" in the colorspace names? For example, kPixFormat_Y8_FR as opposed to kPixFormat_Y8.
I have another question that you can probably answer quicker than I can. IVDXStreamSourceV3 is great stuff and a lot of my plugins use it. I noticed this in the vdinputdriver.h comments:
| CODE | // V3 (1.8.0): VBR audio support.
// NOTE: Due to a bug, VirtualDub never properly announced this version.
// You must declare your input driver as V2 API compatible to work
// on 1.8.x and 1.9.x releases. |
But I thought IVDXStreamSourceV3 was added in VirtualDub version 1.8.8, and I've been stating that my plugins which need it require that version as a minimum. Am I misremembering?
--------------------
May the FOURCC be with you... |
|
| jpsdr |
| Posted: Dec 24 2011, 07:15 AM |
|
|
Advanced Member
Group: Members
Posts: 335
Member No.: 20490
Joined: 23-December 06
|
FR is for Full Range : 0-255 instead of Y 16-235 and Cb/Cr 16-240. |
|
| fccHandler |
| Posted: Dec 24 2011, 05:03 PM |
|
|
Administrator n00b
Group: Moderators
Posts: 3961
Member No.: 280
Joined: 13-September 02
|
Hmm... Then I am a little unsure about what to do in the MPEG-2 plugin. It seems to me that the FR distinction only matters if one plans to convert the data into a different colorspace. In other words, it's a way of interpreting the data, but doesn't affect the data itself. So I think in these two cases the plugin should return the same Y data...
--------------------
May the FOURCC be with you... |
|
| phaeron |
| Posted: Dec 24 2011, 08:52 PM |
|
|
Virtualdub Developer
Group: Administrator
Posts: 7773
Member No.: 61
Joined: 30-July 02
|
I'll double check the timing. I had to do some archaeology to back-annotate the APIs, and I might have made a mistake there.
You should absolutely not return the same data for both standard and _FR modes. This will cause VirtualDub to display and send the wrong ranges to downstream filters and codecs because the black and white levels will be wrong. Doing this kind of thing is why we have the levels mess that exists in VFW and DirectShow.
|
|
| fccHandler |
| Posted: Dec 25 2011, 02:26 AM |
|
|
Administrator n00b
Group: Moderators
Posts: 3961
Member No.: 280
Joined: 13-September 02
|
| QUOTE (phaeron @ Dec 24 2011, 04:52 PM) | | You should absolutely not return the same data for both standard and _FR modes. This will cause VirtualDub to display and send the wrong ranges to downstream filters and codecs because the black and white levels will be wrong. |
I have to respectfully disagree, for the simple reason that we have no way to know that the black and white levels are correct in the original. It would be bad to make that assumption, at least with MPEG-1 (ISO/IEC 11172-2).
Although the MPEG-1 colorspace is defined as CCIR 601, it is possible to encode full range luma and chroma in MPEG-1. (TMPGEnc actually offers this as an option.) The problem is, there isn't any way for us to know that this has happened. There is nothing in the resulting MPEG-1 video stream to indicate it.
Even if I analyze the data and see out-of-range values, it doesn't prove anything. I can't know whether these are full range colors (which should be kept), or blacker-than-black/whiter-than-white artifacts of the encoding process (which should be clamped).
If the user has discovered an MPEG which was encoded with full range, then he should have the option to interpret the data as such, by choosing a full range color depth in VirtualDub. The MPEG-2 plugin should deliver the data unmodified in either case, otherwise color destruction will occur. Again, since there is no way for me to know how the data was encoded, the decision must be at the user's discretion.
This argument may or may not apply for other codecs. I will probably have to decide on a case-by-case basis...
I've uploaded some videos to serve as examples of what I mean:
Original uncompressed RGB avi:
http://fcchandler.home.comcast.net/ColorBars.avi
Encoded as limited (Y 16-235) 4:2:0:
http://fcchandler.home.comcast.net/ColorBa...BarsLimited.m1v
Encoded as full (Y 0-255) 4:2:0:
http://fcchandler.home.comcast.net/ColorBarsFR.m1v
--------------------
May the FOURCC be with you... |
|
| fccHandler |
| Posted: Dec 28 2011, 06:20 AM |
|
|
Administrator n00b
Group: Moderators
Posts: 3961
Member No.: 280
Joined: 13-September 02
|
Man, trying to support all of these new colorspaces is kicking my butt. I think the only sane thing to do is to return the raw YCbCr data as it is, and let the user make the final decision on how it should be interpreted, by choosing an appropriate colorspace.
I'm sure this isn't what Avery intended, but in my experience the original levels are wrong as often as they are right, and in the past I received a lot of complaints about it. That is the reason I was forced to add an extended option "use matrix coefficients" which is disabled by default.
If Autoselect is chosen then I will try to be smart about it, otherwise all bets are off. If you select a decompression format which doesn't match the detected MPEG colorspace (if any) then that is your problem.
Sheesh, the MPEG-2 plugin File Information dialog is sorely lacking. There is a heck of a lot of important information gathered by the plugin which should really be exposed there.
I am currently embarked on a major overhaul of this plugin which will be the first really significant upgrade it has had in years. Unfortunately I am having to discard a lot of the assembly language color conversion code, which will undoubtedly hurt its performance. But computers are a lot faster nowadays, so we'll see how it goes...
--------------------
May the FOURCC be with you... |
|
| fccHandler |
| Posted: Dec 31 2011, 08:26 PM |
|
|
Administrator n00b
Group: Moderators
Posts: 3961
Member No.: 280
Joined: 13-September 02
|
To phaeron:
A colorspace once chosen seems to be fixed for the whole video stream, but in MPEG-2 each frame has its own "progressive_frame" flag which can change on a frame-by-frame basis, and affects how we should upsample 4:2:0 to 4:2:2.
Maybe the VDXVideoFrameInfo.mFrameType member could have a flag to indicate that a frame is interlaced, so that VirtualDub could switch colorspaces on the fly in this situation. I imagine it might be a nightmare to implement though... Just throwing the idea out there.
--------------------
May the FOURCC be with you... |
|
| fccHandler |
| Posted: Jan 2 2012, 06:14 AM |
|
|
Administrator n00b
Group: Moderators
Posts: 3961
Member No.: 280
Joined: 13-September 02
|
To phaeron:
While experimenting with the MPEG-2 plugin, I am noticing that I probably don't need to support ALL of the new colorspaces. For example 4:1:1 and 4:1:0. If I report that I do not support these, VirtualDub queries me for 4:2:2 planar (which I do support) and then seems to convert to 4:1:x internally; i.e., no error is reported in these cases. Needless to say I like that, since it means I don't have to support them directly.
However, if I also disable support for the YUY2 and UYVY formats, and then set my decompression format to YUY2 or UYVY, my plugin is queried for RGB888. In the context of MPEG that is quite an expensive conversion, so I wonder why VirtualDub doesn't query for 4:2:2 planar in these cases as well. Even 4:4:4 planar would be less expensive than RGB...
--------------------
May the FOURCC be with you... |
|
| phaeron |
| Posted: Jan 8 2012, 06:39 AM |
|
|
Virtualdub Developer
Group: Administrator
Posts: 7773
Member No.: 61
Joined: 30-July 02
|
You definitely do not need to support all of the color spaces, and I actually wouldn't recommend implementing the ones that require downsampling chroma. However, I would definitely recommend supporting YUY2 at least. As you've discovered, the current logic doesn't query for 4:2:2 planar as YV16 is not a common format in Windows; that should probably change now that it is also directly accessible in plugins. I don't think there would be a big problem with adding YV16 to the format degredation chain as that's not a commonly supported format. The logic for implementing that fallback is a bit nasty, though. :S
Interlaced/noninterlaced switching... ugh. Doable, I guess, but it'd have to be another format with a special side flag which I haven't previously done. |
|
| fccHandler |
| Posted: Jan 15 2012, 06:25 AM |
|
|
Administrator n00b
Group: Moderators
Posts: 3961
Member No.: 280
Joined: 13-September 02
|
I don't know if this is really "finished", but I'm going to push it out anyway:
http://fcchandler.home.comcast.net/Plugins/MPEG2
Let me know if you find anything ugly. Otherwise, enjoy. 
--------------------
May the FOURCC be with you... |
|
| levicki |
| Posted: May 10 2012, 02:09 PM |
|
|
Advanced Member
Group: Members
Posts: 167
Member No.: 22605
Joined: 13-December 07
|
@Avery Lee:
Few comments/questions regarding SDK 1.2:
1. runProc()
- In SDK runProc returns int but in VDXFrame it is declared as void so you can't return anything on failure. Is that by design? If so, how to handle errors in runProc()? What kind of (recoverable) exceptions can be thrown to VirtualDub to ensure it doesn't crash? Same question for startProc() and endProc().
- Your sample code involves pointer incrementing:
| CODE |
dst = (uint32 *)((char *)dst + dstpitch);
src = (const uint32 *)((const char *)src + srcpitch);
|
IMO, that is not the best teaching example because if you want to thread the loop using OpenMP you have to declare src and dst as thread private which is just unnecessary hassle and something that developers starting with OpenMP will often fail to grasp. It would be better to change the example like this instead:
| CODE |
uint32 pixelValue = src[y * srcpitch + x];
// divide all channels by 2
dst[y * dstpitch + x] = (pixelValue & 0xfefefe) >> 1;
|
That way there is no need to change src and dst pointers, and threading is more straightforward.
2. Alternate image formats
- What is the difference between using mpPixmap and mpPixmapLayout? I am not seeing any obvious differences (i.e. the structure is the same) and it seems to me that if I return FILTERPARAM_SUPPORTS_ALTFORMATS and do not set fa->dst.depth to zero I can still use mpPixmap and get the alternate formats. What is the proper way to use mpPixmapLayout for in-place filters?
- In the enum VDXPixmapFormat there are a lot of new formats, some of them may be self-explanatory but still documenting that enum (even with comments in the header only) would be usefull.
3. Aspect ratio
- With filters now supporting aspect ratio, it would be good if the user could override the aspect ratio reported by the video source. With "override" I don't mean only for display purpose (i.e. from the context menu) but also what will be reported to the filters.
4. CPU features
- What is the best place to call getCPUFlags()? Can the flags change between two runProc() calls?
- How to refuse running the filter in a most meaningfull way for the user if certain instructions are not supported? Perhaps introducing FILTERPARAM_NEEDS_SSE2, FILTERPARAM_NEEDS_AVX, etc? That would enable VirtualDub to always show a same looking message to the user instead of everyone popping up their own dialog boxes.
- Can you add functions that return the number of logical and the number of physical cores? If people start using threading in the filters (I did already) everyone will start using their own detection and it ain't going to be pretty because accurate detection is tricky. You can also return core count and a flag that says whether HyperThreading is enabled but it is necessary to know if there are logical cores because most filters will want to avoid those.
- There was an idea to run filters in parallel when possible -- I would like to have a way for a filter to prevent this from happening because I want to use threading in the filter itself. How to do that?
5. Interlaced content
- kPixFormat_YUV420i_Planar, kPixFormat_YUV420it_Planar, kPixFormat_YUV420ib_Planar what are they, and when the filter will get the bitmap in those formats?
- Is there any example how to write a filter that can work on frames and on fields differently depending on what it gets as the source?
6. Website out of date
- SDK 1.2 cannot be downloaded from virtualdub.org
That's it for now, I'll add more if I encounter more ambiguities. |
|
| phaeron |
| Posted: May 12 2012, 05:36 PM |
|
|
Virtualdub Developer
Group: Administrator
Posts: 7773
Member No.: 61
Joined: 30-July 02
|
| QUOTE | | In SDK runProc returns int but in VDXFrame it is declared as void so you can't return anything on failure. Is that by design? |
Yes, this is intentional. The return value from runProc is deprecated as it doesn't provide any context, so it causes VirtualDub to throw a non-specific error at the user. I specifically changed the prototype in the VideoFilter class to force the error throw methods on FilterFunctions to be used instead, which require an error message.
| QUOTE | - Your sample code involves pointer incrementing:
IMO, that is not the best teaching example because if you want to thread the loop using OpenMP you have to declare src and dst as thread private which is just unnecessary hassle and something that developers starting with OpenMP will often fail to grasp. It would be better to change the example like this instead:
| CODE | | uint32 pixelValue = src[y * srcpitch + x]; |
|
I absolutely do not want to teach people to use this style, for two reasons:
- It assumes that srcpitch is a multiple of the pixel size. This is not always true -- for instance, it is not true for the 10-bit per component format that VirtualDub supports internally. It is also not guaranteed by the spec, although it will happen to always be true for 32-bit pixels.
- It is far too easy to code this in a way that is less efficient, such as using the pitch field on the structure directly. Done in the inner loop this can cause the compiler to reload the pitch variable for every single pixel.
If you are using OpenMP, I would expect a higher level of experience to know which transformations are safe and effective.
| QUOTE | | What is the difference between using mpPixmap and mpPixmapLayout? I am not seeing any obvious differences (i.e. the structure is the same) and it seems to me that if I return FILTERPARAM_SUPPORTS_ALTFORMATS and do not set fa->dst.depth to zero I can still use mpPixmap and get the alternate formats. What is the proper way to use mpPixmapLayout for in-place filters? |
The layout is used during paramProc and startProc, and you have to change the layout in paramProc, not the pixmap itself. The pixmap is only guaranteed to be valid in runProc, since it contains an actual pointer to a buffer. Older versions of VirtualDub always bound buffers when calling startProc; this no longer occurs in the latest versions with the latest API version.
| QUOTE | | - What is the best place to call getCPUFlags()? Can the flags change between two runProc() calls? |
This is up to you, but these flags can indeed change as the filter chain is not stopped when the preferences dialog is displayed. If you are making changes within your filter that change internal structures and can't accommodate an on the fly change, you should cache the result of this in startProc. It is also acceptable simply to cache it on load as changing the CPU support level dynamically is not a common user path.
| QUOTE | | kPixFormat_YUV420i_Planar, kPixFormat_YUV420it_Planar, kPixFormat_YUV420ib_Planar what are they, and when the filter will get the bitmap in those formats? |
Besides filters, the input driver can produce these formats, although I'm not sure if any do other than the built-in ones. The "Other" setting in Video Depth can trigger this if the input driver supports it. The test driver supports the interlaced format, as it uses the internal blitter library to do the conversion.
The "it" and "ib" formats stand for interlaced top field and interlaced bottom field. They are there to support discarding one field from the interlaced chroma format, as the result is non-interlaced chroma with an unusual vertical offset.
| QUOTE | | Is there any example how to write a filter that can work on frames and on fields differently depending on what it gets as the source? |
No, not currently.
| QUOTE | | - SDK 1.2 cannot be downloaded from virtualdub.org |
Sorry, I need to get around to that... been too busy. There's also a bug in the 1.2 SDK I need to fix where the VDXFrame library can cause a crash in filters on old versions of VirtualDub that don't support 3D acceleration -- it tries to read the VDXAContext field without checking if it is present. The fix involves pushing the API version into the VDXFrame library so it can decide whether this field can be read safely.
|
|
| levicki |
| Posted: May 12 2012, 08:11 PM |
|
|
Advanced Member
Group: Members
Posts: 167
Member No.: 22605
Joined: 13-December 07
|
| QUOTE | | QUOTE | | Yes, this is intentional. The return value from runProc is deprecated as it doesn't provide any context, so it causes VirtualDub to throw a non-specific error at the user. I specifically changed the prototype in the VideoFilter class to force the error throw methods on FilterFunctions to be used instead, which require an error message. |
Could you give an example of such throw?
|
ff->Except() or ff->ExceptOutOfMemory(). These throw an exception out of the current function with the given message.
| QUOTE |
| QUOTE | | It assumes that srcpitch is a multiple of the pixel size. |
No, it assumes that between y and y + 1 in src there are srcpitch bytes. Are you saying that is no longer true?
|
You're indexing uint32 pixels, so this would require srcpitch to be in elements or for the indexing to use srcpitch/4. If you're indexing bytes then this will work, but it's still not advised.
| QUOTE |
| QUOTE | | Done in the inner loop this can cause the compiler to reload the pitch variable for every single pixel. |
Come on... not even MSVC is so brain dead these days -- maybe in debug mode it would reload, but with /O1 or O2? I'd like to see a test case for that.
|
It's not brain-dead, it's called aliasing.
| CODE |
#include <stdint.h>
struct Image {
void *p;
ptrdiff_t pitch;
uint32_t height;
};
void foo(const Image& dst, const Image& src) {
uint32_t sum;
const uint8_t *ps = (const uint8_t *)src.p;
uint8_t *pd = (uint8_t *)src.p;
for(uint32_t y=0; y<src.height; ++y) {
pd[y * dst.pitch] = ps[y * src.pitch];
}
}
?foo@@YAXABUImage@@0@Z (void __cdecl foo(struct Image const &,struct Image const &)):
00000000: 8B 4C 24 08 mov ecx,dword ptr [esp+8]
00000004: 33 C0 xor eax,eax
00000006: 56 push esi
00000007: 8B 31 mov esi,dword ptr [ecx]
00000009: 39 41 08 cmp dword ptr [ecx+8],eax
0000000C: 76 20 jbe 0000002E
0000000E: 53 push ebx
0000000F: 57 push edi
00000010: 8B 7C 24 10 mov edi,dword ptr [esp+10h]
00000014: 8B 51 04 mov edx,dword ptr [ecx+4]
00000017: 8B 5F 04 mov ebx,dword ptr [edi+4]
0000001A: 0F AF D0 imul edx,eax
0000001D: 8A 14 32 mov dl,byte ptr [edx+esi]
00000020: 0F AF D8 imul ebx,eax
00000023: 40 inc eax
00000024: 88 14 33 mov byte ptr [ebx+esi],dl
00000027: 3B 41 08 cmp eax,dword ptr [ecx+8]
0000002A: 72 E8 jb 00000014
0000002C: 5F pop edi
0000002D: 5B pop ebx
0000002E: 5E pop esi
0000002F: C3 ret
|
This is with /Ox with Visual Studio 2010 SP1 Pro. Notice the imul instructions in the loop.
The reason it happens is because the compiler is unsure about whether the writes to the destination buffer alias to the structures that are passed to the filter. This is particularly apt to occur since the structure contains ptrdiff_t and uint32 types, and because Visual C++ errs more on the side of safety with regard to aggressive aliasing optimizations. Indexing rows with (char *) will also exacerbate this problem due to the special aliasing status of that type in the language.
Careful use of __restrict can avoid this, of course, and in this case it causes the compiler to recognize that the pitches are loop-invariant constants and allows it to apply inductive optimizations. I prefer not having to second-guess the compiler all of the time in critical loops.
| QUOTE |
| QUOTE | | The layout is used during paramProc and startProc, and you have to change the layout in paramProc, not the pixmap itself. The pixmap is only guaranteed to be valid in runProc, since it contains an actual pointer to a buffer. |
So if I understand you correctly, we still continue to use mpPixmap to access data in runProc() and we only use mpPixmapLayout in paramProc and startProc to change format if needed and to preallocate buffers? Am I safe to assume that I don't need to touch mpPixmapLayout if I am writing an in-place filter (i.e. not changing output format)?
|
You must use exclusively the pixmap layout in the paramProc and startProc functions. Furthermore, you must use at least fa->src.mpPixmapLayout in a filter using the current API version, because API V14 requires the FILTERPARAM_SUPPORTS_ALTFORMATS flag to be set, and you cannot successfully use this flag without affirmatively checking the input format as you will be caught by the runtime's invalid format check. (I may relax this check as there are actually a few cases where a filter can legitimately work on any format without having explicit support for each format.)
You may use either the layout or the pixmap structures in runProc. It is guaranteed that the two are consistent.
| QUOTE |
- How to refuse running the filter in a most meaningfull way for the user if certain instructions are not supported? Perhaps introducing FILTERPARAM_NEEDS_SSE2, FILTERPARAM_NEEDS_AVX, etc? That would enable VirtualDub to always show a same looking message to the user instead of everyone popping up their own dialog boxes.
|
Two ways I can think of: throw an exception on initProc or startProc, or simply fail to add the filter(s) in the module init procedure. If you are planning on compiling entire modules to use those extensions, say with /arch on Visual Studio, then you may need to compile the module init procedure explicitly without those flags so it can reject the load. The third way is that you can simply crash, but that's of course unfriendly to the user. That having been said, current signs do point to SSE2 being a decent baseline.
| QUOTE |
- Can you add functions that return the number of logical and the number of physical cores? If people start using threading in the filters (I did already) everyone will start using their own detection and it ain't going to be pretty because accurate detection is tricky. You can also return core count and a flag that says whether HyperThreading is enabled but it is necessary to know if there are logical cores because most filters will want to avoid those.
|
Logical cores is not really a problem. In fact, I would actually recommend just counting the number of bits in the process affinity mask rather than checking the CPU for purposes of estimating a parallelism target. This will make your filter execute more reasonably if someone intentionally restricts core usage on the process.
Checking physical cores and/or HyperThreading, however, is a pain in the butt. I know how to do it, I'm honestly not sure it would be a good idea for me to provide this, especially since it's likely to be something the user would want or need to tune.
| QUOTE |
- There was an idea to run filters in parallel when possible -- I would like to have a way for a filter to prevent this from happening because I want to use threading in the filter itself. How to do that?
|
There is intentionally no way to do this. If a filter tries to do this manually it can cause VirtualDub's filter pipeline to deadlock by introducing a dependency cycle into the filter graph.
What is guaranteed, however, is that the runProc entry point for your filter is not run in parallel. Two instances of your filter can run in parallel -- and this will happen if filter multithreading is enabled -- but a single instance is always serialized for all frame requests that it services. In other words, VirtualDub requires that your filter's code be reentrant, but not each instance. If I introduce a feature in the future to allow a single instance to run in parallel, this will always be opt-in or through a cloning mechanism that allows the filter to safely duplicate or mediate any shared mutable structures.
If the issue is controlling parallelism, one way to deal with that is to use the Win32 thread pool APIs, thus allowing all instances of your filter to share the thread pool. You can also use a mutex lock to mediate control to shared resources, at the cost of reduced throughput. In any case, you must restrict any synchronization to runProc itself as attempting to do so from prefetchProc/2 will cause lockups. Those functions must be reentrant and non-blocking.
| QUOTE |
I would like to rephrase the last one to "how to signal the user that he should disable the parallel execution of the filters if he wants to have the best performance from my filter?". Is there any way to query VirtualDub for user preference (Video filter threading)?
|
I'm not sure this is much of a worry. If your filter uses a shared thread pool between all instances and uses granular enough work items to reduce the impact of context switches, it shouldn't be that bad even if filter parallelism in VirtualDub is active. VirtualDub's filter system is designed to accommodate high latency when running in threaded mode. In a case where the pipeline is CPU bound on your filter, the filter system should be able to queue enough frames through to keep the cores busy, at which point the main concern is just context switching between processing for the various frames.
|
|
| phaeron |
| Posted: May 12 2012, 09:37 PM |
|
|
Virtualdub Developer
Group: Administrator
Posts: 7773
Member No.: 61
Joined: 30-July 02
|
Ack, I was an idiot and edited your post instead of responding to it. Hope it still makes sense to people.
|
|
|
