Printable Version of Topic
Click here to view this topic in its original format
Unofficial VirtualDub Support Forums > Testing / Bug Reports > Vd Crashes After Resizing With Vdshader Plugin


Posted by: Rumbah Jun 4 2012, 03:43 PM
Virtualdub 1.9.11 and 1.10.2 crash when doing the following steps:

Install vdShaders 1.3
Open a video
Add the cdShaders filter and in the gui save the example "convert to greyscale" to the disk, e.g. as test.fx
Load the file from disk in the shader editor
Unclick the "Keep existing size"
Select a size bigger than the video
Close the shader editor
Press Ok in the filters window
Virtualdub crashes

Here a crash report:
CODE
VirtualDub crash report -- build 34807 (release)
--------------------------------------

Disassembly:
06200440: 2400            and    al, 00h
06200442: 0400            add    al, 00h
06200444: 000f            add    [edi], cl
06200446: 29b424f0030000  sub    [esp+3f0], esi
0620044d: 0f297c2460      movaps [esp+60h], xmm7
06200452: 0f288424f00300  movaps xmm0, [esp+3f0]
         00              
0620045a: 0f288c24000400  movaps xmm1, [esp+400]
         00              
06200462: 0f289424100400  movaps xmm2, [esp+410]
         00              
0620046a: 0f289c24200400  movaps xmm3, [esp+420]
         00              
06200472: 0f2825c01b1e07  movaps xmm4, [071e1bc0]
06200479: 0f59c4          mulps  xmm0, xmm4
0620047c: 0f59cc          mulps  xmm1, xmm4
0620047f: 0f59d4          mulps  xmm2, xmm4
06200482: 0f59dc          mulps  xmm3, xmm4
06200485: 660f            db     0fh
06200487: 5b              pop    ebx
06200488: c0660f5b        shl    byte ptr [esi+0fh], 5bh
0620048c: c9              leave  
0620048d: 660f            db     0fh
0620048f: 5b              pop    ebx
06200490: d2660f          shl    byte ptr [esi+0fh], cl
06200493: 5b              pop    ebx
06200494: db              db     0dbh
06200495: 660f6bc0        packssdw xmm0, xmm0
06200499: 660f6bc9        packssdw xmm1, xmm1
0620049d: 660f6bd2        packssdw xmm2, xmm2
062004a1: 660f6bdb        packssdw xmm3, xmm3
062004a5: 660f67c0        packuswb xmm0, xmm0
062004a9: 660f67c9        packuswb xmm1, xmm1
062004ad: 660f67d2        packuswb xmm2, xmm2
062004b1: 660f67db        packuswb xmm3, xmm3
062004b5: 660f60d1        punpcklbw xmm2, xmm1
062004b9: 660f60c3        punpcklbw xmm0, xmm3
062004bd: 81ee04000000    sub    esi, 00000004
062004c3: 660f61d0        punpcklwd xmm2, xmm0
062004c7: 0f8825000000    js     062004f2
062004cd: 0f1117          movups [edi], xmm2      <-- FAULT
062004d0: 8d7f10          lea    edi, [edi+10h]
062004d3: 0f8572fcffff    jnz    0620014b
062004d9: 8d65f4          lea    esp, [ebp-0ch]
062004dc: 8b6c240c        mov    ebp, [esp+0ch]
062004e0: 8b7c2408        mov    edi, [esp+08h]
062004e4: 8b742404        mov    esi, [esp+04h]
062004e8: 8b1c24          mov    ebx, [esp]
062004eb: 81c410000000    add    esp, 00000010
062004f1: c3              ret    
062004f2: 81c604000000    add    esi, 00000004
062004f8: 660f7e17        movd   [edi], xmm2
062004fc: 81c704000000    add    edi, 00000004
06200502: 660f70d2f9      pshufd xmm2, xmm2, 0f9h
06200507: 81ee01000000    sub    esi, 00000001
0620050d: 0f85e5ffffff    jnz    062004f8
06200513: e9c1ffffff      jmp    062004d9
06200518: 0000            add    [eax], al
0620051a: 0000            add    [eax], al
0620051c: 0000            add    [eax], al
0620051e: 0000            add    [eax], al
06200520: 0000            add    [eax], al
06200522: 0000            add    [eax], al
06200524: 0000            add    [eax], al
06200526: 0000            add    [eax], al
06200528: 0000            add    [eax], al
0620052a: 0000            add    [eax], al
0620052c: 0000            add    [eax], al
0620052e: 0000            add    [eax], al
06200530: 0000            add    [eax], al
06200532: 0000            add    [eax], al
06200534: 0000            add    [eax], al
06200536: 0000            add    [eax], al
06200538: 0000            add    [eax], al
0620053a: 0000            add    [eax], al
0620053c: 0000            add    [eax], al
0620053e: 0000            add    [eax], al

Built on Althena on Sun Jun 03 14:28:23 2012 using compiler version 1400

Windows 6.1 (Windows 7 x64 build 7601) [Service Pack 1]
Memory status: virtual free 1820M/2048M, commit limit 8188M, physical total 4095M

EAX = 091faf3c
EBX = 0018f9e8
ECX = 000000ca
EDX = 0018f9e8
EBP = 0018f948
ESI = 00000084
EDI = 094f9ff1
ESP = 0018f500
EIP = 062004cd
EFLAGS = 00010206
FPUCW = 027f
FPUTW = ffff

Crash reason: Access Violation

Crash context:
An out-of-bounds memory access (access violation) occurred at 062004cd...

...writing address 094FA000...

...while running filter "vdshader" (FilterInstance.cpp:2198).

Pointer dumps:

EAX   091faf38: ff000000 ff000000 ff000000 ff000000 ff000000 ff000000 ff000000 ff000000
EBX   0018f9e8: 3a4ccccd 3f7fbc25 00000000 00000000 3acccccd 00000000 00000000 00000000
EDX   0018f9e8: 3a4ccccd 3f7fbc25 00000000 00000000 3acccccd 00000000 00000000 00000000
EDI   094f9ff1: 00000000 00000000 00000000
ESP   0018f500: 000000c9 000000c9 000000ca 000000ca 000000ed 000000ed 000000ed 000000ed
     0018f520: 00000000 0ff00000 00000000 0ff00000 00000000 0ff00000 00000000 0ff00000
     0018f540: 00000000 0ff00000 00000000 0ff00000 00000000 0ff00000 00000000 0ff00000
     0018f560: 3f4b6674 3f4bccda 3f4c3341 3f4c99a7 466e0000 466e0000 466e0000 466e0000
EBP   0018f948: 002b9c74 071a0af4 094f9811 0018f9e8 00000280 07198643 094f9811 0018f9e8
     0018f968: 00000280 0076d380 0076d378 00000000 00000000 093793c0 000001e0 09379ab0
     0018f988: 002b9c74 3a4ccccd 00000002 00000001 0072c1a0 00000010 00000001 0051c001
     0018f9a8: 00000000 092a0001 00000100 0076d3d8 3b088889 09480000 00000001 00000280

Thread call stack:
062004cd: 062004cd
759c7038: USER32!GetWindowLongW [759b0000+16ffe+3a]
759c7038: USER32!GetWindowLongW [759b0000+16ffe+3a]
72d70f7f: uxtheme!00010f7f
72d70f47: uxtheme!00010f47
72d70f28: uxtheme!00010f28
72d70fa6: uxtheme!00010fa6
72d70d73: uxtheme!00010d73
759c62fa: USER32!gapfnScSendMessage [759b0000+15fc8+332]
7767b412: ntdll!RtlFormatCurrentUserKeyPath [77640000+3b161+2b1]
7767b395: ntdll!RtlFormatCurrentUserKeyPath [77640000+3b161+234]
77673c94: ntdll!RtlImageNtHeader [77640000+33184+b10]
77673cc3: ntdll!RtlImageNtHeader [77640000+33184+b3f]
7766e38c: ntdll!RtlInitUnicodeString [77640000+2e228+164]
7766e38c: ntdll!RtlInitUnicodeString [77640000+2e228+164]
7766e0f2: ntdll!RtlAllocateHeap [77640000+2e046+ac]
759d0c91: USER32!GetClientRect [759b0000+20c62+2f]
005e4bc6: malloc()
071a0af4: vdshader!VirtualdubFilterModuleInit2 [07190000+a7e0+6314]
07198643: vdshader!VirtualdubFilterModuleDeinit [07190000+7f70+6d3]
0051c001: VDMemcpyRect()
071c2bb3: vdshader!VirtualdubFilterModuleInit2 [07190000+a7e0+283d3]
0043dc07: FilterInstance::RunFilterInner()
00439b30: VDFilterFrameManualSource::GetNextRequest()
0043b4ca: VDFilterFrameVideoSource::RunRequests()
0043feca: _catch$?RunFilter@FilterInstance@@IAEXXZ$0()
0044097a: FilterInstance::RunProcess()
00447944: FilterSystem::Run()
00471036: VDProject::UpdateFrame()
00447680: FilterSystem::RequestFrame()
00473084: VDProject::DisplayFrame()
0047394f: VDProject::UpdateFilterList()
004774d7: VDProjectUI::SetVideoFiltersAsk()
759c81c8: USER32!IsProcessDPIAware [759b0000+181a6+22]
759c8326: USER32!GetClassNameW [759b0000+182a9+7d]
0047f039: VDProjectUI::MenuHit()
759c6b00: USER32!gapfnScSendMessage [759b0000+15fc8+b38]
0048c7d0: VDUIFrame::DefProc()
0047fe67: _catch$?MainWndProc@VDProjectUI@@IAEJIIJ@Z$0()
00475f40: VDProjectUI::WndProc()
0048ce85: VDUIFrame::StaticWndProc()
759c62fa: USER32!gapfnScSendMessage [759b0000+15fc8+332]
759c6d3a: USER32!GetThreadDesktop [759b0000+16c63+d7]
759c6ce9: USER32!GetThreadDesktop [759b0000+16c63+86]
759c77c4: USER32!CharPrevW [759b0000+1768c+138]
759d60dc: USER32!PeekMessageA [759b0000+25f74+168]
759c7bca: USER32!DispatchMessageA [759b0000+17bbb+f]
0045fe46: WinMain@16()
005e9586: __tmainCRTStartup()
753f339a: kernel32!BaseThreadInitThunk [753e0000+13388+12]
77679ef2: ntdll!RtlInitializeExceptionChain [77640000+39e8f+63]
77679ec5: ntdll!RtlInitializeExceptionChain [77640000+39e8f+36]

-- End of report

If you need any additional data just let me know.

Posted by: phaeron Jun 9 2012, 10:31 PM
Got it... it's a bug in the filter. Try this version:

http://virtualdub.org/beta/vdshader-1.4-test1.zip
http://virtualdub.org/beta/vdshader-1.4-test1-src.zip

Posted by: Rumbah Jun 10 2012, 01:45 PM
Thank you, this version has fixed it, no crash anymore.

Powered by Invision Board (http://www.invisionboard.com)
© Invision Power Services (http://www.invisionpower.com)