Recently I bumped into this weird problem when I tried to compile a HLSL pixel shader. The D3DCompile function just did a infinite recursive call and caused a stack overflow. I simplified the pixel shader as following:
uniform float end : register(c0); uniform float start : register(c1); static float4 c[1] = { float4(0, 0, 0, 0) }; struct PS_INPUT { float4 v0 : TEXCOORD0; }; struct PS_OUTPUT { float4 gl_Color0 : SV_TARGET0; }; PS_OUTPUT main(PS_INPUT input) { c[0] = input.v0; if (c[0].y > start && c[0].y < end) { c[0] *= 1.0; } else if (c[0].y >end) { c[0] *= 2.0; } PS_OUTPUT output; output.gl_Color0 = c[0]; return output; }
Then, I tried to compile it with D3DCompile function as following:
HMODULE mD3DCompilerModule = LoadLibrary(D3DCOMPILER_DLL); auto mD3DCompileFunc = reinterpret_cast<pD3DCompile>(GetProcAddress(mD3DCompilerModule, "D3DCompile")); mD3DCompileFunc(s, strlen(s), "D:\\fakepath", NULL, NULL, "main", "ps_5_0", 0, 0, NULL, NULL);
which is pretty standard.
The crash(not a failure return value) happend at the last line, in d3dcompiler_47.dll.
Though I am new to DirectX, I think my shader should be correct with grammar, so what is wrong with this example?
Some cases that will make the compilation successful that I found:
- using a static varaible c instead of c[1] array.
- using 'else' instead of 'else if' in the condition control flow, however the semantics is changed.
- using D3DCOMPILE_SKIP_OPTIMIZATION when compiling, losing binary optimization