Quantcast
Channel: GameDev.net
Viewing all articles
Browse latest Browse all 17560

D3DCompile crashes with stack overflowing when compiling this HLSL. Very Strange

$
0
0

Recently I bumped into this weird problem when I tried to compile a HLSL pixel shader. The D3DCompile function just did a infinite recursive call and caused a stack overflow. I simplified the pixel shader as following:

uniform float end : register(c0); 
uniform float start : register(c1); 
static float4 c[1] =
{
    float4(0, 0, 0, 0)
};
struct PS_INPUT
{
    float4 v0 : TEXCOORD0;
};
struct PS_OUTPUT
{
    float4 gl_Color0 : SV_TARGET0;
};  
PS_OUTPUT main(PS_INPUT input)
{
    c[0] = input.v0; 
    if (c[0].y > start && c[0].y < end)
    {
        c[0] *= 1.0; 
    }
    else if (c[0].y >end)
    {
        c[0] *= 2.0; 
    }
    PS_OUTPUT output; 
    output.gl_Color0 = c[0]; 
    return output;
}

Then, I tried to compile it with D3DCompile function as following:

HMODULE mD3DCompilerModule = LoadLibrary(D3DCOMPILER_DLL);
auto mD3DCompileFunc = reinterpret_cast<pD3DCompile>(GetProcAddress(mD3DCompilerModule, "D3DCompile"));
mD3DCompileFunc(s, strlen(s), "D:\\fakepath", NULL, NULL, "main", "ps_5_0",  0, 0, NULL, NULL);

which is pretty standard.

The crash(not a failure return value) happend at the last line, in d3dcompiler_47.dll.

Though I am new to DirectX, I think my shader should be correct with grammar, so what is wrong with this example?

Some cases that will make the compilation successful that I found:

  1. using a static varaible c instead of c[1] array.
  2. using 'else' instead of 'else if' in the condition control flow, however the semantics is changed.
  3. using D3DCOMPILE_SKIP_OPTIMIZATION when compiling, losing binary optimization

Viewing all articles
Browse latest Browse all 17560


<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>