Buffer as both StructuredBuffer & RWStructuredBuffer

I am trying to use a buffer as a rwstructuredbuffer in one call and in another reuse that as a structuredbuffer to read the data from. However, that's not working. Can you not do that? There is no error thrown when binding the buffer as both ShaderResource & UnordererdAccess. It also works fine if I replace the StructuredBuffer with a RWStructuredBuffer in the reading call.

and I am also doing the following already:
- use StructuredBuffer<uint> : register(tx) vs RWStructuredBuffer<uint> : register(ux) in the shader
- use CSSetShaderResource for structured vs CSSetUnorderedAccessViews for rwstructured

I am hoping to use this buffer in different shaders (vs, ps, cs, but these are all different draw call/dispatches), and since rwstructuredbuffer is not usable in VS AFAIK, I was hoping I can bind it as both structured and rwstructured. Any thoughts?

Thanks!