Strange slowdown when model(s) goes off screen

Got a pretty strange issue, perhaps someone recognizes it.

I have a model that I created the vertex and index buffers for and that I render using DrawIndexedPrimitive, after that I render a bunch of models that are loaded from .x files. Whatever I render after the first model gets really slow when all the models goes off screen, the fps goes from 150 to 20. What could be causing this?

If I render the first model after the "bunch of other models" the issue goes away but if I render another bunch of models after that the issue is seen again.