As you know, Olli, I had written a program on macos to do some Fourier Optics with (I called it "Airy" :-). So I just made a quick test to see what a notched circular aperture (to simulate the prism shadow) looks like.

Notice that the nice Airy rings have fractured, and there is a strong serrated spike in the vertical direction. So, instead of stars looking like an Airy disk, it looks like the point spread function on the right panel above.
Notice too, that in addition to a vertical spike, a bright star would not even be round!
This is sort of an extreme case. Most people do not have such a bad shadow as I modeled on the left panel. But it is just a matter of degree of error.
Chen