Added really cheap "fake" shadows during photon gathering. Here's what I did.
Basically, after photon gathering is done and I have the final color value, I trace a shadow ray to the light source. If it's blocked, I multiply the gathered color value by a fractional constant, making it dimmer. That's it! This can also be easily extended to soft shadows, but it might be more expensive (and that'll encroaching on the area of monte carlo path tracing.)
Looks pretty convincing as far as I can tell, and this is much cheaper than the blending between full out ray tracing and photon gathering that I had earlier.