What are the rules for detecting spies?
ie: is it the ship sensors or planet sensors or the higher of the 2?
What are the probabilities of not detecting?

If you mean using the clean planet/ship then it is based upon the ship or planet sensors compared to the cloak value of the spy on the ship/planet. When you place a spy on a planet/ship the CURRENT cloaking value of the ship doing the placing is used by that spy. So if your ship has a cloak of 200 then your spy will have a cloak of 200. The spies cloak value will NEVER change from that point on. The only way to change the spies cloak value is to go back to the planet, pick up the spy with a HIGHER cloak ship and place the spy back on the planet.

Understand the spy cloak value, I am more concerned in how I detect enemy spies on my worlds
If I use the clean option on the planet, what are the rules in the detection calcs?
is it the planet sensor level, the ship sensor level, the levels of the spies on that planet?

Yes, I've always been curious about the same thing myself. I assumed it was the ship's tech level, or else low level planets would never find spies with really big cloak, which hasn't been the case in my experience.