How does compression work?
 

More specificly assuming the same A/F ratio and displacement is there a formula that would calculate gains from changing the CR.

Also a turbo artificialy changes the compression by forcing more air, correct? What would happen if someone used ungodly high compression rotors that would simulate a boosted application.

Boy, you asked a mouthful! The higher you raise the compression ratio the higher octane fuel you need to keep from detonating. Theoretically you could "build up" a set of rotors and balance them. You better be ready to run it on racing fuel though.
I am not sure whether this has been tried before, but I would not want to be the first. You could calculate how much to build the rotors up to reach a certain ratio by measuring the volume of the reccession on a turbo rotor and an n/a rotor.
Subtract the difference of these volumes. Make sure that you know exactly what year the rotors are from and look up their ratio. Subtract the ratio's between the two rotors. To make this easy lets say the turbo is 8:1 and the n/a is 9:1. Let's say the difference in the volumes is 10cc. If you wanted an 11:1 ratio you would add 20cc to an n/a rotor.
These are not ACTUAL figures, just examples of how to calculate. So have fun and keep us updated on your progress!:D

Re: How does compression work?
 

That's pretty complicated. Basically, a higher compresison ratio results in a lower the specific fuel consumption, greater thermal efficiency, but increased probability of detonation and preignition.

No. A supercharger (turbo, blower, etc.) compresses air so that more air mass is fed into the engine. The engine compresses the fuel/air ratio in order to build more pressure to push against the rotor (or piston), producing power. Basically, the turbo's job is to produce air mass, while the rotor's job is to produce pressure. Only the Miller cycle combines these two concepts.
http://www.howstuffworks.com/question132.htm

Your engine would blow to pieces from severe detonation.

Thank you for the information

The math behind this is actually pretty complicated, I have the derivation buried somewhere in my notes, but I still haven't gotton to a level where I can fully grasp the math. The simple answer goes more like this:

If you've taken any thermodynamics, work=I pdv (integral of Pdv) where P=pressure and dv is a change in volume. If you raise your compression ratio, your peak P value is going to be much higher, so for each piston stroke, you'll be doing more work. Think of it as more area under the curve; it is that peak area when the piston is right at TDC where most of the work comes from, so by raising up that area you can make a big difference. Power is just work/time, and your time scale doesn't change, so there you go, more power.