a lighter flywheel also lowers (tell me if I'm wrong) some of the momentum of the engine so you'll lose a little "off the line kick", but you'll rev-up much quicker. Is this right?

If you still have the cats in your exhaust system you're losing a lot right there too, at least get a straight pipe to replace those if you're not gonna do a whole exhaust. It's not that expensive.