Brake pedal too hard
 

I'm having a weird brake problem. When ever it's warm out and I use my brakes heavily the pedal will become very stiff to where barely touching the pedal engages the brakes.

I looked up a few suggestions but I don't think it's the booster since it does return to normal after a while, and I don't think it's pad fade since I can still stop.

A good booster wouldn't do that so don't rule it out just because it works "sometimes". Sometimes working isn't fully functional either. Outside of that, I'd probably take a look at your master cylinder and clean it out. Your return port might be clogged which can result in the brakes to not release properly.
Wouldn't hurt to take a look at everything brake related. This is the most important system in the car you know!

The problem seems to have started when I did a 4 piston caliper swap. Something I should check on the calipers?

Caliper pistons could be sticking and therefore not realeasing as was said. If you bought the calipers used and they were off of a car for a while without being capped or stored properly they could have easily rusted up inside. Do you have excessive brake dust? Look at the pads, are they wearing evenly? Etc.

Sort of sounds like a bad check valve.

I'd check the pedal>booster adjustment, which sounds too tight.
As the fluid heats up and expands, you lose freeplay and eventually, the master cylinder cannot fully close and the brakes drag.

So, with brakes cold, loosen up the adjustment till there's obviously too much freeplay (but the brakes still work) and then drive her around till the brakes reach operating temp. Now readjust the freeplay to the normal spec. Drive some more and make sure that even when hot there is freeplay at the pedal.

Maybe solve the issue, maybe not.
It's free though, so there's that.