I was using a S4 TII transmission on my REW swap and it made no noises for the first little bit. Then the pilot bearing must have went bad because with the car in neutral and the clutch pedal out, it made a whirring sound. When I pressed the clutch in, it went away. I figured that was a minor pilot bearing issue that is easy enough to replace (I have the engine out now anyways).
But after a few more months, it started making a higher pitched "whop whop whop* sound when down shifting in to 2nd or 1st (1st being louder).

Since I have everything taken apart now I was considering replacing the transmission but wanted to make sure it's not anything else first.

 

You have it backwards, when in neutral and the clutch out, the pilot bearing, clutch, input shaft and half the gears in the transmission are all moving at engine speed, only the transmission is not engaged in any gear so the output shaft does not move. When the clutch is depressed, engine running, all of the transmission stops(and so does your noise, I'd bet) and the pilot bearing is spinning around a stationary input shaft at engine speed. The pilot bearing and throwout bearing both work at the same time and noises between the two can get confused.

 

Whirring neutral goes away when I push the clutch in. Pilot bearing was never replaced but the throw out bearing was brand new.

 

The noise is in the transmission. Whether or not it is going out depends on how loud it is. Both of my cars make a rattling noise when idling in neutral, worse with the a/c on, bu, I know that is is normal. I have known both cars for 20 and 30 years.

 

check the transmission mounts, sounds like you have the tranny sitting on the crossmember.

 

Me or wms10th? I'm going to get new mounts while I'm doing the bay clean up anyways.