Need help with gear shift bushing
 

I am changing out my my bushings in the gear shift and I replaced one. The other will not come off. On the shifter itself there are two balls, a small one at the bottom and a larger one above that. One of the bushings is above the large ball, so it is between the shift nob and the large ball. Is it suppose to be there? The dealership said no and a friend said yes. The other question is how do I get it off, I have tried everything.
I hope any of these makes sense. If not I will try to explain it more.

Thanks, Eric

Did you not take the shift knob off? I should just unscrew.