Compass should work as long as you are near a planet. Basically, it points you towards wherever the cursor is on the planet map. If you lock your cursor to a ship or monster, it will follow that. If you move it to a patch of ground, it will point in that direction. It shouldn't point in the opposite direction, I haven't seen this behaviour before. Basically, if you line up the purple squares you will be flying in the shortest line to where you want to go (sometimes this goes over the poles instead of east-west which is a bit counterintuitive). The other thing, suppose you activate autopilot and then move the cursor. The compass will point to the cursor, not the autopilot destination. That could also be it.

You can also disable the compass if it's visually too annoying by pressing [C].