the compass is weak and the magnets are strong. you are changing the polarity on the compass with the magnets. i have experienced this as well. when the north needle is pointing north, then move the magnet in from the side(like east) slowly and see if the needle moves toward or away. this way you should be able to tell without adversely effecting the compass. you should know which way is north where you are at so check the compass first.
one other thing. some magnets are both pos and neg on the same side. fridge magnets are notorious for this. you want dipole magnets where one side is all pos and the other side is all neg.