When logging in to a server via SSH using a private key with a password, OSX saves your password. The next time you try to login it will not prompt you for your password. To disable this behaviour simply run this command:
launchctl unload -w /System/Library/LaunchAgents/org.openbsd.ssh-agent.plist
Hat tip to [mr_dbr on reddit] for this.