From time to time you should change your password. Everyone knows. For your SSH key this is as simple as changing your Linux login password:

ssh-keygen -p -f ~/.ssh/id_rsa

Maybe you have to change the path/name depending on your setup.