Problem

You try to stay to secure. Say what? Yes, too secure. If you work for different customers and you are like me you might not want to use the same key for every customer. Now, for convenience, you add all your keys to your local agent:

ssh-add key1
    ...
ssh-add keyN

Yikes. Secure and still convenient. Unless you get an auth error:

Received disconnect from a.b.c.d port 22:2: Too many authentication failures

You double check the username. You double check your key is loaded to the agent. You start to doubt yourself. Yet nothing seems to be an explanation.

Reason

After some searching in the net you suddenly get the answer:

Your SSH client sends over the first key. If it is not allowed the second and so on. And if you made your server somewhat secure after a certain amount of tries the server just disconnects with the above error. The tricky thing now is, that for some servers it works (those where the key is within the configured limit) and others not. As soon as you read the explanation it becomes clear and totally understandable.

Another obstacle: Just passing -i will not work, as -i does not limit the keys to the one provided.

Solution

You can work around this by passing options to your ssh client limiting the key to a provided one:

-o "IdentitiesOnly yes" -i <path to your private key>

The obvious drawback is you have either to remember this options or configure it in your ssh config for each server.

Another reason is to reconfigure your agent depending on the work you are doing. Because we never jump between tasks back and forth, do we? To go down that road you can handle this with direnv like this:

ssh-add -D
ssh-add <key>

Switching to that directory would remove all existing keys and add the one specified. The drawback here is, that this works well in a terminal but graphical environments are out of scope. Additionally, when jumping around your keys get removed in one terminal and the other terminal is invalid for the workload you’re on there.

You could start another agent and export the socket for each shell separately but I have not tested this.