Why is SSH public key auth considered more secure?

Often it is suggested, that you disable ssh password authentication completely and use public key only.

Granted, brute-force attackers can't break in anymore. But my main concern is, that the machine (PC, notebook) with the keys on it could be stolen. Happened to us once, thankfully the .ssh/ directory was encrypted, but of course we deleted the keys on the servers anyway.

I think, that password auth with a good password, root login disabled and running fail2ban or similar might be equally if not even more secure than public key auth.