Use GPG smartcard and gpg-agent to protect multiple SSH keys

I have a Yubikey acting as a GPG smartcard. SSH is configured to use the smart card socket for authentication, and authenticating with the GPG key with Authentication capabilities work fine.

ssh-add -l 2048 SHA256:ey5VPl70RKvXSdaon6ugxiO1ZrzqxJwz7VWZM7zvN/c cardno:000607329647 (RSA) 

I have some additional SSH keys that are used for SSH authentication to various servers. I can add these to the ssh agent with ssh-add somekey, but then I need to provide a passphrase per key. Is there a way to use the smartcard to protect these keys instead?

The desired end state is that multiple SSH keys are stored encrypted on disk, and the smartcard and gpg-agent is used to unlock and cache them.