What are the security risks if my server with github ssh access key is hacked?

I connect to my github repo using ssh keys which I have whitelisted in GitHub. I do pull/push to this repo from my linux server which is hosted in AWS.

If a hacker gains access to this AWS server, what all damage can he do to my code in github? Can he/she remove my repos permanently from GitHub?