How bad is rsync with no-password sudo?

I need to backup files with preserved attributes from a source workstation to a LAN server (both on Linux Mint, the server is running sshd and Samba). One of the solutions which preserves files’ source attributes is to run rsync over ssh, something like that:

rsync -a --rsync-path="sudo rsync" -e ssh /media/user1/source user2@server:/media/user2/destination/ 

However for this to work as expected, rsync needs to be added to the sudoer list as NOPASSWD on the server side:

user2 ALL=NOPASSWD:/usr/bin/rsync 

This setup makes backing up with attribute preservation work fine. But how secure it is to have a passwordless rsync on the server? Is it inviting problems? Or I’m thinking too much? Our main security concern is unauthorised copying of sensitive data by a motivated hacker. Clearly if you can sudo rsync you can send any file from the server to an arbitrary internet location.

What are your thoughts? If it’s that bad, any suggestions on a LAN backup which would preserve attributes from the source on the LAN workstation?