Does using S3 for uploads mean redownloading all files when creating backups?

That will happen only when you enable the include_s3_uploads_in_backups site setting. It’s disabled by default.