Also, if you do have them in app.yml and have the backup on S3, then you need only the yml file to set up a new server and restore the database.
If it’s possible, sharing a topic where you have this problem will give someone trying to see what’s going on a lot more information. If your site is public, then the only thing you need to hide are your keys, which are already hidden.