Looks like a bug with backup restore … somehow your instance is stuck thinking it is in the middle of a backup/restore operations.
@zogstrip we need some guarantees this can not happen, recommend you set the key with a 1 minute expiry and have the job performing work extend the expiry by a minute every 30 seconds.
That means that if you kill 9 the whole business the key will go away.
Workaround to fix issue:
./launcher ssh app
irb > BackupRestore.mark_as_not_running!
Same fix should apply to sidekiq pausing, its a very hard problem to solve if it creeps up.
Bonus points, I was able to debug this by looking at logs when I ran a backup job