Automatic backups are a hit or miss

We’ve configured backup to run every 3 days (however, that sometimes happens and other times not) Sidekiq had completed the ScheduleBackup task 5 Hours ago but no backup was created.

Reading earlier topic, I found out people having this issue if they’re low on storage or memory however, our forum is relatively happy in terms of specs (8GB RAM and 512GB Solid Storage with Quad Core)

Manually creating backup works without an error however, automatic backup sometime works and sometimes Job is triggered but no backup is created. :frowning:

Is there anything in the logs (any of them)? Where are all the Discourse logs

2 Likes

Is there anything specific that I should be looking for?
I was searching the logs for last few days with “backup” as the search keyword and Production.log only had entries for the manual backups I triggered or the times when I visited backup page:
e.g.
Started GET "/admin/backups/status.json?_=1529382308307" for xxx.xxx.xx.xxx at 2018-06-19 05:31:30 +0000
I don’t think that tells much about the situation.

in the logs web UI, there are a lot of these:
TypeError: Discourse._registerPluginCode is not a function at https://discourse.orng.co/u/activate-account/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx:46:21
but those are from a 11th June (backup ran successfully on 14th though!

Are you seeing any symptoms similar to ours?
https://meta.discourse.org/t/sidekiq-restarts-missed-backups-admin-incomplete/90180

My backups aren’t failing!

They are either happening successfully or not happening at all. However, sidekiq seems like triggering the job at right times but somehow something in between is not happy.

I’m still trying to understand what may be the potential culprit because in the last 15 days, it ran successfully only 2 times (non-consecutively).

Also, Check for updates is working fine and I’m able to update from docker manager as well as through SSH.

We don’t always get a failed backup either. But the automated ones do “skip”, similar to what you’re seeing.

Anything interesting in /logs, and /sidekiq?

1 Like

in /logs, Only some register plugin code related warnings (which predate the issue)
And in /sidekiq, everything seems happy (27 failed jobs which predate the issue and are related to a telegram notification plugin which was since removed)

Is your sidekiq running out of memory and restarting/missing heartbeats?

That ours restarts every 30 min on the dot seems very unusual to me.

I don’t think so! never had any alerts regarding that anywhere! How do you test if it is out of memory (mine has a peak of only 40 MB memory)

/logs shows messages:

Sidekiq is consuming too much memory (using: 2307.83M) for 'forums.jag-lovers.com', restarting

and

Sidekiq heartbeat test failed, restarting

Looking at /sidekiq/scheduled you can see sidekiq working through the backlog after a missed heartbeat.

Nothing as such in sidekiq

No backlog either:

So I decided to give it another shot!
Reset every setting related to backups to default

Then after the 1 week time, a backup was created.
Then I amended settings to what we wanted (daily Back up including attachments) and it worked!

I don’t know what made it work but I’m glad whatever it was, it’s back to business.

This topic can be closed! Thanks