We have Discourse pumping out to our StatsD regularly, courtesy of the ‘nunes’ gem, and have noticed that the PeriodicalUpdates job causes our database to keep growing considerably each time it runs. In an earlier version of Discourse, our PostgreSQL database size ballooned over time from 1GB to 400GB+, even with autovacuuming enabled.
When overlaid in Graphite, the database size appears to step up by approximately 200MB every 15 minutes as the PeriodicalUpdates job runs and the current instance stands at around 16GB.
Naturally we’d prefer if the database didn’t keep growing without limit. Is there anyway to control/optimise this behaviour? Is there a bug somewhere in the code that is causing the stepping and will it plateau?
Before we reached around 27GB before deciding to kill the Sidekiq instance completely, but obviously this is detrimental to the performance of the server over time as maintenance jobs will not operate. However, if we have to balance between an evergrowing PostgreSQL dataset and something that is manageable, we’ll kill Sidekiq again.
We’ve already placed a retry: false option in the PeriodicalUpdate job to stop it repeatedly hammering our server, but even this doesn’t stop the stepping growth.
Any tips or help to stop this or at least control it better?