Hi,
we’re facing huge transaction loads on our storage from time to time. We couldn’t find any schedule or time-pattern for this event but it happens at least once a day. Even the duration changes from 10min to several hours.
While these loads our whole installation behaves slight strange, e.g. reading topics won’t be recognised so they still appear in “New” and/or “Unread”.
It seems like Discourse is moving huge amounts of files. Especially READ
- Operations are increasing. We’ve already checked if external traffic is increasing as well but it doesn’t. Only Discourse <-> Storage traffic is affected.
We’ve recognized this behaviour the first time after upgrading from Discourse 2.4.0.beta9 to 2.4.0.beta10 but we’re not sure if it already happened before. We’re now running 2.5.0.beta4.
Our Discourse installation is running in a Azure environment with a Premium Storage attached via SMBv3 which normally runs pretty well.
Can someone explain what is going on? At first we’ve suspected the sidekiq-job MigrateUploadScheme
but if this job is responsible for these transactions we should see these high loads way more often than we do. Besides we haven’t found any other job which could be responsible.
Because of “burst IOPS” you can see this peek at ~800k Transactions/30min. After these credits are used up it gets throttled down to ~250k Transaction/30min. So please don’t mind this peek as it’s just a limited/credited bonus of the azure storage tier.
Normally we have 5k - 40k Transaction per 30min.
At this point we don’t know where to look at and any idea/hint is appreciated.
Kind Regards
Sascha