The discussion revolves around improving disk space calculation performance for upload-heavy Discourse instances. Overgrow initially proposed replacing the slow du command with a faster df-based approximation to address performance issues in the admin dashboard, submitting a PR here. tgxworld suggested two alternatives: using a direct database sum of Upload and OptimizedImage file sizes or introducing a background job to cache the total upload size in Redis. Overgrow acknowledged the feedback, noting du took over a minute on HDDs and ~20 seconds on SSDs, and later submitted a revised PR here incorporating the suggested improvements.
The ‘du’ command, which is used to retrieve disk space used by uploads, was causing performance issues for my admin dashboard. Yes, we have a LOT of image uploads. Rather than disabling it altogether (we have grafana/prometheus dashboard after all), I decided to replace it with a much faster very rough approximation using ‘df’. This change is of course selectable by the admin with the default being ‘du’.
I have submitted a PR for this change. It is my first PR, so please go easy on me :))
Do you have a rough idea how long the du was taking for you? I’m not really keen on this approach to resolve the performance issue and I think there are two alternatives:
Just use Upload.sum(:filesize).to_i + OptimizedImage.sum(:filesize).to_i to determine uploads_used_bytes like how we do for external stores.
Introduce a background job that periodically recalculates the bytes used by uploads and cache it in Redis.
Personally, I’m more keen on (1) as it is a much simpler solution. We do lose some accuracy but we don’t need 100% accuracy here. Still it will be much better than the approximation we get from df.