Can’t do a reboot right now, will report back once I have a chance to do it. It will be at least a few days.
Should I be alarmed? It’s fine if I have to allocate more resources
Yes, you should be alarmed.
You barely have any leeway and there is almost no room for disk caching either.
But before allocating more resources you should try to find out what is causing this quite extraordinary memory usage.
1 Like
Can’t do much right now, I’m on the move. I’ll be back at it this weekend and definitely try to figure out what is going on. Thanks for the pointer 
3 Likes
Looks like it was just an overdue reboot that was causing this. Managed to reboot the server and the numbers look much much better as of now
root@discourse:~# free
total used free shared buff/cache available
Mem: 3927308 977624 2246208 42880 703476 2646836
Swap: 2097148 0 2097148
2 Likes
It might be useful, if you find yourself in a similar situation again, to ask discourse what processes are using memory - the output at
https://example.com/admin/upgrade#/processes
is sorted in order of RAM usage. That will, I think, only show processes running with the container. At the command line, you could use
ps aux | sort -nr -k 4 | head -22
or similar, to see the usage of all processes including those in all containers.
If a reboot solves a memory starvation problem, there’s a fair chance you have some sort of runaway process somewhere which is going to ramp up (perhaps slowly) until it causes trouble.
Edit: Hmm, I see I mention reading off processes according to RAM usage (RSS). That can be useful, but in this case we’re actually more interested in virtual memory usage: we should be sorting by the VSZ column, column 5, for that.
2 Likes
This discourse instance was first setup many years ago, so perhaps before the swapfile check was done. I’ve gone ahead and applied those lines of code manually and swapfile is now created. Thank you.
To make docker use the swapfile, do I need to apply this? Or rather, what is the point of this recommendation?
I found this that discusses what vm.overcommit_memory is, but it doesn’t make sense to me why any such change is needed:
This has to do with running Redis, not with upgrading Discourse.
I think it’s more than that - let me try to explain. The recommendation comes from redis, and redis recommends it because forking a process demands a lot of virtual memory, and redis forks to do background saves, and yet the virtual memory claimed is never going to be needed.
This is typical for many unix applications: they will fork, but will not need to double their memory usage. Because it’s typical for many, and because this is a kernel setting which changes the behaviour for all processes in all containers, it may well convert a failure into a success when virtual memory is under pressure.
On the small cheap instances which many of us use, virtual memory is often under pressure. And especially so during upgrades or rebuilds.
So changing this setting might well relate to whether an upgrade succeeds or fails, especially if there’s recently been a change which increases the demand on virtual memory.
As-shipped, the kernel will reject allocations which it can’t satisfy. With this tweak, it will accept those allocations, and failure might be averted, or it might happen later when the allocation becomes usage.
If your total of RAM and swap is big enough, you’ll never need to change this setting. If your total is not big, changing it might help.
2 Likes
The free command will tell you how much swap is configured, and how much of it is in use. I think you find it will be in use as soon as you ran those commands.
It’s to increase the amount of virtual memory available. (That is, the sum of RAM and swap.) If you run out of RAM, you start to suffer performance problems. But if you run out of virtual memory, processes will fail to start or will die or be killed. It gets brutal.
Those of us with both small RAM and small disk may not be free to add loads of swap, but 2G seems to be a good minimum. (If you had 16G of RAM you might not need any swap, but that’s another story. It’s the sum of the two which matters, when the problem is things failing.)
2 Likes
Aha. I think that I sort of understood that, but that is a very good explanation. So that’s why I’ve been setting that on every installation that I’ve done vm I’ve created for clients.
I wonder if we should make discourse-setup change these settings when it creates swap.
1 Like
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.