Crashing with out of memory error when opening a topic

[BDub]

I have the latest version of discourse running on a 2GB / 1CPU / 40GB SSD Ubuntu 14.04.5 DigitalOcean droplet. It has been fine for the longest time, and I recently resized the old plan that was the same specs from $20/mo. to this one that is $10/mo. After resizing everything it was running and very snappy (better than before I’d say), but now it will load the main site but if I click on any topics it won’t load and ends up crashing the instance.

Any advice on where to go digging for the right logs?

[BDub]

I can get to the error logs from the admin panel if that’s helpful:

Here’s the details of the deadlock; recursive locking error
https://pastebin.com/aL3iZkZM

[ssvenn]

“failed to allocate memory” sounds like maybe the database is tuned to use too much ram from the previous plan, my empty test instance uses about 1gb on a 4gb centos vm

[root@discocent discourse]# free
              total        used        free      shared  buff/cache   available
Mem:        3881412      912764      413192       33168     2555456     2401820
Swap:       2097148        1544     2095604

Does the droplet have any swap configured? Have you tried rebuilding the app container?
Check for out-of-memory errors in dmesg or /var/log/syslog too

Try to tail -f /var/discourse/shared/standalone/log/rails/production.log when opening a topic and look for error messages there maybe, that log is pretty verbose.

[BDub]

Yes I tried rebuilding which finishes fine and loads the site.

Memory available should be the same as before, 2GB.

It does appear something is consuming up all of the memory… and there are lots of threads on here that say discourse can run on the smallest droplet. My site is basically a ghost town as well… so nothing should be consuming so much RAM.

Oh here we go…

this is just after rebooting the disc, running free -m every second.

root@test:~# free -m
             total       used       free     shared    buffers     cached
Mem:          2001        756       1245         22         77        271
-/+ buffers/cache:        407       1594
Swap:            0          0          0
root@test:~# free -m
             total       used       free     shared    buffers     cached
Mem:          2001        878       1123         28         92        289
-/+ buffers/cache:        497       1504
Swap:            0          0          0
root@test:~# free -m
             total       used       free     shared    buffers     cached
Mem:          2001        961       1040         28         99        330
-/+ buffers/cache:        532       1469
Swap:            0          0          0
root@test:~# free -m
             total       used       free     shared    buffers     cached
Mem:          2001       1022        978         28         99        389
-/+ buffers/cache:        534       1467
Swap:            0          0          0
root@test:~# free -m
             total       used       free     shared    buffers     cached
Mem:          2001       1328        673         28         99        456
-/+ buffers/cache:        772       1229
Swap:            0          0          0
root@test:~# free -m
             total       used       free     shared    buffers     cached
Mem:          2001       1744        257         32         99        467
-/+ buffers/cache:       1177        824
Swap:            0          0          0
root@test:~# free -m
             total       used       free     shared    buffers     cached
Mem:          2001       1938         62         35         77        448
-/+ buffers/cache:       1413        588
Swap:            0          0          0

I’ll try digging into the various logs, thanks!

[ssvenn]

Check out High-CPU / High memory on fresh discourse install and Discourse 2.2.0.beta8 Release Notes

Important Note: Expect high CPU usage for a few hours after this upgrade due to image re-optimization. Sites with large numbers of images may see high CPU usage for a longer period of time.

It could be that you need to tune the number of unicorn workers and reduce the cache size for the database for that initial optimization to work.

[Falco]

You need to configure swap.

Try running this to configure swap:

https://github.com/discourse/discourse_docker/blob/master/discourse-setup#L141-L147

[BDub]

Thanks @Falco that did the trick! It appears to be stable now

# free -m
             total       used       free     shared    buffers     cached
Mem:          2001       1922         79         28         49        306
-/+ buffers/cache:       1566        435
Swap:         2047          0       2047