# Memory is running out and Discourse stops working
**URL:** https://meta.discourse.org/t/memory-is-running-out-and-discourse-stops-working/25038
**Category:** Bug
**Created:** [09.Февраль.2015 20:43:19 UTC](https://meta.discourse.org/t/memory-is-running-out-and-discourse-stops-working/25038 "2015-02-09T20:43:19Z")
**Posts on this page:** 20
**Page:** 2
### Author:  [@sam](https://meta.discourse.org/u/sam)
#### Post date: [09.Февраль.2015 22:30:00 UTC](https://meta.discourse.org/t/memory-is-running-out-and-discourse-stops-working/25038/21 "2015-02-09T22:30:00Z")
The big numbers don’t really help here, I need to do some heap dumping to isolate the culprit.
---
### Author:  [@DeanMarkTaylor](https://meta.discourse.org/u/DeanMarkTaylor)
#### Post date: [10.Февраль.2015 02:27:59 UTC](https://meta.discourse.org/t/memory-is-running-out-and-discourse-stops-working/25038/22 "2015-02-10T02:27:59Z")
@sam is there an `app dump` or similar you could add to get consistent dumps of data (in-memory heaps) the way you want them?
---
### Author:  [@sam](https://meta.discourse.org/u/sam)
#### Post date: [10.Февраль.2015 02:31:24 UTC](https://meta.discourse.org/t/memory-is-running-out-and-discourse-stops-working/25038/23 "2015-02-10T02:31:24Z")
Working on some memory snapshotting stuff at the moment. We have `/admin/memory_stats` now which I have been improving, but no cross process version of it and no snapshotting yet.
Its probably going to take me another day or so to get all the bits in.
---
### Author:  [@DeanMarkTaylor](https://meta.discourse.org/u/DeanMarkTaylor)
#### Post date: [10.Февраль.2015 02:56:48 UTC](https://meta.discourse.org/t/memory-is-running-out-and-discourse-stops-working/25038/24 "2015-02-10T02:56:48Z")
As you requested, excluding `redis-cli KEYS '*' | awk '{print $1}' | sort` because that was just pages of what looked like non-important info which I doubt would fit in a post here.
```
root@forum:/# redis-cli INFO
# Server
redis_version:2.8.18
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:44fa2530a359c310
redis_mode:standalone
os:Linux 3.13.0-32-generic x86_64
arch_bits:64
multiplexing_api:epoll
gcc_version:4.8.2
process_id:30
run_id:6dc9fae563192bd69ec93e510f181d94294ea147
tcp_port:6379
uptime_in_seconds:19182
uptime_in_days:0
hz:10
lru_clock:14250637
config_file:/etc/redis/redis.conf
# Clients
connected_clients:29
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:1
# Memory
used_memory:386369688
used_memory_human:368.47M
used_memory_rss:413036544
used_memory_peak:398744176
used_memory_peak_human:380.27M
used_memory_lua:35840
mem_fragmentation_ratio:1.07
mem_allocator:jemalloc-3.6.0
# Persistence
loading:0
rdb_changes_since_last_save:1189
rdb_bgsave_in_progress:0
rdb_last_save_time:1423536511
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:4
rdb_current_bgsave_time_sec:-1
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
# Stats
total_connections_received:37
total_commands_processed:180994
instantaneous_ops_per_sec:6
total_net_input_bytes:29630357
total_net_output_bytes:71038502
instantaneous_input_kbps:48.32
instantaneous_output_kbps:0.08
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:708
evicted_keys:0
keyspace_hits:94843
keyspace_misses:4208
pubsub_channels:1
pubsub_patterns:0
latest_fork_usec:29471
# Replication
role:master
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
# CPU
used_cpu_sys:38.97
used_cpu_user:40.13
used_cpu_sys_children:44.74
used_cpu_user_children:217.81
# Keyspace
db0:keys=164423,expires=2466,avg_ttl=6070294389
root@forum:/# redis-cli KEYS '*' | awk '{print $1}' | wc -l
164424
root@forum:/#
```
---
### Author:  [@riking](https://meta.discourse.org/u/riking)
#### Post date: [10.Февраль.2015 03:13:53 UTC](https://meta.discourse.org/t/memory-is-running-out-and-discourse-stops-working/25038/25 "2015-02-10T03:13:53Z")
> [@DeanMarkTaylor](#):
>
> db0:keys=164423,expires=2466,avg\_ttl=6070294389
> db0:keys=758,expires=270,avg\_ttl=127271656996
That’s a _lot_ more keys that don’t expire on yours than mine…
---
### Author:  [@DeanMarkTaylor](https://meta.discourse.org/u/DeanMarkTaylor)
#### Post date: [10.Февраль.2015 04:58:43 UTC](https://meta.discourse.org/t/memory-is-running-out-and-discourse-stops-working/25038/26 "2015-02-10T04:58:43Z")
Perhaps because I have a few more topics / posts ?
---
### Author:  [@sam](https://meta.discourse.org/u/sam)
#### Post date: [10.Февраль.2015 05:00:10 UTC](https://meta.discourse.org/t/memory-is-running-out-and-discourse-stops-working/25038/27 "2015-02-10T05:00:10Z")
Very likely, I think we have a couple of redis keys per topic for view tracking etc.
---
### Author:  [@sam](https://meta.discourse.org/u/sam)
#### Post date: [10.Февраль.2015 06:07:22 UTC](https://meta.discourse.org/t/memory-is-running-out-and-discourse-stops-working/25038/28 "2015-02-10T06:07:22Z")
I am pretty sure this will fix the issue:
[https://github.com/discourse/discourse/commit/acda6ebd605d36aa5a265fe36f3cb023bf438db0](https://github.com/discourse/discourse/commit/acda6ebd605d36aa5a265fe36f3cb023bf438db0)
Please confirm.
Note:
I will be adding a memory governer into unicorn to ensure that leaking processes get cycled, I do notice that a month in often sidekiqs can climb up to 1GB of RSS, will force a cycle after 30 minutes of uptime and more than 400mb rss (configurable)
---
### Author:  [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [10.Февраль.2015 06:10:29 UTC](https://meta.discourse.org/t/memory-is-running-out-and-discourse-stops-working/25038/29 "2015-02-10T06:10:29Z")
So this is a new bug related to the additional view tracking data you added?
---
### Author:  [@sam](https://meta.discourse.org/u/sam)
#### Post date: [10.Февраль.2015 06:22:52 UTC](https://meta.discourse.org/t/memory-is-running-out-and-discourse-stops-working/25038/30 "2015-02-10T06:22:52Z")
yes, its related to view tracking
---
### Author:  [@DeanMarkTaylor](https://meta.discourse.org/u/DeanMarkTaylor)
#### Post date: [10.Февраль.2015 07:59:12 UTC](https://meta.discourse.org/t/memory-is-running-out-and-discourse-stops-working/25038/31 "2015-02-10T07:59:12Z")
Thanks @sam!
So I have ran rebuild from the command line
```
root@forum:/var/discourse# ./launcher rebuild app
WARNING: No swap limit support
Updating discourse docker
remote: Counting objects: 11, done.
...
~Build Start: 2015-02-10T07:37:02.391921
~Build End: 2015-02-10T07:48:42.554345
```
Current system status
```
System information as of Mon Feb 9 21:50:43 EST 2015
System load: 0.1 Processes: 121
Usage of /: 63.1% of 39.25GB Users logged in: 1
Memory usage: 78% IP address for eth0:
Swap usage: 2% IP address for docker0:
```
Current version: `Discourse 1.2.0.beta6 - https://github.com/discourse/discourse version 0ce6524153b7e5545a834d2edf1a6b0de1cdcaef`
Now we play the waiting game.
---
### Author:  [@DeanMarkTaylor](https://meta.discourse.org/u/DeanMarkTaylor)
#### Post date: [10.Февраль.2015 08:04:26 UTC](https://meta.discourse.org/t/memory-is-running-out-and-discourse-stops-working/25038/32 "2015-02-10T08:04:26Z")
Memory stats for reference - might be useful when comparing to non-working state if occurs again.
(unable to run with `?full=1` as ends up reporting `Bad Gateway`).
```
forum pid:128 unicorn worker E production c config/unicorn conf rb
GC STATS:
count: 51
heap_used: 2657
heap_length: 2937
heap_increment: 280
heap_live_num: 570066
heap_free_num: 510811
heap_final_num: 0
total_allocated_object: 5559315
total_freed_object: 4989249
Objects:
TOTAL: 1080875
FREE: 510762
T_STRING: 295318
T_ARRAY: 72002
T_DATA: 66176
T_OBJECT: 47373
T_NODE: 41502
T_HASH: 25286
T_CLASS: 11559
T_REGEXP: 3680
T_ICLASS: 3382
T_MODULE: 1658
T_STRUCT: 1234
T_RATIONAL: 892
T_MATCH: 15
T_BIGNUM: 14
T_FLOAT: 11
T_FILE: 10
T_COMPLEX: 1
Process Info:
Name: ruby
State: S (sleeping)
Tgid: 128
Ngid: 0
Pid: 128
PPid: 66
TracerPid: 0
Uid: 1000 1000 1000 1000
Gid: 33 33 33 33
FDSize: 64
Groups: 33
VmPeak: 1044208 kB
VmSize: 1044204 kB
VmLck: 0 kB
VmPin: 0 kB
VmHWM: 226068 kB
VmRSS: 226068 kB
VmData: 822488 kB
VmStk: 136 kB
VmExe: 2736 kB
VmLib: 25964 kB
VmPTE: 996 kB
VmSwap: 0 kB
Threads: 7
SigQ: 0/15845
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000008301801
SigCgt: 0000000182006646
CapInh: 00000000a80425fb
CapPrm: 0000000000000000
CapEff: 0000000000000000
CapBnd: 00000000a80425fb
Seccomp: 0
Cpus_allowed: 3
Cpus_allowed_list: 0-1
Mems_allowed: 00000000,00000001
Mems_allowed_list: 0
voluntary_ctxt_switches: 1080
nonvoluntary_ctxt_switches: 2636
Classes:
Class report omitted use ?full=1 to include it
```
---
### Author:  [@DeanMarkTaylor](https://meta.discourse.org/u/DeanMarkTaylor)
#### Post date: [10.Февраль.2015 08:20:06 UTC](https://meta.discourse.org/t/memory-is-running-out-and-discourse-stops-working/25038/33 "2015-02-10T08:20:06Z")
> [@codinghorror](#):
>
> So this is a new bug related to the additional view tracking data you added?
> [@sam](#):
>
> yes, its related to view tracking
Looking though the code changes the traffic related changes occured ~5-6 days ago:
[https://github.com/discourse/discourse/commit/c9adfa65a073e3071c3083fc36397a222b1aa2dc](https://github.com/discourse/discourse/commit/c9adfa65a073e3071c3083fc36397a222b1aa2dc)
[https://github.com/discourse/discourse/commit/e105f0965c2cfad32b74d157d972f90c75e64e84](https://github.com/discourse/discourse/commit/e105f0965c2cfad32b74d157d972f90c75e64e84)
However if the following topic is related - it occurred ~10 days ago:
[https://meta.discourse.org/t/memory-usage-and-blank-pages/24511](https://meta.discourse.org/t/memory-usage-and-blank-pages/24511)
This is more of an FYI to make sure bases are covered.
---
### Author:  [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [10.Февраль.2015 08:23:02 UTC](https://meta.discourse.org/t/memory-is-running-out-and-discourse-stops-working/25038/34 "2015-02-10T08:23:02Z")
There’s also an issue where we are being sloppy with Redis and storing
1. Way, _way_ too many “permanent” keys
2. Too large keys (on the key side, the key is absurdly long)
So we need to fix that as well and I believe it is an independent issue.
---
### Author:  [@DeanMarkTaylor](https://meta.discourse.org/u/DeanMarkTaylor)
#### Post date: [10.Февраль.2015 08:23:45 UTC](https://meta.discourse.org/t/memory-is-running-out-and-discourse-stops-working/25038/35 "2015-02-10T08:23:45Z")
Just noticed this in admin on current build:
EDIT: Created separate bug:
> [@Unexpected (\[missing {{size} value\] free) in admin for uploads](https://meta.discourse.org/t/unexpected-missing-size-value-free-in-admin-for-uploads/25070):
>
> Continuing the discussion from [Low memory / ImageMagicK / White pages](https://meta.discourse.org/t/low-memory-imagemagick-white-pages/25038/35): Current version: Discourse 1.2.0.beta6 - [GitHub - discourse/discourse: A platform for community discussion. Free, open, simple. · GitHub](https://github.com/discourse/discourse) version 0ce6524153b7e5545a834d2edf1a6b0de1cdcaef After refreshing the page the it’s now correctly displaying details. [image] Perhaps the first load kicks off a task? (the traffic numbers were also all zeros the first time).
---
### Author:  [@DeanMarkTaylor](https://meta.discourse.org/u/DeanMarkTaylor)
#### Post date: [10.Февраль.2015 08:40:13 UTC](https://meta.discourse.org/t/memory-is-running-out-and-discourse-stops-working/25038/36 "2015-02-10T08:40:13Z")
> [@codinghorror](#):
>
> There’s also an issue where we are being sloppy with Redis and storing
>
> Way, way too many “permanent” keys
> Too large keys (on the key side, the key is absurdly long)
>
> So we need to fix that as well and I believe it is an independent issue.
**FYI - this issue was originally reported / experienced by me on a build from ~6 days ago, I believe this was prior to tracking being added?**
[https://github.com/discourse/discourse/commit/8689c85b54efccbccfa216c800522e5ba1a79dd1](https://github.com/discourse/discourse/commit/8689c85b54efccbccfa216c800522e5ba1a79dd1)
---
### Author:  [@Brideau](https://meta.discourse.org/u/Brideau)
#### Post date: [10.Февраль.2015 18:54:17 UTC](https://meta.discourse.org/t/memory-is-running-out-and-discourse-stops-working/25038/37 "2015-02-10T18:54:17Z")
Hey Sam, my memory usage dipped briefly after updating (assuming it’s because the server restarted), but it’s creeping upward at exactly the same rate as before (~1MB/min).
---
### Author:  [@sam](https://meta.discourse.org/u/sam)
#### Post date: [11.Февраль.2015 09:05:47 UTC](https://meta.discourse.org/t/memory-is-running-out-and-discourse-stops-working/25038/38 "2015-02-11T09:05:47Z")
What does `admin/memory_stats` show you?
---
### Author:  [@sam](https://meta.discourse.org/u/sam)
#### Post date: [11.Февраль.2015 09:08:59 UTC](https://meta.discourse.org/t/memory-is-running-out-and-discourse-stops-working/25038/39 "2015-02-11T09:08:59Z")
I spent more time on this today, I cut down our initial boot memory by 10%.
I am not noticing any extreme growth around ruby heaps they seem fine, not seeing a leak there. Also added a large object report into `?full=true` so we can catch any objects that just keep growing.
Also RSS and Peak seem to be pretty stable.
---
### Author:  [@DeanMarkTaylor](https://meta.discourse.org/u/DeanMarkTaylor)
#### Post date: [11.Февраль.2015 09:28:00 UTC](https://meta.discourse.org/t/memory-is-running-out-and-discourse-stops-working/25038/41 "2015-02-11T09:28:00Z")
> [@sam](#):
>
> I spent more time on this today, I cut down our initial boot memory by 10%.
Well done!
> [@sam](#):
>
> Also added a large object report into ?full=true so we can catch any objects that just keep growing.
I mentioned before I have been unable to run with `?full=1`.
> [@DeanMarkTaylor](#):
>
> (unable to run with ?full=1 as ends up reporting Bad Gateway).
Would you like me to update?
Or get the current status (what commands would you like me to run?), then update?
[Previous page](https://meta.discourse.org/t/memory-is-running-out-and-discourse-stops-working/25038.md?page=1)
[Next page](https://meta.discourse.org/t/memory-is-running-out-and-discourse-stops-working/25038.md?page=3)