Totally blank page after upgrading


#1

Hello everyone,

I’m having a problem with my Discourse instance after upgrading it via the UI. It was running fine back then. The upgrade was successful, however, reloading the page has resulted in a blank page since then. I found a relevant thread at Blank Page after Upgrading due to old Tagging Plugin and tried to rebuild the app, however, things still haven’t recovered.

Here is the log of the launcher after I rebuilt the app:

run-parts: executing /etc/runit/1.d/00-ensure-links
run-parts: executing /etc/runit/1.d/anacron
run-parts: executing /etc/runit/1.d/cleanup-pids
Cleaning stale PID files
run-parts: executing /etc/runit/1.d/copy-env
run-parts: executing /etc/runit/1.d/enable-brotli
run-parts: executing /etc/runit/1.d/ensure-web-nginx-read
Started runsvdir, PID is 31
sh: echo: I/O error
ok: run: redis: (pid 41) 0s
ok: run: postgres: (pid 40) 0s
                _._                                                  
           _.-``__ ''-._                                             
      _.-``    `.  `_.  ''-._           Redis 3.0.6 (00000000/0) 64 bit
  .-`` .-```.  ```\/    _.,_ ''-._                                   
 (    '      ,       .-`  | `,    )     Running in standalone mode
 |`-._`-...-` __...-.``-._|'` _.-'|     Port: 6379
 |    `-._   `._    /     _.-'    |     PID: 41
  `-._    `-._  `-./  _.-'    _.-'                                   
 |`-._`-._    `-.__.-'    _.-'_.-'|                                  
 |    `-._`-._        _.-'_.-'    |           http://redis.io        
  `-._    `-._`-.__.-'_.-'    _.-'                                   
 |`-._`-._    `-.__.-'    _.-'_.-'|                                  
 |    `-._`-._        _.-'_.-'    |                                  
  `-._    `-._`-.__.-'_.-'    _.-'                                   
      `-._    `-.__.-'    _.-'                                       
          `-._        _.-'                                           
              `-.__.-'                                               
41:M 15 Jun 22:02:09.879 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
41:M 15 Jun 22:02:09.879 # Server started, Redis version 3.0.6
41:M 15 Jun 22:02:09.880 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
41:M 15 Jun 22:02:09.880 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
41:M 15 Jun 22:02:09.890 * DB loaded from disk: 0.010 seconds
41:M 15 Jun 22:02:09.890 * The server is now ready to accept connections on port 6379
rsyslogd: command 'KLogPermitNonKernelFacility' is currently not permitted - did you already set it via a RainerScript command (v6+ config)? [v8.16.0 try http://www.rsyslog.com/e/2222 ]
rsyslogd: imklog: cannot open kernel log (/proc/kmsg): Operation not permitted.
rsyslogd: activation of module imklog failed [v8.16.0 try http://www.rsyslog.com/e/2145 ]
rsyslogd: Could not open output pipe '/dev/xconsole':: No such file or directory [v8.16.0 try http://www.rsyslog.com/e/2039 ]
Server listening on 0.0.0.0 port 22.
Server listening on :: port 22.
2016-06-15 22:02:10 UTC [60-1] LOG:  database system was shut down at 2016-06-15 22:01:09 UTC
2016-06-15 22:02:10 UTC [60-2] LOG:  MultiXact member wraparound protections are now enabled
2016-06-15 22:02:10 UTC [40-1] LOG:  database system is ready to accept connections
2016-06-15 22:02:10 UTC [64-1] LOG:  autovacuum launcher started
supervisor pid: 42 unicorn pid: 66

I did see the following error in production.log:

Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED) subscribe failed, reconnecting in 1 second. Call stack ["/var/www/discourse/vendor/bundle/ru
by/2.3.0/gems/redis-3.3.0/lib/redis/client.rb:345:in `rescue in establish_connection'", "/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/redis-3.3.0/lib/redi
s/client.rb:331:in `establish_connection'", "/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/redis-3.3.0/lib/redis/client.rb:101:in `block in connect'", "/va
r/www/discourse/vendor/bundle/ruby/2.3.0/gems/redis-3.3.0/lib/redis/client.rb:293:in `with_reconnect'", "/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/redi
s-3.3.0/lib/redis/client.rb:100:in `connect'", "/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/redis-3.3.0/lib/redis/client.rb:364:in `ensure_connected'", "
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/redis-3.3.0/lib/redis/client.rb:221:in `block in process'", "/var/www/discourse/vendor/bundle/ruby/2.3.0/gems
/redis-3.3.0/lib/redis/client.rb:306:in `logging'", "/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/redis-3.3.0/lib/redis/client.rb:220:in `process'", "/var
/www/discourse/vendor/bundle/ruby/2.3.0/gems/redis-3.3.0/lib/redis/client.rb:134:in `block in call_loop'", "/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/r
edis-3.3.0/lib/redis/client.rb:280:in `with_socket_timeout'", "/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/redis-3.3.0/lib/redis/client.rb:133:in `call_loop'", "/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/redis-3.3.0/lib/redis/subscribe.rb:43:in `subscription'", "/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/redis-3.3.0/lib/redis/subscribe.rb:12:in `subscribe'", "/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/redis-3.3.0/lib/redis.rb:2760:in `_subscription'", "/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/redis-3.3.0/lib/redis.rb:2138:in `block in subscribe'", "/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/redis-3.3.0/lib/redis.rb:58:in `block in synchronize'", "/usr/local/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize'", "/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/redis-3.3.0/lib/redis.rb:58:in `synchronize'", "/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/redis-3.3.0/lib/redis.rb:2137:in `subscribe'", "/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/message_bus-2.0.0.beta.11/lib/message_bus/backends/redis.rb:304:in `global_subscribe'", "/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/message_bus-2.0.0.beta.11/lib/message_bus.rb:504:in `global_subscribe_thread'", "/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/message_bus-2.0.0.beta.11/lib/message_bus.rb:456:in `block in new_subscriber_thread'"]

Could you guys advise me on a way to debug and resolve the problem in this case? Thanks!


(Jeff Atwood) #2

Are you out of disk space

df -m

or out of memory?

free -m

a 2GB swapfile is required to successfully upgrade Discourse these days so make sure you have that!


#3

Thanks @codinghorror for your quick suggestion. I just configured a 4G swap file for the instance and re-built but the site is still blank unfortunately. Anything else I should check that you can think of?

free -m
total       used       free     shared    buffers     cached
Mem:          3953       2982        971         63        124       1591
-/+ buffers/cache:       1266       2687
Swap:         4095          0       4095

df -m
Filesystem                1M-blocks  Used Available Use% Mounted on
udev                           1973     1      1973   1% /dev
tmpfs                           396     1       395   1% /run
/dev/disk/by-label/DOROOT     60346 14368     42891  26% /
none                              1     0         1   0% /sys/fs/cgroup
none                              5     0         5   0% /run/lock
none                           1977     1      1977   1% /run/shm
none                            100     0       100   0% /run/user
none                          60346 14368     42891  26% /var/lib/docker/aufs/mnt/8636397e98efa876570218ac96e25252db7dac760c701ae063922be76bfc22d2
shm                              64     1        64   1% /var/lib/docker/containers/af992ff86385f7066645563e0f76301d35777cb607f69e3f953721051913a303/shm

(Jeff Atwood) #4

I would remove all plugins that aren’t first party (from us) in your app.yml and then rebuild. Do you get any errors after rebuilding?


#5

I have removed all the third-party plugins and used only the ones from discourse in app.yml. Rebuilding the app still results in a blank page. There are no errors in the log. Still see the problem of connecting to Redis in the production.log as well.

hooks:
  after_code:
    - exec:
        cd: $home/plugins
        cmd:
          - git clone https://github.com/discourse/docker_manager.git
          - git clone https://github.com/discourse/discourse-akismet.git

(Jeff Atwood) #6

Did you have any JavaScript customizations in admin, customize? What does the F12 console in your browser say on the white page?


#7

Thanks a lot @codinghorror! You really pointed me towards the right direction. F12 showed me a lot of JavaScript errors. Upon some investigation, I found out that the reason is actually due to CloudFlare’s minification feature. So I simply disabled it and purged the cache. The site is now back to normal!