White screen after deleting custom emoji

Hi everyone,

I have a problem with a Discourse install where the screen is completely blank after I deleted a custom emoji from the admin panel.

I have tried running ./launcher rebuild app and it didn’t work, I’m still getting the white screen :frowning:

In the nginx log I got a whole series of errors like these as the last entries before fail:

[error] 212#212: *9 connect() failed (111: Connection refused) while connecting to upstream, client: 555.555.555.55, server: _, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:3000/", host: xxx

Does anyone know what’s going on please?

Any help would be very much appreciated!

Thanks!!

p.s. I am running Discourse Docker on a Digital Ocean Droplet

Disk full? Out of memory? Swap file set up? Docker installed properly? I think you have some other problem. Check disk, memory, and docker info.

Thanks, I’ve checked and all seems ok, talking to Digital Ocean to see if they can spot anything wrong.

I will report back :slight_smile:

Update:

Docker seems fine:

# docker info
Containers: 1
 Running: 1
 Paused: 0
 Stopped: 0
Images: 4
Server Version: 17.03.0-ce
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 7
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 977c511eda0925a723debdc94d09459af49d082a
runc version: a01dafd48bc1c7cc12bdb01206f9fea7dd6feb70
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.4.0-75-generic
Operating System: Ubuntu 16.04.2 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.953 GiB
Name: ubuntu-o
ID: Q34X:7PTI:SI67:4XJX:4YGR:EJGC:MR74:N3DT:SWBY:6WKX:YMJK:SHDO
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Docker stats:

CONTAINER           CPU %               MEM USAGE / LIMIT       MEM %               NET I/O             BLOCK I/O           PIDS
e97b6427d2cf        5.98%               1.017 GiB / 1.953 GiB   52.07%              161 kB / 216 kB     144 MB / 15.6 MB    65

Memory usage:

# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            981M     0  981M   0% /dev
tmpfs           201M  3.2M  197M   2% /run
/dev/vda1        30G  9.9G   20G  34% /
tmpfs          1001M  596K 1000M   1% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs          1001M     0 1001M   0% /sys/fs/cgroup
none             30G  9.9G   20G  34% /var/lib/docker/aufs/mnt/99c0f2807a ... 2
shm              64M  4.0K   64M   1% /var/lib/docker/containers/e ... a/shm
tmpfs           201M     0  201M   0% /run/user/0

I’m really tired and can’t see anything wrong and I’m not sure how to test that the swap file is set up, it should be though! As I say, everything was working fine and nothing was changed except deleting the emoji.

From my local machine:

$ curl -i <myip>
HTTP/1.1 301 Moved Permanently
Server: nginx/1.11.6
Date: Wed, 26 Apr 2017 00:03:36 GMT
Content-Type: text/html
Content-Length: 185
Location: https://<myserver>
Connection: keep-alive

<html>
<head><title>301 Moved Permanently</title></head>
<body bgcolor="white">
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx/1.11.6</center>
</body>
</html>

Does this help any?

Thanks again!! :slight_smile:

OK, a bit more awake this morning, here’s the log where the 500 errors started and as I said, it was literally after I deleted a custom emoji, of course there still could be some other problem somewhere that is causing this, but now I am pretty convinced this was the trigger event:

Started GET "/admin/customize/emojis.json?_=1493157113083" for 213.205.253.56 at 2017-04-25 21:52:21 +0000
Processing by Admin::EmojisController#index as JSON
  Parameters: {"_"=>"1493157113083"}
Completed 200 OK in 2ms (Views: 0.1ms | ActiveRecord: 0.0ms)
Started DELETE "/admin/customize/emojis/ord_88" for 213.205.253.56 at 2017-04-25 21:52:25 +0000
Processing by Admin::EmojisController#destroy as */*
  Parameters: {"id"=>"ord_88"}
Completed 200 OK in 80ms (Views: 4.7ms | ActiveRecord: 13.5ms)
Started GET "/latest.json?order=default&_=1493157113084" for 213.205.253.56 at 2017-04-25 21:52:28 +0000
Processing by ListController#latest as JSON
  Parameters: {"order"=>"default", "_"=>"1493157113084"}
Completed 200 OK in 341ms (Views: 0.6ms | ActiveRecord: 45.9ms)
Started GET "/" for 213.205.253.56 at 2017-04-25 21:52:30 +0000
Processing by ListController#latest as HTML
Completed 500 Internal Server Error in 55ms (ActiveRecord: 15.1ms)
NoMethodError (undefined method `url' for nil:NilClass)
/var/www/discourse/app/models/emoji.rb:98:in `block (2 levels) in load_custom'
Started GET "/" for <myip> at 2017-04-25 21:52:34 +0000
Processing by ListController#latest as HTML
Completed 500 Internal Server Error in 85ms (ActiveRecord: 14.8ms)
NoMethodError (undefined method `url' for nil:NilClass)
/var/www/discourse/app/models/emoji.rb:98:in `block (2 levels) in load_custom'

From this moment onwards there have been problems and the site is unusable :frowning:

Does anyone know what has happened here and how to fix it please?

Thank you!!

OK problem fixed following the advice given here:

In short there was a broken emoji, deleting the custom emoji and clearing the emoji cache solved the problem.

Thanks everyone, we’re back up and running again, hope this helps others :slight_smile:

2 Likes

I’ve have a look through the code path again to see if I missed something. You shouldn’t have to go through all the trouble just to remove a custom emoji.

4 Likes

Thanks for this - please let me know if you need any further information or have any questions and I will try and help :slight_smile:

I can’t really tell what the root cause is here but I’ve added preventive measures to not blow up the whole site just because one custom emoji is broken.

https://github.com/discourse/discourse/commit/e61959e6a8c8c7095ac69ec4a96486a7dcb8b511

4 Likes

That’s good enough for me - thank you for your efforts!!

All the best
Kel

1 Like

This topic was automatically closed after 3 days. New replies are no longer allowed.