Oops. The software powering this discussion forum encountered an unexpected problem

I suddenly got this today.

Tried ./launcher rebuild app a few times without help.

Can anybody suggest where I can check to find out the error?

Oops. The software powering this discussion forum encountered an unexpected problem

Did you see that as an error message? Where? :slight_smile:

Did your site rebuilt okay? Any errors there?

It shows up when I serve to the home page of the forum.

Rebuild is successful afaik… It starts up fine with the last line:

+ /usr/bin/docker run --shm-size=512m -d --restart=always -e LANG=en_US.UTF-8 -e RAILS_ENV=production -e UNICORN_WORKERS=2 -e UNICORN_SIDEKIQS=1 -e RUBY_GLOBAL_METHOD_CACHE_SIZE=131072 -e RUBY_GC_HEAP_GROWTH_MAX_SLOTS=40000 -e RUBY_GC_HEAP_INIT_SLOTS=400000 -e RUBY_GC_HEAP_OLDOBJECT_LIMIT_FACTOR=1.5 -e DISCOURSE_DB_SOCKET=/var/run/postgresql -e DISCOURSE_DB_HOST= -e DISCOURSE_DB_PORT= -e LETSENCRYPT_DIR=/shared/letsencrypt -e DISCOURSE_FORCE_HTTPS=true -e DISCOURSE_HOSTNAME=support.chenhsong.com -e DISCOURSE_DEVELOPER_EMAILS= -e DISCOURSE_SMTP_ADDRESS=smtp.sendgrid.net -e DISCOURSE_SMTP_PORT=587 -e DISCOURSE_SMTP_USER_NAME=apikey -e DISCOURSE_SMTP_PASSWORD=??? -e LETSENCRYPT_ACCOUNT_EMAIL= -h support-app -e DOCKER_HOST_IP=172.17.0.1 --name app -t -p 80:80 -p 443:443 -v /var/discourse/shared/standalone:/shared -v /var/discourse/shared/standalone/log/var-log:/var/log --mac-address 02:ad:4c:fe:ac:75 local_discourse/app /sbin/boot
f0795eafda2ebc91dac07c5c38040a0b14eda0a9b86ce90b6bc79c51acb519ba

syslog does have a very strange set of errors with a wrong date stamp (it happens during the past couple of days):

Feb  3 01:35:01 support-app rsyslogd: action 'action-0-builtin:omfile' (module 'builtin:omfile') message lost, could not be processed. Check for additional error messages before this one. [v8.1901.0 try https://www.rsyslog.com/e/2027 ]
Feb  3 01:45:01 support-app rsyslogd: file '8' write error: No space left on device [v8.1901.0 try https://www.rsyslog.com/e/2027 ]
Feb  3 01:45:01 support-app rsyslogd: action 'action-0-builtin:omfile' (module 'builtin:omfile') message lost, could not be processed. Check for additional error messages before this one. [v8.1901.0 try https://www.rsyslog.com/e/2027 ]
Feb  3 01:45:01 support-app rsyslogd: file '8' write error: No space left on device [v8.1901.0 try Feb  4 04:38:13 support-app rsyslogd:  [origin software="rsyslogd" swVersion="8.1901.0" x-pid="371" x-info="https://www.rsyslog.com"] start

This means that your disk is full. You can delete some backups and try

 cd /var/discourse
./launcher cleanup
1 Like

One of the first things I checked. No, disk is not full. Only 65% usage max on all volumes.

But I have made sure that the disk is clean with lots of room. Then rebuilt a couple of times, which all succeeded. But still this error.

EDIT: Restoring to my previous backup (two betas ago) and it all works fine. So something has to do with this beta.

1 Like

Are you certain?

I’m awake in the middle of the night due to insomnia so this might be one of my less intelligent posts, but if the datestamp suggests the problem wasn’t recent and the log entry does not accurately describe your current situation, my first thought would be the log entry is old & unrelated.

At the very least, I would be curious whether additional such log entries continue to be generated!

As an aside… could you check the device’s current date & time? If the timedatectl terminal command works, great! Otherwise, the date command should work on just about any Linux box.

 

Have you done this?

Got hit by this error after update just now :frowning:

A disk full error can also occur when you run out of inodes. What does df -i say?

Would /logs be more revealing in this case?

I locked myself out of my test site with a similar error recently and that pointed me in the direction of a particular plugin conflict.

We (krita-artists.org) referred to this thread and it helped us in getting the site back up -

2 Likes

Just in reply to a few comments above:

  • df -i not even close. <10%

  • logs no other errors in syslog and syslog.1 etc.

The issue in the topic quoted a post or two up was having an old plugin still in your app.yml. Could this be the same for you too?

Could you list your plugins here for us to see?

1 Like

Could be this thing!

I have this in my app.yml

git clone https://github.com/merefield/discourse-topic-previews.git -b "theme_sidecar"

This is obviously an old repo with a special branch!

Will try with the latest one:

git clone https://github.com/paviliondev/discourse-topic-previews-sidecar

EDIT: It works!!! :champagne:

Thanks a great deal to @raghukamath

4 Likes