Oops. The software powering this discussion forum encountered an unexpected problem

[schungx (Stephen Chung)]

I suddenly got this today.

Tried ./launcher rebuild app a few times without help.

Can anybody suggest where I can check to find out the error?

[maiki]

Oops. The software powering this discussion forum encountered an unexpected problem

Did you see that as an error message? Where?

@schungx

Tried ./launcher rebuild app a few times without help.

Did your site rebuilt okay? Any errors there?

[schungx (Stephen Chung)]

@maiki

Did you see that as an error message? Where?

It shows up when I serve to the home page of the forum.

Rebuild is successful afaik… It starts up fine with the last line:

+ /usr/bin/docker run --shm-size=512m -d --restart=always -e LANG=en_US.UTF-8 -e RAILS_ENV=production -e UNICORN_WORKERS=2 -e UNICORN_SIDEKIQS=1 -e RUBY_GLOBAL_METHOD_CACHE_SIZE=131072 -e RUBY_GC_HEAP_GROWTH_MAX_SLOTS=40000 -e RUBY_GC_HEAP_INIT_SLOTS=400000 -e RUBY_GC_HEAP_OLDOBJECT_LIMIT_FACTOR=1.5 -e DISCOURSE_DB_SOCKET=/var/run/postgresql -e DISCOURSE_DB_HOST= -e DISCOURSE_DB_PORT= -e LETSENCRYPT_DIR=/shared/letsencrypt -e DISCOURSE_FORCE_HTTPS=true -e DISCOURSE_HOSTNAME=support.chenhsong.com -e DISCOURSE_DEVELOPER_EMAILS= -e DISCOURSE_SMTP_ADDRESS=smtp.sendgrid.net -e DISCOURSE_SMTP_PORT=587 -e DISCOURSE_SMTP_USER_NAME=apikey -e DISCOURSE_SMTP_PASSWORD=??? -e LETSENCRYPT_ACCOUNT_EMAIL= -h support-app -e DOCKER_HOST_IP=172.17.0.1 --name app -t -p 80:80 -p 443:443 -v /var/discourse/shared/standalone:/shared -v /var/discourse/shared/standalone/log/var-log:/var/log --mac-address 02:ad:4c:fe:ac:75 local_discourse/app /sbin/boot
f0795eafda2ebc91dac07c5c38040a0b14eda0a9b86ce90b6bc79c51acb519ba

syslog does have a very strange set of errors with a wrong date stamp (it happens during the past couple of days):

Feb  3 01:35:01 support-app rsyslogd: action 'action-0-builtin:omfile' (module 'builtin:omfile') message lost, could not be processed. Check for additional error messages before this one. [v8.1901.0 try https://www.rsyslog.com/e/2027 ]
Feb  3 01:45:01 support-app rsyslogd: file '8' write error: No space left on device [v8.1901.0 try https://www.rsyslog.com/e/2027 ]
Feb  3 01:45:01 support-app rsyslogd: action 'action-0-builtin:omfile' (module 'builtin:omfile') message lost, could not be processed. Check for additional error messages before this one. [v8.1901.0 try https://www.rsyslog.com/e/2027 ]
Feb  3 01:45:01 support-app rsyslogd: file '8' write error: No space left on device [v8.1901.0 try Feb  4 04:38:13 support-app rsyslogd:  [origin software="rsyslogd" swVersion="8.1901.0" x-pid="371" x-info="https://www.rsyslog.com"] start

[pfaffman (Jay Pfaffman)]

@schungx

write error: No space left on device [

This means that your disk is full. You can delete some backups and try

 cd /var/discourse
./launcher cleanup

[schungx (Stephen Chung)]

One of the first things I checked. No, disk is not full. Only 65% usage max on all volumes.

But I have made sure that the disk is clean with lots of room. Then rebuilt a couple of times, which all succeeded. But still this error.

EDIT: Restoring to my previous backup (two betas ago) and it all works fine. So something has to do with this beta.

[khenmu (John Sweeney)]

@schungx

(it happens during the past couple of days)

Are you certain?

I’m awake in the middle of the night due to insomnia so this might be one of my less intelligent posts, but if the datestamp suggests the problem wasn’t recent and the log entry does not accurately describe your current situation, my first thought would be the log entry is old & unrelated.

At the very least, I would be curious whether additional such log entries continue to be generated!

As an aside… could you check the device’s current date & time? If the timedatectl terminal command works, great! Otherwise, the date command should work on just about any Linux box.

 

@schungx

Check for additional error messages before this one.

Have you done this?

[raghukamath (Raghavendra Kamath)]

Got hit by this error after update just now

[RGJ (Richard - Communiteq)]

@schungx

No, disk is not full. Only 65% usage max on all volumes

A disk full error can also occur when you run out of inodes. What does df -i say?

[JammyDodger]

Would /logs be more revealing in this case?

I locked myself out of my test site with a similar error recently and that pointed me in the direction of a particular plugin conflict.

[raghukamath (Raghavendra Kamath)]

We (krita-artists.org) referred to this thread and it helped us in getting the site back up -

Undefined column 'boomarks.post_id' after update

Faced the same issue while updating today. Adding back the post_id column helped fix it. Steps: enter the discourse container ./launcher enter app Enter psql sudo -u postgres psql discourse Add back the post_id column in the bookmarks table ALTER TABLE bookmarks ADD COLUMN post_id int;

[schungx (Stephen Chung)]

Just in reply to a few comments above:

• df -i not even close. <10%

• logs no other errors in syslog and syslog.1 etc.

[JammyDodger]

The issue in the topic quoted a post or two up was having an old plugin still in your app.yml. Could this be the same for you too?

Could you list your plugins here for us to see?

[schungx (Stephen Chung)]

Could be this thing!

I have this in my app.yml

git clone https://github.com/merefield/discourse-topic-previews.git -b "theme_sidecar"

This is obviously an old repo with a special branch!

Will try with the latest one:

git clone https://github.com/paviliondev/discourse-topic-previews-sidecar

EDIT: It works!!!

Thanks a great deal to @raghukamath