500 error in AWS s3 code

I’ve just been setting up s3 backups for a discourse installation, and have run into problems. the site is now giving me 500 errors for all pages.

I’d entered at least most of the s3 setup, but the backups weren’t getting uploaded, and I was looking for why that was. I had entered bucket details for the backups, but not bucket details for general file uploads.

I’m a bit hazy on the sequence of events. I wound up doing a container restart and even a rebuild at one point, which hasn’t fixed things, but I’m fairly sure the error had started occurring before that.

In the rails/production.log file, I see:

Job exception: undefined method `every' for Jobs::CreateBackup:Class

Job exception: undefined method `every' for Jobs::CreateBackup:Class

Started GET "/" for 161.202.72.162 at 2015-09-16 02:35:11 +0000
Processing by ListController#latest as HTML
  Rendered list/list.erb within layouts/application (1.6ms)
  Rendered layouts/_head.html.erb (3.7ms)
  Rendered common/_special_font_face.html.erb (0.6ms)
  Rendered common/_discourse_stylesheet.html.erb (0.3ms)
  Rendered application/_header.html.erb (0.3ms)
  Rendered common/_discourse_javascript.html.erb (2.3ms)
Completed 500 Internal Server Error in 75ms (ActiveRecord: 22.1ms)
ActionView::Template::Error (s3_upload_bucket)
/var/www/discourse/lib/file_store/s3_store.rb:95:in `s3_bucket'

Job exception: undefined method `every' for Jobs::CreateBackup:Class

Job exception: undefined method `every' for Jobs::CreateBackup:Class

The Job exception lines repeat at around one such error per second. The bit in between is the output for a single web page request.

Any suggestions on how I should proceed? I’m guessing I might need to go in through the rails console to adjust settings. What are the names of the settings involved?

Any ideas here @zogstrip?

@mc0e Are you running the latest version of Discourse?

I find it weird that there’s an exception on “every” in the “Jobs::CreateBackup” class since we don’t use that…

‘git pull’ and ‘./launcher rebuild app’ doesn’t fix the problem, and it potentially could have been part of causing the problem. i.e. the problem is not simply that discourse is out of date, but it could be that something in an older database hasn’t been updated to be compatible with the modern code.

I’ve just become involved with this instance of discourse. I’m not entirely sure when it was originally built, but I have noticed that the containers/app.yml file had some differences from what is in https://github.com/discourse/discourse_docker/blob/master/samples/standalone.yml . e.g. /var/log didn’t have a volume. I presume this is a reflection of this discourse instance having been around a while. Maybe there’s some hang-over from an earlier time in either the config outside the container, or in the database.

I expect the bit about “every” in the “Jobs::CreateBackup” class is not the primary issue. It goes back before the current failure to render pages at all, and before there was any s3 configuration in place. I’m obviously more concerned with the complete unavailability of the site, and unless you think it’s important I’ll ignore the Jobs::CreateBackup issue until the other is resolved.

Looking at the error above, I see the reference to /var/www/discourse/lib/file_store/s3_store.rb:95, which is the 3rd line below:

def s3_bucket
  return @s3_bucket if @s3_bucket
  raise Discourse::SiteSettingMissing.new("s3_upload_bucket") if SiteSetting.s3_upload_bucket.blank?
  @s3_bucket = SiteSetting.s3_upload_bucket.downcase
end

It seems to me that:

  • it’s odd that the s3 code is being invoked for rendering pages, but perhaps this relates to some sort of poor man’s cron arrangement with job execution hanging off page rendering calls?
  • either s3 should only be invoked if the requisite settings are there, or the exception should be graciously handled and other parts of the request allowed to proceed.
  • fixing the s3_upload_bucket setting might help
  • fixing some other setting might cause this code not to be invoked in the first place, but I don’t yet know where to look for that.

I have values entered for the following:

  • SiteSetting.s3_access_key_id
  • SiteSetting.s3_backup_bucket
  • SiteSetting.s3_region
  • SiteSetting.s3_secret_access_key
  • SiteSetting.s3_use_iam_profile (= false)

EDIT: also had SiteSetting.enable_s3_uploads set

Not set:

  • SiteSetting.s3_cdn_url
  • SiteSetting.s3_upload_bucket

As I understand it, the values look reasonable.

Could it be that the problem arises because I have set SiteSetting.s3_backup_bucket, but not SiteSetting.s3_upload_bucket? That seems reasonable to do in that I’m curently concerned with off-site backups, not storing uploads off-site, but perhaps it doesn’t work to do this?

SiteSetting.s3_upload_bucket = SiteSetting.s3_backup_bucket does indeed bring the site back up. It’s not really what I want to do with uploaded files though.

Looks like I had SiteSetting.enable_s3_uploads set, which is quite understandably incompatible with not setting SiteSetting.s3_upload_bucket, but it should be handled more gracefully.

3 Likes