Sending notifications is slow, throws exceptions

(Felix Freiberger) #1

In a recent multisite install, I’ve set up some users to watch a category by changing the default user preferences and following the instructions by @sam.

Now, a post has been created in this category. However, the corresponding PostAlert job seems to be running into an exception. Here are the symptoms:

  1. Notifications are sent very slowly, only one is sent whenever Jobs::PostAlert runs.

  2. In Sidekiq (in the main site), I see a scheduled retry:

  3. In the logs of the main site, the error is logged.

The log entry says Job exception: unexpected return, with the following backtrace:

/var/www/discourse/plugins/discourse-push-notifications/plugin.rb:81:in `block (2 levels) in activate!'
/var/www/discourse/lib/discourse_event.rb:12:in `block in trigger'
/usr/local/lib/ruby/2.3.0/set.rb:306:in `each_key'
/usr/local/lib/ruby/2.3.0/set.rb:306:in `each'
/var/www/discourse/lib/discourse_event.rb:11:in `trigger'
/var/www/discourse/app/services/post_alerter.rb:392:in `create_notification'
/var/www/discourse/app/services/post_alerter.rb:509:in `block in notify_post_users'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activerecord- `each'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activerecord- `each'
/var/www/discourse/app/services/post_alerter.rb:508:in `notify_post_users'
/var/www/discourse/app/services/post_alerter.rb:105:in `after_save_post'
/var/www/discourse/app/services/post_alerter.rb:5:in `post_created'
/var/www/discourse/app/jobs/regular/post_alert.rb:7:in `execute'
/var/www/discourse/app/jobs/base.rb:154:in `block (2 levels) in perform'

What’s going on here? Did I hit a #bug, or did I manage to damage the database?

I’m running version Discourse 1.8.0.beta10 - GitHub - discourse/discourse: A platform for community discussion. Free, open, simple. version 6bb2dd0584cede80c741a43f0628f1dea23ae7a1

Upgrading Discourse before upgrading plugins causes problems
(Sam Saffron) #2

First thing, I would try to remove the push notifications plugin for now and see what happens, you may have hit a bug with it cc @tgxworld

(Felix Freiberger) #3

I just tried a rebuild without removing the plugin, and the rebuild failedhere are the parts of the log that hadn’t already scrolled out of the buffer.

I’ll try a rebuild without the push plugin now…

(Felix Freiberger) #4

No dice – the rebuild failed even without the push plugin :sadpanda:
The log looks basically identical.

(Lucas Nicodemus) #5

Are you using any other plugins – specifically those that may have altered the database? Do you have recent backups of your sites?

Looks like database issues with user warnings not wanting to migrate from the logs.

(Felix Freiberger) #6

Here’s my list of plugins:

- git clone
- git clone
- git clone
- git clone
# - git clone
- git clone
- git clone

The last plugin is custom, but does not touch the database (it only defines a onebox).

I do have a recent backup of the multisite that was modified, but I’m not sure how I’d go about restoring it.

(Rafael dos Santos Silva) #7

The logs said it’s staff notes…

(Lucas Nicodemus) #8

Noted: Don’t try to interpret logs on mobile. Doh.

This commit looks very related to the problem. To clarify, this is the problem with the rebuild, not push notifications.

Just in general, it’s really good practice to make sure you can always restore your backups prior to attempting upgrades of applications like this. A rebuild will trigger an effective upgrade, so be on the look out in the future (especially in the case of multisite installs).

(Felix Freiberger) #10

Without staff notes, the rebuild completes – CC @tgxworld.

The notifications issue returned after re-enabling the push notifications. Does anyone have an idea what’s broken here?

Given that Discourse is booting again, I once again know how I could restore (but I’d like to avoid that if possible).

(Alan Tan) #11

Ah @fefrei do you happen to deploy Discourse from the stable branch? I added support for Ruby 2.4 to Discourse but unfortunately have to rename the Warning class to UserWarning because Ruby 2.4 added a Warning module. While waiting for us to release a new version of stable which includes the table rename, you can pin the plugin to this commit.

I’ve have a look at the push-notification issue. That is another bug of its own.

(Felix Freiberger) #12

No, but from beta. Did the rename already trickle through to that?

Edit: Looks like the answer is no, here’s the commit:

(Felix Freiberger) #13

I’ve just updated to beta 11.

The good news: staff notes is working again :slight_smile:

The bad news: The push notifications issue is still there :frowning:
Shall I create a separate #bug topic for this issue, @tgxworld?

(Felix Freiberger) #14

Sorry to bump this, but I’m still unable to use push notifications because of this :sadpanda:
Can you offer any advice here, @tgxworld?