Goodbye Sidetiq

Feature
1

We have been plagued with problems related to our scheduler sidetiq.

In particular we have had a few major recurring issues related to high CPU

https://meta.discourse.org/t/performance-issue-on-digital-ocean-with-discourse-docker/11895

We have been working to address them upstream, however even once addressed there are some design issues that we need fixed as well.

Recently, I was debugging erratic perf issue with @supermathie here and found a second serious issue.

Sidetiq always schedules every job at exactly the same time, this leads to periods where the same job on every site we host runs at the same time leading to network saturation.

I have been building a light weight scheduler to address the perf and scheduling issues. I expect to have it done today.

So, this is just an upfront warning that the scheduler is changing, if you notice any new issues with scheduled jobs please let me know.

Commit that activates this is:

10 Likes
2

This topic is now pinned. It will appear at the top of its category until it is either unpinned by a moderator, or the Clear Pin button is pressed.

3

Note: the UI is much improved over what sidetiq was giving us:

Pasted image1231×774 69.6 KB

4 Likes
4

Wow, that looks really nice! Will you be adding a link to it from the admin panel?

5

Sure, open to that, where would you link this ?

6

This looks like something that’s a candidate for extracting into it’s own library. Thoughts?

1 Like
7

Will require some real world usage first :wink:

2 Likes
8

Nice, I’m glad to ditch a freedom patch too.

9

I am still holding back to see if tobias takes in all the features we need.

10

What’s the upgrade impact (if any) of this change and should the install/upgrade doc(s) be updated? https://github.com/discourse/discourse/blob/master/docs/INSTALL-ubuntu.md

11

We’re going to switch over to the Docker install instructions as the new standard next week. @sam I know that is on your list, but make sure.

The no-Docker, old style install will be available but not our primary supported or recommended install any more. So we’ll move it around a bit.

12

What’s the transition from an environment running no-Docker to Docker like?

(Sorry if I’m being annoying. Just wondering how much time I need to budget for this transition. :smile: (I have a Bitnami-created VM hosted on Azure))

13

Yes. We also want to have a “howto convert from regular to Docker install” posted soon. @sam has done a number of conversions from old-style Discourse install to new Docker installs now and has documented the process internally.

@zogstrip and @sam are working on necessary import/export improvements and once that is ready, it should go up.

3 Likes
14

We’re having issues.

The sidekiq/scheduler page shows that all recurring jobs are ‘Not scheduled yet’

sidekiq.png1179×421 27.4 KB

Hitting the ‘Trigger’ button on for instance ‘Version Checks’ results in a screen showing “Internal Server Error” and this in the logs:

    ArgumentError - comparison of Class with Class failed:
            /var/www/discourse/lib/scheduler/web.rb:10:in `sort'
            /var/www/discourse/lib/scheduler/web.rb:10:in `block (2 levels) in registered'
/var/www/discourse/vendor/gems/rails_multisite/lib/rails_multisite/connection_management.rb:45:in `with_connection'
            /var/www/discourse/lib/scheduler/web.rb:8:in `block in registered'

From that moment on, the scheduler screen is broken (keeps showing ‘Internal Server Error’ until Sidekiq is restarted.

2 Likes
15

What version of ruby are you on ?

1 Like
16

ruby 2.0.0p353 (2013-11-22 revision 43784) [x86_64-linux]

1 Like
18

is there anything custom in your initializer, you are going to need to trace through the sidekiq initializer to see why its not doing an initial enqueue of the jobs

19

(Your redis question apparently disappeared, but flushing redis does make it recover from the “Internal Server Error” on sidekiq/scheduler)

The initializer is unmodified, actually this is a clean v0.9.8.4 checkout.

20

Can you do a clean checkout of master?

21

Sidekiq does seem to work, I see a ‘stat’ job coming by, it’s only the scheduled jobs that are failing.

Hitting the “Trigger” button on “Version check” does seem to run the version check, the following keys appear in redis

1) "default:last_installed_version"
 4) "default:missing_versions_count"
 5) "default:critical_updates_available"
 8) "default:last_version_check_at"
 9) "default:discourse_latest_version"
10) "default:_scheduler_Jobs::VersionCheck"
14) "default:_scheduler_queue_"

Removing the key “default:_scheduler_Jobs::VersionCheck” fixes the Internal Server Error

127.0.0.1:6379> get "default:_scheduler_Jobs::VersionCheck"
"{\"next_run\":1391979510,\"prev_run\":1391893854,\"prev_duration\":327,\"prev_result\":\"OK\"}"
127.0.0.1:6379> del "default:_scheduler_Jobs::VersionCheck"
(integer) 1