Reschedule sidekiq scheduled job

Support
1

I’m trying to move a Sidekiq job to 3 AM since it is really stressing out the database.
However, whatever I try, Sidekiq does not reload the job info and it executes it at the original time.

It shows up at 3 AM in /sidekiq/scheduler but the job runner is just ignoring that (or: doesn’t know it changed)

What should I do (or what am I doing wrong) ?

name = 'Jobs::CalculateAvgTime'
klass = name.constantize
info = klass.schedule_info
t = Time.now.utc
info.next_run = Time.utc(t.year, t.month, t.day).to_i + 27*3600
info.write!
2

Do you have a lot of “megatopics” (20k+ replies) in this database? Perhaps from migrations?

1 Like
3

EDIT some large topics, it is indeed a migrated forum.

db=# select posts_count from topics order by posts_count desc limit 20;
 posts_count 
-------------
       24338
       18737
       17171
       15380
       11847
       11609
       10183
       10001
       10001
       10001
       10001
       10001
       10001
       10000
        9991
        9988
        9860
        9337
        9295
        8595

post_timings has over 200M records and the disk isn’t the fastest.

I was going to spend some time optimizing Post.calculate_avg_time but I thought moving the job to deep in the night would be a quick relief.

4

That’s not too crazy – we have some migrations with 60k - 100k replies in topics and those REALLY hurt. A lot. Like… a lot a lot.

You might consider splitting the ones that are 15k and over, and closing those topics too.

With a slow disk you might have a tough job in front of you, IMO it is always easier to throw a bit of money at the problem and get faster disks involved. I don’t think this one is easy to optimize cc @sam.

1 Like
5

There was an earlier shot at this, I think it was never actually changed in the code though: Post.calculate_avg_time() taking up a long time - #15 by Falco

I completely agree with you but since the customer has ran out of budget now we’re stuck at the Sidekiq trick - which should have been simple :wink:

2 Likes
6

You still have “forcefully split topics” as a tool in your toolkit. Only worth it for 15k or higher though.

2 Likes
7

Honestly, I don’t see much value in having this data aggregated

I think number of people who read the post is probably good enough, I also am not sure what real help it provides in best of mode

Moving schedules around is not particularly easy, would involve patching

2 Likes
8

That’s fine, but what does it give @RGJ to do?

1 Like
9

If we give up the requirement for a gemetric mean of read time on every post on large topics, we can make the job much faster and then its a lot less likely to kill his system at 3AM.

In theory that code should work, my guess is that it is writing it to the wrong part of redis, you want to be at the default site of the multisite.

2 Likes
10

Yeah, already found that out… (the hard way) but this code is writing it correctly.
get "default:_scheduler_Jobs::CalculateAvgTime" returns the correct values. Seems like Sidekiq is keeping stuff in memory or something…

11

Still, it’s not killing our systems at 3AM nor do we have a slew of meta reports about this. I am not against further optimizing this, but it sounds most of all like a client with a lot of data who wants to cut corners on price …

12

I do not think there is anything 3AM about this job, it fires daily and has some natural drift built in. Yeah I am not rushed at all to work on this cause it is not a killer at all in our infrastructure on any of our sites.

13

Lesson: don’t be that client who cuts corners on price.

There are further lessons here, but I’ll leave them for the reader to suss out.

2 Likes
14

Yeah you’re completely right - but on the other hand: the forum is running pretty fine on that hardware, except for this one job causing havoc 0.5% of the time. So if we could minimize its impact then some people will be happier :slight_smile:

15

It is possible, however at the moment your incentives and our incentives are not aligned on this one. (Usually they are, though.)

3 Likes
16

Indeed - that’s why my initial question was limited to ‘how do I move this Sidekiq job to the darkest hours’ :slight_smile:

Which, by the way, appears to work now :thinking:

2 Likes
17

So it was the multisite issue described here?

18

Yes and no, that was a former issue but I already found out that one before creating this topic. (But I’ll mark it as a solution :slight_smile: )
I honestly have no idea why it didn’t work before and why it does work now.

1 Like
19

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.