How best to queue a long-running job

dev
1

So I’ve got a plugin that kicks off Ansible to do a Discourse installation (I really, really, want to stop using WordPress to take installation orders!).

Initially, I was calling fork to kick it off, which worked fine in testing, but I thought that I should do it a more “Discourse Way”.

So then made a once-off job to run the process, but it gets killed like this:

E, [2020-12-16T12:53:39.662683 #3158277] ERROR -- : worker=0 PID:3158425 timeout (61s > 60s), killing

I tried adding sidekiq_options queue: 'low' to the class, and then to the function that runs the job.

Then I skipped having the job run the process, and just ran it directly in the model with

         Discourse::Utils.execute_command(*instructions)

It works just fine if I run it from rails c, but however I run it (this is a Ubuntu development environment, started with ./bin/unicorn, it gets killed.

1 Like
2

Once-off jobs are jobs that will only execute, well, once. Looks like what you need is a regular job that will be scheduled by a trigger, like a request hitting a controller.

Regular jobs can be longer than 60s, as some do in normal usage.

1 Like
3

Well, I moved it to a regular job and tried running with ./bin/rails s instead of ./bin/unicorn and still no joy.

This makes no sense because I know that lots of those jobs take more than a minute.

4

How exactly are you running the job? Can you paste the relevant part of the ruby code ?

5

You bet!

https://github.com/pfaffman/discourse-pfaffmanager/blob/master/app/jobs/regular/create_droplet.rb

https://github.com/pfaffman/discourse-pfaffmanager/blob/master/app/models/pfaffmanager/server.rb#L150-L172

6

Why are you using

Jobs::CreateDroplet.new.execute(server_id: id)

instead of

Jobs.enqueue(:create_droplet, server_id: id)

in https://github.com/pfaffman/discourse-pfaffmanager/blob/49f7369b1dc0b1c8f63065b14c92ac7ecc3ab2b3/app/models/pfaffmanager/server.rb#L146?

2 Likes
7

And that’s why I need to ask questions! It’s because I’m just a caveman. (No offense intended to anyone who lives in a cave.)

There is, of course, no reason. I just couldn’t figure out how I was supposed to call it!

Thanks so much!

image

… and it’s still running! …

image

Thanks so much. This was at least a couple hours…

Next step: To run it on an actual production server…

4 Likes