Add plugin for displaying sidekiq statistic in web ui


(Davydovanton) #1

Hello,
I created simple plugin for displaying sidekiq statistic in web UI. And I create PR in github repository with this changes.
Also I have some screenshots for index page with my plugin.



/cc @sam


(Sam Saffron) #2

This is super awesome, one thing I would love is a way to integrate the stats for our schedular as well somehow.

Perhaps some sort of interface we can call to inform it we are running tasks outside of sidekiq?


(Davydovanton) #3

“we are running tasks outside of sidekiq”

sorry, but could you explain this moment? You use resque or something else?


(Sam Saffron) #4

Yes we have this:

Which runs regular jobs for us in dedicated thread, it is initialized here:

https://github.com/discourse/discourse/blob/master/config/initializers/sidekiq.rb#L24-L42

I would like to have a way for schedular manager to inform statistics that these regular jobs ran…


(Davydovanton) #5

So, I use native redis hash and list for collect all statistic. I think in your scheduler you can use this for collecting information. What do you think?


(Sam Saffron) #6

Sounds good, can you expose it as an API ? that way it will be less fragile long term integration wise

Also is there some sort of storage limit here? Not sure if we need to keep all this information around beyond 1 month


(Davydovanton) #7

yep, I can add this information to README file :+1:

I collect information per day and I haven’t any data live limits now.


(Davydovanton) #8

So, I created section “how you can update statistic inside middleware”. You can find its here:


(Davydovanton) #9

Also I have interesting idea. What do you think if I can collect any data from redis hash?
i.e. I will create table columns dynamically on index page.
For example: you add yyyy-mm-dd:WorkerName:next-runtime key and on the index page there is a new column in the table.


(Sam Saffron) #10

Can you expand a bit, not following.

Looks like this is not working right on meta:


(Davydovanton) #11

Could you perform any job? It’s look like as empty statistic hash.

I mean that in theory I can display any information from redis hash. It’s mean that you can add any information (for example date of next run time) and you will see this in statistic table on index page.


(Sam Saffron) #12

Tons of jobs have been running in the last 15 minutes since we deployed it.


(Davydovanton) #13

and what about table? it’s displaying correct?


(Sam Saffron) #14

Table is blank … maybe somehow… it is not talking to the right redis?

Can you try on a docker install? I can provide you with an image to play with on digital ocean if you wish.


Yeah I can see it working on local … its got to be talking incorrectly to redis.

… aha … this has to be related to the forking …

It is not being initialized … having a look


(Davydovanton) #15

It’s really strange because I use only Sidekiq#redis for calling redis :hushed:


(Sam Saffron) #16

I see the bug, we fork out our Redis so we need to manually initialize on fork.


(Davydovanton) #17

how I can to help you?


(Sam Saffron) #18

It’s fine, in theory this should fix it :slight_smile:

Confirmed my patch fixed this.


(Kane York) #19

I think @sam wanted something more along the lines of:

begin
  job.execute(args)
  finish = Time.now
  Sidekiq::Statstic.report_success(job.class, finish - start)
rescue => e
  finish = Time.now
  Sidekiq::Statstic.report_failure(job.class, finish - start)
end

Would be a lot more future-proof than manually inserting values into Redis.


(Sam Saffron) #20

has been included in discourse for ages, no point keeping this open forever