One thing I noticed is that the sidekiq queue issue started right after I upgraded to 3.4.0.beta4-dev, the other instance I installed around the same time seems to have the issue too (confirming now)
also looked at the ids, it seems the system is trying to send chat summary to every users 
is it normal?
update:
Turning off chat prevents those jobs from being queued again, but my site kind of needs the chat function, and I am still looking into other solutions.
another update:
I turned off the chat function for 2 days, turned it on again, and changed the Chat channel retention days from 180 to 3 days. Now it seems to work fine, will keep an eye on it still.