503 errors, can't post, edit, like, etc

I migrated my site to a new instance (Azure to Lightsail, 32gb ram) and am having a host of problems.

The site is up and running but not seemingly working.

It seems like I’m crashing my instance despite not having much traffic at the moment and having the max Lightsail instance.

I can’t seem to do anything on the site: post threads, like posts, etc.

I’m getting these errors:

Message

'hijack topics timings ' is still running after 90 seconds on db default, this process may need to be restarted!

Backtrace

/var/www/discourse/lib/scheduler/defer.rb:105:in `block (2 levels) in do_work'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/message_bus-4.3.8/lib/message_bus/timer_thread.rb:117:in `do_work'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/message_bus-4.3.8/lib/message_bus/timer_thread.rb:95:in `block (2 levels) in queue'```

Failed to load resource: the server responded with a status of 503 ()
/admin/reports/bulk?reports%5Bdau_by_mau%5D%5Bfacets%5D%5B%5D=prev_period&reports%5Bdau_by_mau%5D%5Bstart_date%5D=2023-11-28&reports%5Bdau_by_mau%5D%5Bend_date%5D=2023-12-28&reports%5Bdaily_engaged_users%5D%5Bfacets%5D%5B%5D=prev_period&reports%5Bdaily_engaged_users%5D%5Bstart_date%5D=2023-11-28&reports%5Bdaily_engaged_users%5D%5Bend_date%5D=2023-12-28&reports%5Bnew_contributors%5D%5Bfacets%5D%5B%5D=prev_period&reports%5Bnew_contributors%5D%5Bstart_date%5D=2023-11-28&reports%5Bnew_contributors%5D%5Bend_date%5D=2023-12-28&reports%5Bpage_view_total_reqs%5D%5Bfacets%5D%5B%5D=prev_period&reports%5Bpage_view_total_reqs%5D%5Bstart_date%5D=2023-11-28&reports%5Bpage_view_total_reqs%5D%5Bend_date%5D=2023-12-28:1 
        
        
       Failed to load resource: the server responded with a status of 503 ()
/admin/reports/bulk?reports%5Bvisits%5D%5Bfacets%5D%5B%5D=prev_period&reports%5Bvisits%5D%5Bstart_date%5D=2023-11-28&reports%5Bvisits%5D%5Bend_date%5D=2023-12-28&reports%5Btime_to_first_response%5D%5Bfacets%5D%5B%5D=prev_period&reports%5Btime_to_first_response%5D%5Bstart_date%5D=2023-11-28&reports%5Btime_to_first_response%5D%5Bend_date%5D=2023-12-28&reports%5Blikes%5D%5Bfacets%5D%5B%5D=prev_period&reports%5Blikes%5D%5Bstart_date%5D=2023-11-28&reports%5Blikes%5D%5Bend_date%5D=2023-12-28&reports%5Bflags%5D%5Bfacets%5D%5B%5D=prev_period&reports%5Bflags%5D%5Bstart_date%5D=2023-11-28&reports%5Bflags%5D%5Bend_date%5D=2023-12-28:1 
        
        
       Failed to load resource: the server responded with a status of 503 ()
/admin/reports/bulk?reports%5Buser_to_user_private_messages_with_replies%5D%5Bfacets%5D%5B%5D=prev_period&reports%5Buser_to_user_private_messages_with_replies%5D%5Bstart_date%5D=2023-11-28&reports%5Buser_to_user_private_messages_with_replies%5D%5Bend_date%5D=2023-12-28&reports%5Busers_by_type%5D%5Bfacets%5D%5B%5D=prev_period&reports%5Busers_by_type%5D%5Bstart_date%5D=2023-12-28&reports%5Busers_by_type%5D%5Bend_date%5D=2023-12-28&reports%5Busers_by_trust_level%5D%5Bfacets%5D%5B%5D=prev_period&reports%5Busers_by_trust_level%5D%5Bstart_date%5D=2023-12-28&reports%5Busers_by_trust_level%5D%5Bend_date%5D=2023-12-28&reports%5Bstorage_stats%5D%5Bfacets%5D%5B%5D=prev_period&reports%5Bstorage_stats%5D%5Bstart_date%5D=2023-12-28&reports%5Bstorage_stats%5D%5Bend_date%5D=2023-12-28:1 
        
        
       Failed to load resource: the server responded with a status of 503 ()
/admin/reports/bulk?reports%5Btop_referred_topics%5D%5Bfacets%5D%5B%5D=prev_period&reports%5Btop_referred_topics%5D%5Bstart_date%5D=2023-12-22&reports%5Btop_referred_topics%5D%5Bend_date%5D=2023-12-28&reports%5Btop_referred_topics%5D%5Blimit%5D=8&reports%5Btrending_search%5D%5Bfacets%5D%5B%5D=prev_period&reports%5Btrending_search%5D%5Bstart_date%5D=2023-11-28&reports%5Btrending_search%5D%5Bend_date%5D=2023-12-28&reports%5Btrending_search%5D%5Blimit%5D=8:1 
        
        
       Failed to load resource: the server responded with a status of 503 ()
9953_2.png:1 
        
        
       Failed to load resource: the server responded with a status of 503 ()
24.png:1 
        
        
       Failed to load resource: the server responded with a status of 503 ()
24.png:1 
        
        
       Failed to load resource: the server responded with a status of 503 ()
24.png:1 
        
        
       Failed to load resource: the server responded with a status of 503 ()
24.png:1 
        
        
       Failed to load resource: the server responded with a status of 503 ()
24.png:1 
        
        
       Failed to load resource: the server responded with a status of 503 ()
24.png:1 
        
        
       Failed to load resource: the server responded with a status of 503 ()
24.png:1 
        
        
       Failed to load resource: the server responded with a status of 503 ()
af975f5419770f3b2e0d38133d107299c2192045.png:1 
        
        
       Failed to load resource: the server responded with a status of 404 ()
posts:1 
        
        
       Failed to load resource: the server responded with a status of 502 ()
ajax-error.js:35 SyntaxError: Unexpected token '<', "<html>
<h"... is not valid JSON
    at Function.parse [as parseJSON] (<anonymous>)
    at l (ajax-error.js:31:20)
    at u (ajax-error.js:74:10)
    at e.createPost (composer.js:1165:13)
l @ ajax-error.js:35
u @ ajax-error.js:74
createPost @ composer.js:1165
presence.js:211 PresenceChannel '/discourse-presence/reply/14067' dropped message (received 28, expecting 29), resyncing...
jquery.js:9940 
        
        
       POST https://forum.umhoops.com/topics/timings 503 (Service Unavailable)
send @ jquery.js:9940
ajax @ jquery.js:9521
f @ ajax.js:173
(anonymous) @ rsvp.js:459
R @ rsvp.js:915
k @ ajax.js:192
sendNextConsolidatedTiming @ screen-track.js:188
flush @ screen-track.js:301
tick @ screen-track.js:365
(anonymous) @ screen-track.js:60
$._run @ index.ts:665
$.run @ index.ts:339
d @ index.js:108
(anonymous) @ screen-track.js:60
screen-track.js:231 Failed to update topic times for topic 14067 due to 503 error
(anonymous) @ screen-track.js:231
_ @ rsvp.js:434
y @ rsvp.js:420
m @ rsvp.js:369
p.invoke @ queue.ts:203
p.flush @ queue.ts:98
h.flush @ deferred-action-queues.ts:75
$._end @ index.ts:616
$.end @ index.ts:298
$._run @ index.ts:667
$.run @ index.ts:339
d @ index.js:108
t.error @ ajax.js:148
u @ jquery.js:3223
fireWith @ jquery.js:3353
x @ jquery.js:9629
(anonymous) @ jquery.js:9888
load (async)
send @ jquery.js:9907
ajax @ jquery.js:9521
f @ ajax.js:173
(anonymous) @ rsvp.js:459
R @ rsvp.js:915
k @ ajax.js:192
sendNextConsolidatedTiming @ screen-track.js:188
flush @ screen-track.js:301
tick @ screen-track.js:365
(anonymous) @ screen-track.js:60
$._run @ index.ts:665
$.run @ index.ts:339
d @ index.js:108
(anonymous) @ screen-track.js:60
jquery.js:9940 
        
        
       POST https://forum.umhoops.com/topics/timings 503 (Service Unavailable)
send @ jquery.js:9940
ajax @ jquery.js:9521
f @ ajax.js:173
(anonymous) @ rsvp.js:459
R @ rsvp.js:915
k @ ajax.js:192
sendNextConsolidatedTiming @ screen-track.js:188
tick @ screen-track.js:370
(anonymous) @ screen-track.js:60
$._run @ index.ts:665
$.run @ index.ts:339
d @ index.js:108
(anonymous) @ screen-track.js:60
screen-track.js:231 Failed to update topic times for topic 14067 due to 503 error
(anonymous) @ screen-track.js:231
_ @ rsvp.js:434
y @ rsvp.js:420
m @ rsvp.js:369
p.invoke @ queue.ts:203
p.flush @ queue.ts:98
h.flush @ deferred-action-queues.ts:75
$._end @ index.ts:616
$.end @ index.ts:298
$._run @ index.ts:667
$.run @ index.ts:339
d @ index.js:108
t.error @ ajax.js:148
u @ jquery.js:3223
fireWith @ jquery.js:3353
x @ jquery.js:9629
(anonymous) @ jquery.js:9888
load (async)
send @ jquery.js:9907
ajax @ jquery.js:9521
f @ ajax.js:173
(anonymous) @ rsvp.js:459
R @ rsvp.js:915
k @ ajax.js:192
sendNextConsolidatedTiming @ screen-track.js:188
tick @ screen-track.js:370
(anonymous) @ screen-track.js:60
$._run @ index.ts:665
$.run @ index.ts:339
d @ index.js:108
(anonymous) @ screen-track.js:60
jquery.js:9940 
        
        
       POST https://forum.umhoops.com/topics/timings 503 (Service Unavailable)
send @ jquery.js:9940
ajax @ jquery.js:9521
f @ ajax.js:173
(anonymous) @ rsvp.js:459
R @ rsvp.js:915
k @ ajax.js:192
sendNextConsolidatedTiming @ screen-track.js:188
tick @ screen-track.js:370
(anonymous) @ screen-track.js:60
$._run @ index.ts:665
$.run @ index.ts:339
d @ index.js:108
(anonymous) @ screen-track.js:60
screen-track.js:231 Failed to update topic times for topic 14067 due to 503 error
(anonymous) @ screen-track.js:231
_ @ rsvp.js:434
y @ rsvp.js:420
m @ rsvp.js:369
p.invoke @ queue.ts:203
p.flush @ queue.ts:98
h.flush @ deferred-action-queues.ts:75
$._end @ index.ts:616
$.end @ index.ts:298
$._run @ index.ts:667
$.run @ index.ts:339
d @ index.js:108
t.error @ ajax.js:148
u @ jquery.js:3223
fireWith @ jquery.js:3353
x @ jquery.js:9629
(anonymous) @ jquery.js:9888
load (async)
send @ jquery.js:9907
ajax @ jquery.js:9521
f @ ajax.js:173
(anonymous) @ rsvp.js:459
R @ rsvp.js:915
k @ ajax.js:192
sendNextConsolidatedTiming @ screen-track.js:188
tick @ screen-track.js:370
(anonymous) @ screen-track.js:60
$._run @ index.ts:665
$.run @ index.ts:339
d @ index.js:108
(anonymous) @ screen-track.js:60
jquery.js:9940 
        
        
       POST https://forum.umhoops.com/topics/timings 503 (Service Unavailable)
send @ jquery.js:9940
ajax @ jquery.js:9521
f @ ajax.js:173
(anonymous) @ rsvp.js:459
R @ rsvp.js:915
k @ ajax.js:192
sendNextConsolidatedTiming @ screen-track.js:188
tick @ screen-track.js:370
(anonymous) @ screen-track.js:60
$._run @ index.ts:665
$.run @ index.ts:339
d @ index.js:108
(anonymous) @ screen-track.js:60
screen-track.js:231 Failed to update topic times for topic 14067 due to 503 error
(anonymous) @ screen-track.js:231
_ @ rsvp.js:434
y @ rsvp.js:420
m @ rsvp.js:369
p.invoke @ queue.ts:203
p.flush @ queue.ts:98
h.flush @ deferred-action-queues.ts:75
$._end @ index.ts:616
$.end @ index.ts:298
$._run @ index.ts:667
$.run @ index.ts:339
d @ index.js:108
t.error @ ajax.js:148
u @ jquery.js:3223
fireWith @ jquery.js:3353
x @ jquery.js:9629
(anonymous) @ jquery.js:9888
load (async)
send @ jquery.js:9907
ajax @ jquery.js:9521
f @ ajax.js:173
(anonymous) @ rsvp.js:459
R @ rsvp.js:915
k @ ajax.js:192
sendNextConsolidatedTiming @ screen-track.js:188
tick @ screen-track.js:370
(anonymous) @ screen-track.js:60
$._run @ index.ts:665
$.run @ index.ts:339
d @ index.js:108
(anonymous) @ screen-track.js:60
jquery.js:9940 
        
        
       POST https://forum.umhoops.com/topics/timings 503 (Service Unavailable)
send @ jquery.js:9940
ajax @ jquery.js:9521
f @ ajax.js:173
(anonymous) @ rsvp.js:459
R @ rsvp.js:915
k @ ajax.js:192
sendNextConsolidatedTiming @ screen-track.js:188
tick @ screen-track.js:370
(anonymous) @ screen-track.js:60
$._run @ index.ts:665
$.run @ index.ts:339
d @ index.js:108
(anonymous) @ screen-track.js:60
screen-track.js:231 Failed to update topic times for topic 14067 due to 503 error
(anonymous) @ screen-track.js:231
_ @ rsvp.js:434
y @ rsvp.js:420
m @ rsvp.js:369
p.invoke @ queue.ts:203
p.flush @ queue.ts:98
h.flush @ deferred-action-queues.ts:75
$._end @ index.ts:616
$.end @ index.ts:298
$._run @ index.ts:667
$.run @ index.ts:339
d @ index.js:108
t.error @ ajax.js:148
u @ jquery.js:3223
fireWith @ jquery.js:3353
x @ jquery.js:9629
(anonymous) @ jquery.js:9888
load (async)
send @ jquery.js:9907
ajax @ jquery.js:9521
f @ ajax.js:173
(anonymous) @ rsvp.js:459
R @ rsvp.js:915
k @ ajax.js:192
sendNextConsolidatedTiming @ screen-track.js:188
tick @ screen-track.js:370
(anonymous) @ screen-track.js:60
$._run @ index.ts:665
$.run @ index.ts:339
d @ index.js:108
(anonymous) @ screen-track.js:60

It was working at one point, then a few posts seem to have crashed the whole thing. Is it possible there is some sort of set of jobs running or something causing the issues?

Sounds like a overwhelmed database.

Did you tweak the app.yml to use that RAM?

Did you run a database vacuum after the move?

1 Like

If that’s this setting, then yes. I doubled it to this.

Is there a best practice of how to do this? I have not.

Ran VACUUM CLEAN in postgres.

Still getting these errors in the log:
'hijack topics timings ’ is still running after 90 seconds on db default, this process may need to be restarted!
‘Track Visit’ is still running after 90 seconds on db default, this process may need to be restarted!