Yes, in part.
Update was done on the morning of the 21st. Inbound network traffic seems to be normalizing now. Outbound still higher than usual, but I assume it’s while avatars are getting cached. The amount of 503 to user_avatars/show is much smaller now. I’m guessing these will slowly get sorted over time as more avatars are processed.
However, we are still seeing many 503 errors on the logs for mainly two other endpoints:
POST /topics/timings
Still many 503 errors to this endpoint and some users reporting visited topics are not being marked as read. I have not found any info about it as the request doesn’t seem to be logged on production.log at all. The /logs don’t show anything related.
Where would one go about debugging these 503? Is there some other logs I’m unaware of, or is there a way to make logs more verbose perhaps (on a production system)?
POST /uploads.json?client_id=....
All I find on production.log for these 503 errors are along these lines:
Extract from production.log
Started POST “/uploads.json?client_id=X” for x.x.x.x at 2023-10-24 10:24:55 +0000
Processing by UploadsController#create as JSON
Parameters: {“upload_type”=>“composer”, “relativePath”=>“null”, “name”=>“Screenshot 2023-10-24 at 11.22.32.png”, “type”=>“image/png”, “sha1_checksum”=>“d1f11731320437724003c3840c5dcc5f934ba25a”, “file”=>#<ActionDispatch::Http::UploadedFile:0x00007f3c5e3c9898 @tempfile=#Tempfile:/tmp/RackMultipart20231024-1991-b30vit.png, @content_type=“image/png”, @original_filename=“Screenshot 2023-10-24 at 11.22.32.png”, @headers=“Content-Disposition: form-data; name="file"; filename="Screenshot 2023-10-24 at 11.22.32.png"\r\nContent-Type: image/png\r\n”>, “client_id”=>“X”}
Rendered text template (Duration: 0.0ms | Allocations: 1)
Completed 503 Service Unavailable in 10ms (Views: 0.4ms | ActiveRecord: 0.0ms | Allocations: 5007)
Our users report that they retry a few times until it works… I’m able to reproduce the error more or less consistently if I try to upload (another) file if one is still uploading. Uploading them one by one seems less prone to it for some reason.
# free -h
total used free shared buff/cache available
Mem: 15Gi 3.8Gi 621Mi 1.1Gi 10Gi 10Gi
# lscpu --parse=core | egrep -v ^# | sort -u | wc -l
2
UNICORN_WORKERS: 4
db_shared_buffers: "1024MB"