Sorry for the late update.
I just got around testing with 1024MB and 2048MB of memory allocated but the result is still the same. Any more suggestions would be highly appreciated!
Edit: So after going through the logs again and again, I checked the code for the job and saw that this is not intented to be retried. Naturally, this is exactly what I did. I edited the file and found in Sidekiq that my issue is actually related to network related things.
The retry job gives me
Jobs::HandledExceptionWrapper: Wrapped Net::OpenTimeout: execution expired
as an error I can actually work with.
The takeaway for anyone in the future is: If URI.open(your url).read in the rails console does not work, it’s likely a ruby-network issue instead of an Discourse issue. I had the idea from stackoverflow