Upon further investigation, it looks like Discourse is redirecting that link but it’s not loading the image. If I go to the URL directly it still doesn’t work.
However, if I upload an image directly to S3 myself it doesn’t indeed load from the url. The bucket policy seems to check out, as well as the permissions, not sure why it wouldn’t be loading from Discourse-uploaded images.