The discussion revolves around a bug where oneboxed image links are not updating to locally downloaded images after being downloaded. Oni reports that remote images are no longer being downloaded, despite the disk not being full and Sidekiq running. After a rebuild, errors appear in the logs related to duplicate key violations, but the issue persists. David investigates and identifies that thumbnails are being generated, indicating images were downloaded, but the onebox links still point to the original source. David confirms the issue and provides a fix, which is further refined after Oni reports it’s still not working. David releases another update, and Oni confirms the issue is resolved.
Hi,
Suddenly remote images aren’t downloaded on my server anymore even though I didn’t change anything on my config and disk space is also good.
I tried to look in the logs but I don’t see anything suspicious.
The only thing is that I don’t see sidekiq process for the image download in the sideqik queue job.
Am I missing something here ?
Best regards
Is your disk full (you should get a notification on your admin dash if that’s why it’s stopped.
You could reboot and/or rebuild to see that sidekiq is running properly. See also /sidekiq
EDIT:
Oops. Sorry about that.
As I said, disk is not full (30% of use), Sidekiq is running.
Correction: Jobs::UpdateHotlinkedRaw is running but I can’t see any error in the logs
I’ll try a rebuild to see if there’s something new
No problem 
I tried a rebuild but there’s nothing new, remote images are still not downloaded.
Hmm. Well. And no errors in /logs?
Seems like the rebuild made logs error work again because now I finally see errors relative to images.
default: Failed to pull hotlinked image (https://www.2dgalleries.com/planches/800W/2017/139/marvano-les-sept-nains-3140.jpg) post: 945243
PG::UniqueViolation: ERROR: duplicate key value violates unique constraint "index_post_hotlinked_media_on_post_id_and_url_md5"
DETAIL: Key (post_id, md5(url::text))=(945243, af709cfc810efb23bda285d236a393a4) already exists.
But I don’t think that’s the problem here because many others images aren’t downloaded and I don’t see the errors concerning these.
EDIT: Ok so there’s something odd but might be helpful. The thumbnails are generated for the images. Has there been a change in the code recently where pasted image url just generate a thumbnail with a link to the original source ?
We did recently refactor this subsystem. I pinged the right engineer, hold tight.
Hi @Oni - please can you share an example post which isn’t working? (via PM is fine, if you prefer not to share publicly)
^^ this suggests that the images were downloaded - we can’t generate thumbnails from remote images. So it’s strange that the images are still appearing hotlinked in the post 
Here’s a link
The remote image in this post should have been downloaded. It’s been 10-15 days that this problem appeared.
You can see that this is the thumbnail that is displayed but the original source is just linked. It should be the original downloaded image.
Ah I see - looks like the image itself has been pointed to the locally-downloaded version, but the <a href link has not
Let’s see if the same thing happens with this hotlinked image:
Edit: yup, same thing
Thanks for reporting @Oni - I’ll get this fixed up.
I saw that there’s the new media blocking parameter in the last build but the onebox link bug still remains.
Yep, this bug is still on my todo list 
I don’t want to be the complaining guy here but the bug is still here a month later when all there is to do is change a var name…
2 posts were split to a new topic: Fetching images from newspaper websites is failing
This should take care of it:
https://github.com/discourse/discourse/commit/78e03649abbba3961aa9848d2a10022958943cf3
Directly oneboxed images will now be rewritten as markdown images. This was the old behaviour, before my recent refactoring. As before, there is a ~10min delay before the rewrite happens.
It doesn’t seem to work for me.
I made a rebuild.
Ah yes, the new logic wasn’t working when the onebox was the only hotlinked media in a post. This should take care of it: FIX: Ensure pull-hotlinked can rewrite lone oneboxes (#17354) · discourse/discourse@2d5d15b · GitHub
Here’s a hotlinked image onebox - it should be re-written as an image in the next 10 mins or so 
(https://pbs.twimg.com/media/FRdPEnnX0AIy0V8?format=jpg&name=large)
Edit: yup, that seemed to work. Please can you try another update @oni, and let us know how you get on?
I can confirm that it is working like a charm
