Image downloader should be more resilient to download failures


(Pedro Costa) #1

I have the same problem. The images contain in links are broken after 5 minutes and I received a system notification.

My discourse version is v2.1.0.beta3. In 1.9 version this problem didn’t occur.

40

@canapin Did you solve this problem?


Broken image in Discourse (but they work in the message preview)
(Coin-coin le Canapin) #2

No, I didn’t figure out how to fix that.


(Simon Cossar) #3

When you look at the topic after the images have broken, are you seeing the edit (pencil) icon? Is there a revision history similar to this, but with a broken image on the right?


(Pedro Costa) #4

Yes @Simon_Cossar, exactly the same screen.


(Simon Cossar) #5

It seems there is an error when remote images are downloaded to local. If you go to Admin/Logs/Error Logs on your forum, do you see any messages beginning with ‘Failed to pull hotlinked image for post’?

As a test, you could try disabling the ‘download remote images to local’ site setting and then publishing a new post that includes an image you haven’t used before.

Edit: I can reproduce this in my local dev environment when the ‘download remote images to local’ Site Setting is enabled. The image displays correctly in the preview window because in the preview window it’s using the image URL from the original post. In the cooked post it’s using the result of the PullHotlinkedImages job. I am assuming this is the issue on your site. We will need to figure out why downloading the remote image is failing.


(Pedro Costa) #6

I don’t see any log message like this. This issue starts after upgrade from 1.9 to 2.1.

Can you reproduce with my post source? Maybe the issue can be in website where I get the images.

Source: Poker Stars, $1 Buy-in (15/30 blinds, 7 ante) No Limit Hold'em Tournament, 5 Pla - Pastebin.com

Thanks @Simon_Cossar


(Michael Brown) #7

Well! What do you know! I reproduced on our sandbox:

This feels like the first image download works fine (note that the club is the first image in the post), then the rest are blocked (rate-limiting from the remote host) or fail in some other way.


(Michael Brown) #8

On a whim, I noticed that the images are all referenced over HTTP but redirect to HTTPS so I posted again with https links.

And now it’s fine!

Not sure if this is a logic error in our downloader or a specific behaviour in this site, but at least we now have an explanation and workaround.


(Jeff Atwood) #9

I’ve seen this before when the host is rate limiting image requests. It works when you trickle the images in manually, but if you attempt to download a bunch at once, you’ll get that error.


(Pedro Costa) #10

Thanks @supermathie and @Simon_Cossar. This works fine now. In my case was a problem with HTTPS redirection.


(Michael Brown) #11

Great! We’re still going to take a look at this case and see if we can make the downloader behave better, this should have worked for you.

EDIT: the original source:

[b]Preflop[/b]: Hero is BB with 6[img]http://pokertools.holdemmanager.com/images/emoticons/club4.gif[/img] 6[img]http://pokertools.holdemmanager.com/images/emoticons/spade4.gif[/img]
[color=grey]MP folds[/color], [color=red]CO raises to 60[/color], [color=grey]BTN folds[/color], SB calls 45, Hero calls 30
 
[b]Flop[/b]: (215) 7[img]http://pokertools.holdemmanager.com/images/emoticons/diamond4.gif[/img] 7[img]http://pokertools.holdemmanager.com/images/emoticons/club4.gif[/img] K[img]http://pokertools.holdemmanager.com/images/emoticons/spade4.gif[/img][color=blue] (3 players)[/color]
SB checks, Hero checks, [color=red]CO bets 65[/color], SB calls 65, [color=grey]Hero folds[/color]
 
[b]Turn[/b]: (345) 2[img]http://pokertools.holdemmanager.com/images/emoticons/heart4.gif[/img][color=blue] (2 players)[/color]
SB checks, [color=red]CO bets 345[/color], SB calls 345
 
[b]River[/b]: (1,035) 6[img]http://pokertools.holdemmanager.com/images/emoticons/heart4.gif[/img][color=blue] (2 players)[/color]
SB checks, [color=red]CO bets 1,035[/color], SB calls 1,035
 
[b]Results:[/b] 3,105 pot
Final Board: 7[img]http://pokertools.holdemmanager.com/images/emoticons/diamond4.gif[/img] 7[img]http://pokertools.holdemmanager.com/images/emoticons/club4.gif[/img] K[img]http://pokertools.holdemmanager.com/images/emoticons/spade4.gif[/img] 2[img]http://pokertools.holdemmanager.com/images/emoticons/heart4.gif[/img] 6[img]http://pokertools.holdemmanager.com/images/emoticons/heart4.gif[/img]
SB mucked K[img]http://pokertools.holdemmanager.com/images/emoticons/diamond4.gif[/img] T[img]http://pokertools.holdemmanager.com/images/emoticons/diamond4.gif[/img] and lost (-1,512 net)
CO showed K[img]http://pokertools.holdemmanager.com/images/emoticons/heart4.gif[/img] A[img]http://pokertools.holdemmanager.com/images/emoticons/spade4.gif[/img] and won 3,105 (1,593 net)

(Sam Saffron) #12

We should have at least some sort of retry logic here, only kill the image if it failed say 3 times in 24 hours, have a retry regular job. I think it is a bit too aggressive to kill the images after the first failure.

@vinothkannans something to add to your list, not high priority


Rebaking posts does not fix broken locally downloaded linked images