i’m working on a migration to discourse, and i know about SiteSetting.download_remote_images_to_local. it’s awesome. however, when i (slow) rebake to download them all, i’m noticing that Photobucket serves a watermarked image depending on your referer. i looked briefly, and it looks like downloading is handled in /lib/final_destination.rb. i could probably patch that temporarily just for the post-migration rebake, but that doesn’t handle the hypothetical situation of people hotlinking images from crappy image hosts going forward. i think any sane person knows better than to use Photobucket now, and idk of others that watermark, so maybe not a big deal. my question is this.. has anyone already solved this? and is this something a plugin could accomplish? i have not learned about how plugins work yet. alternatively, would it be a bad feature suggestion to just always set the referrer to scheme://domain/ when downloading a remote image? when would that ever be a bad thing to have discourse itself do it? if you want to see what i’m talking about: https://i1111.photobucket.com/albums/h475/scoobystuff/Stereo/S1080033.jpg # watermark: curl -LO \ 'https://i1111.photobucket.com/albums/h475/scoobystuff/Stereo/S1080033.jpg' # no watermark: curl -LO --referer 'https://i1111.photobucket.com/' \ 'https://i1111.photobucket.com/albums/h475/scoobystuff/Stereo/S1080033.jpg' [image]

I’m not a fan of the watermarking they do, but it would actually violate Photobucket’s terms of service to send a false referrer to hide the watermark as one of their requirements on free accounts is that the Watermark is included when using them for hosting. Paid accounts don’t get those watermarks added.

[image] Claudia: it would actually violate Photobucket’s terms of service to send a false referrer what? how do you reach that conclusion? no ToS have been agreed to by a party merely downloading an image without an account (or even authentication). or even presented to said party. the ToS applies to the uploader. specifically: THE FREE ACCOUNT DOES NOT ALLOW IMAGE HOSTING. TO THE EXTENT THAT IN OUR SOLE AND ABSOLUTE DISCRETION, WE ALLOW A FREE ACCOUNT TO HOST AN IMAGE, THE IMAGE WILL INCLUDE A PHOTOBUCKET WATERMARK THAT REFLECTS THAT THE IMAGE IS HOSTED BY US. IF WE PERMIT ANY FREE IMAGE HOSTING, WE RESERVE THE RIGHT TO BLOCK THE IMAGE OR BLUR AND WATRMARK THE IMAGE IN OUR SOLE AND ABSOLUTE DISCRETION. FREE ACCOUNT HOLDERS ARE STRONGLY ENCOURAGED TO UPGRADE TO A PAID ACCOUNT THAT PERMITS THIRD PARTY HOSTING. further, this is a statement merely advising account capability (“THE FREE ACCOUNT DOES NOT ALLOW IMAGE HOSTING…IF WE PERMIT ANY FREE IMAGE HOSTING…”), and by agreeing, the account holder has merely been made aware of it. if i had a free photobucket account and had uploaded that image, i would not even be in violation by hotlinking it in this topic as the language does not imply that i must agree not to take such an action. the purpose of this statement is so that a free account holder cannot, for example, sue Photobucket for denial of service. contrast that to the following statement that i made up and does not appear in their actual ToS: THE FREE ACCOUNT HOLDER AGREES TO NOT USE THE ACCOUNT FOR IMAGE HOSTING AND WILL NOT HOTLINK IMAGES ELSEWHERE.

[image] shyguy: how do you reach that conclusion? no ToS have been agreed to by a party merely downloading an image without an account (or even authentication). or even presented to said party. A friend did exactly what you’re suggesting on his instance and Photobucket blocked his server. Let me see if I can find my copy of the email they sent him.

not legally a ToS violation imo, but photobucket is of course free to deny service to whomever at their discretion (unless contractually obligated). would be interested in seeing the email. did photobucket get his email address from the contact section of his server or something? thanks a lot for the warning! when i pull all the photobucket images (on the final migration), i’ll be sure to do it through a proxy or something. not that it really matters though, lol. no one uses photobucket anymore. i also won’t leave the referer spoof in the production server, but hopefully that won’t be needed anyways for new stuff going forward.

Although it wouldn’t be a ToS violation, it could be a DMCA violation (or a violation another copyright law from another country) to circumvent a method that is used to protect the copyright of an image or control access to the original image. [image] shyguy: when would that ever be a bad thing to have discourse itself do it? I don’t think that referer header forgery counts as being “a good web citizen”, and if the DMCA indeed applies, it would even be illegal to distribute such software. Many import scripts have their own code for attachment downloads, have you considered creating a regexp and applying that logic to inline Photobucket images as well - including the referer?

i don’t think copyright violation or circumvention applies in this case because the ToS states that the uploader retains all rights (though perhaps some law is somehow triggered in some territory). however, point taken about respecting standard browsing behavior and being a “good web citizen” – when you put it that way, i think it makes no sense to ship referer spoofing as an official feature. [image] Richard - Communiteq: Many import scripts have their own code for attachment downloads, have you considered creating a regexp and applying that logic to inline Photobucket images as well - including the referer? yes, i should be able to manage to get the images pulled in (without watermark) for my migration. in my case with the phpbb importer, i think it’s easiest to just temporarily modify discourse itself to spoof referer when downloading remote images and do a rebake since the importer doesn’t handle saving remote images (aside from avatars). i was mostly just wanting to hear from people who have tackled this problem before. i was also curious if it was technically possible to have a plugin do it (still wondering) and if it would even make sense to have discourse officially ship such a feature (it doesn’t).

[image] shyguy: copyright violation or circumvention The fact that the uploader retains their rights - including the right to control how it is used and distributed - does not give others the right to modify or remove watermarks from their content without permission. DMCA also covers access control, which would apply here (i.e. they have the right to serve that image without watermark only if it is surrounded by their ads, and removing that mechanism is accessing the image in another way as they want you to do it)

[image] Richard - Communiteq: DMCA also covers access control, which would apply here haha, maybe. but consider that something like youtube-dl is still up on github. that makes referer spoofing look quite innocent in comparison, and believe me, RIAA did really try to get it taken down. it was in “the news” for a while if you follow that type of news.

And that’s only because legitimate use cases exist. That doesn’t mean that all uses are legitimate. You don’t own rights to the images, recognize that.

Download Remote Images with Referer (plugin possible?)

Development

RGJ (Richard - Communiteq) March 31, 2023, 7:58pm 10

Let me rephrase that: youtube-dl is not still up, it’s back up after it was taken down.

Topic		Replies	Views
Prevent downloading of posted images Product Support	28	3778	January 24, 2024
Try to force https instead of downloading remote images Feature	13	2966	July 9, 2015
Prevent inserting images from other websites (flickr, facebook...) Product Support	7	673	August 1, 2024
Broken images with 'embed truncate' disabled Product Support rss-polling	10	603	February 25, 2022
Hosted to self-hosted migration: past uploads still reference discourse infra Product Support	15	919	October 16, 2023

Download Remote Images with Referer (plugin possible?)

Related topics