The backup has many directory and subdirectory. I think you have to copy the files in directories, from both the buckets recursively to the corresponding location in the local server. For example, all the contents in the directory 1X and 2X of both the bucket has to be copied to 1X and 2X directory in the local server. I am not sure about it, but I think so.
Can you confirm that you have properly copied all the files?
Yes, I fully understand and have ensured that their relative paths remain exactly same.
E.g. if a file earlier existed at:
//bucket1/uploads/original/2x/f/filename.jpg
And the other
//bucket2/uploads/original/1x/a/filename.png
The two now also exist at:
/var/discourse/shared/web_only/uploads/default/original/2x/filename.jpg and …/original/1x/a/filename.png
Problems
Before copying the bucket contents to local server, I find that some of my images only show as icons, and full image can be viewed by website visitor by only clicking that icon.
And after I copied (not moved) all the bucket contents to my local server’s above said location, (with relative paths intact), and commanded
discourse remap:oldurl-or-path new-url-or-path,
Then nothing changed visibly on the site. But then I commanded
rake posts:rebake ,
Then even my image icons disappeared and no url/path showed when I hovered my mouse pointer over the image place holder’s empty place.
Hope I’ve given enough details.
In that case, I think the below will fix your problem.
./launcher enter app
discourse remap //bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/ /uploads/default/
discourse remap //bhdisco.s3.dualstack.ap-south-1.amazonaws.com/uploads/ /uploads/default/
rake posts:rebake
Thanks ji.
But as I already explained that after I do ‘remap’ and ‘rebake’ operation, even the icons of the images disappear (i.e. the situation becomes worse).
You have to give the correct path when remapping, else it may break. I have manually remapped one on the S3 link from the sample output you provided earlier and it works.
An image hosted in the bucket:
The same image hosted in your local server:
You may open the image in a new tab to see the links of both the above image. So I assume this will work.
./launcher enter app
discourse remap //bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/ /uploads/default/
discourse remap //bhdisco.s3.dualstack.ap-south-1.amazonaws.com/uploads/ /uploads/default/
rake posts:rebake
Thanks again.
I right clicked the (first) image and the path it showed in the url bar was not of my bucket:
Same went for 2nd image.
You can check 1 sample post of my website here, (the post is in Hindi though, but you may easily find the small icon of the image in between the post). The full image only shows if one clicks on the small icon.
But If I remap, and then rebake, then this icon too disappears completely. And there is no way left for the visitor to see the image.
It seems like you have previously used CloudFront for caching. Clear your browser cache and try again.
If you have made any CloudFront configuration, I think it better to undo all those configurations.
No cloudflare {edit: CloudFront} (or any other CDN) involved on my end, for my website, for the last around 2 years.
The images on meta ARE stored on cloudflare perhaps.
Not cloudflare. Its CloudFront. Have you cleared the browser cache?
Yes, cloudFront (sorry).
Yes I did. Moreover, I opened this topic in Firefox’s incognito mode.
Now, I am also getting the same CloudFront links. It seems like you have enabled AWS CloudFront for your site. Cloudfront is caching the images in your bucket. I think you have to remove the distribution from CloudFront. You have to login into your AWS account to see if CloudFront distribution is active?
Hey, We misunderstood CloudFront here. Actually, meta discourse is caching the image. That’s my bad.
I never used CloudFront (or any other CDN) in Aws or in front end. Though around 2 years back, I used CloudFlare for a month or so, but then removed that for good.
Nor did I chose cloudfront in Aws. So no question arises there also.
Moreover, on my PC, when I open the images, they show the aws bucket addresses, not any other CDN.
I misunderstood CloudFront here. Actually, meta discourse cached the image. I have edited my previous post. Take a look.
You’re only suggesting that I remap the image paths (from S3 bucket url to local server path). And then rebake.
But as I told above, I’ve already done that and have found that it compounds the problem. (moreover, after rabking, the ‘Restore’ also doesn’t work and its hard to revert).
Any other helpful idea(s) pls?
When you remapped, are you sure that the paths were same as below,
discourse remap //bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/ /uploads/default/
discourse remap //bhdisco.s3.dualstack.ap-south-1.amazonaws.com/uploads/ /uploads/default/
What path you have specified when remapped?
I have already given you an example in a previous post. It worked. I am unaware of any other method.
I’ll try again, with as much care as possible.
But I was searching for some method with which I could find what’s going behind the curtains. At least with 1 sample post, I wanted to see, which posts are having which bucket mapped and what mess I’m in.
Anyway, I’ll try again and let it know here. Thanks once again.
The path you have specified in your previous remap will be reflected in the output of the below command. Since both buckets are still there, I think your previous remap may be failed. Anyways, it’s up to you now. I hope you will get your solution soon. Best of luck.
./launcher enter app
rails c
Upload.all.sample(2000).pluck(:url)
Thank you
Not smarter than you but just went through it - and made it out with @Pravi’s help.
This is not super-easy - take great care with the text and links as it will be a right mess to tidy up if it goes wrong. But it is very doable.
First, install AWS CLI inside the app container
cd /var/discourse
./launcher enter app
sudo apt install awscli
Configure aws with your S3 ID and password (usually straightforward)
aws configure
Then use aws to copy all the content of the bucket into public/uploads/default/
aws s3 sync s3://my-bucket-name/ public/uploads/default/
This is easiest by finding an image on your forum and inspecting the URL. You want every bit until the actual filename (including the last /):
discourse remap //the-long-url-in-your-images-until-the-filename /uploads/default/
rake posts:rebake
exit
./launcher rebuild app
Whew! Done. So far I haven’t encountered any issues. Go and test it out!
Nice work! I believe those are the steps that eventually helped me to succeed, but I had done alot of different remaps that didn’t work as well! Great to have formalized instructions.