Yes, but you over-rode it in your point #2 so by doing that you are back to putting files in S3.
2 Likes
The app.yml file settings are used only when rebuilding the container and are used to populate the config/discourse.conf in the container being built.
1 Like
Ahh! So if I have any s3_ vars in my conf/discourse.conf file, Discourse thinks that I have S3 uploads enabled? Ok… got it… I’ll remove that and give it another crack.
Probably there’s only one that matters… I’m not at my computer to check what all could cause that to trip, but if that check trips, I would expect your site to be uploading all new uploads to S3.
1 Like
It works, it works it works itworksitworks! Yay!!! Thank you so much for all your work and for patiently explaining what I needed to do. It’s great to get this underway.
So a few quick questions just so I can understand what’s going on better…
I’ve been running a batch of 100 at a time. The first few cycles, the tool was migrating posts (?) but now it’s just copying images. I have thousands and thousands of posts and each one usually only has one image attached… so I’m not sure why/how it “ran out” of posts to migrate? What does that mean, to migrate a post? So that’s question 1, what is happening exactly? 
Question 2 is more simple. Now when I run it just copies images, but I notice that 2X images are being copied into 3X locally:
Downloaded 27/100: //my-forum-storage.s3.dualstack.us-east-1.amazonaws.com/original/2X/1/14c56ef9f1dddb7b7a6f14e920234e0f714ea699.jpeg to /uploads/default/original/3X/1/4/14c56ef9f1dddb7b7a6f14e920234e0f714ea699.jpeg
Note the URL is changing from 2X/1/xyz.jpeg to 3X/1/4/xyz.jpeg (extra folder path in there)
Is that all ok?
Finally, I’ve been spot checking the images and they seem fine, but since I have no idea what post that image is associated with, I can’t really do a “live” check on the forum and be 100% sure that my users are seeing the right thing. How can I map the jpeg image filename to a forum post?
1 Like
Exactly what commands were you typing? You’ll need to expand the limit to cover all your posts.
So I ended up specifying a large limit once I was sure I was going on to the next phase, something like this:
bin/rake uploads:batch_migrate_from_s3[100,100000]
It migrates first uploads that it finds in posts; there isn’t referential integrity from post.raw references to an upload and the upload object in the database. It looks for posts with any references to plain URLs or upload:// pseudo-protocol URLs that represent remote content, where it needs to at least re-cook the post after migrating, if not save a modification as for base URLs directly pointing to S3. When it can’t find any content in a post that needs to be migrated, it goes on to the next thing.
The path changes aren’t something driven directly from the migration script, it’s just changes to where discourse puts the files. One more directory level helps with performance when you have a lot of files, and lots of other systems use two and occasionally three levels of directories to spread the files out.
You may run into the same problem I did where migrating the other uploads caused Discourse to lose track of users’ Profile Pictures. I didn’t notice that until it was way too late to restore from backup, so I just messaged affected users and apologized and asked them to fix their avatars. I still don’t know what weny wrong there.
Ah ok, cool. I see this in the progress output:
All Post uploads migrated. Migrating profile uploads...
All profiles uploads migrated. Migrating other non-post uploads...
What are “non-post uploads”? This is where all the work seems to be remaining-- migrating posts and profile uploads don’t do anything.
Thank you for the reassurance about paths for the raw uploads!
I’m not sure if that’s happening or not but it’s ok-- my site uses SSO so I pass the avatar URL every time the user logs in (or changes their picture on the main site).
Honestly, I don’t know what the rest of them were. I rather hoped that if referential integrity were a concern, there would be a foreign key constraint, and that otherwise it would just be looking up the upload and using wherever it was. I certainly did run into foreign key constraints as an indicator of needing to do something special for two uploads attached to user profiles.
1 Like
Hmmm it does seem to be working… though I ran into a serious problem this morning… suddenly my disk usage spiked 25% and filled the drive, crashing the forums completely.
Right now, when I’m doing the rake, it seems like it’s downloading/uploading the images right during the batch run. I enter the rake command, it processes 300 images exactly (I’m running them in batches of 300) and then it ends.
So the important question is: could just moving those download from S3 to the local disk be queued? Could that have built up any kind of batch that then happened at 5am and brought down my forums? 
I made it wait until the queue cleared before doing each new post while migrating posts, each new set of profile uploads, and then each other upload. It should be processing all your images right after they are moved. (The standard version just piles it up and clogs the queue until it is done; for me that would have been 10-12 days of not sending any forum notifications while it was reprocessing images.)
Nothing comes to my mind for what could have caused a spike like that, unless it ran into a bunch of huge uploads?
1 Like
Yeah, it’s a mystery to me. I mean, I’m not even running the batches that often-- I’m manually entering the command each time and watching it run. I don’t have any more posts or profiles to migrate; everything remaining is “other uploads.” When I run the batch, it seems like it’s downloading from S3 and re-uploading to Digital Ocean on the fly. (And I see nothing in sidekiq that implies anything is being queued.)
I’m trying to find any other logs that might indicate what happened at 5:35am.
Read whole of the topic.
But most of it looked confusing to me. Though I’m fully conversant with docker/container commands and basic rails commands.
I’m facing similar problems (many of my posts have missing images, or some don’t show images, till one clicks on them).
I was using 2 diff S3 buckets in 2 diff times of the year. I’ve around 1000 posts and approx so many images.
Now I’ve copied all the images from S3 buckets to my local server. Can anyone tell how can I remap the discourse urls so that those images are now connected correctly to the posts.