Throttle rebake to keep from running out of memory?

Support
1

I’m trying to rebake 45k posts because I’ve changed the S3 CDN URL and it keeps running out of memory and killing it.

I read this about setting up a swapfile…

The first two commands went fine…

  1. Create an empty swapfile

    install -o root -g root -m 0600 /dev/null /swapfile

  2. write out a 2GB file named ‘swapfile’

    dd if=/dev/zero of=/swapfile bs=1k count=2048k

But then on mkswap /swapfile it gave me an error…

mkswap error swapfile is mounted will not make swapspace

I tried this instead…

swapoff /swapfile && rm /swapfile

and got this error…

swapoff: Not superuser

I’m on a Digital Ocean Droplet, so I guess I don’t have the necessary permissions for that?

Hitting those blocks, I tried instead to just rebake uncooked posts running…

rake posts:rebake_uncooked_posts

It will run several hundred, maybe a few thousand if I’m lucky, then kills off again. After babysitting it since yesterday, I’m half baked. :wink:

Is there a way to run a rebake command that is throttled? I don’t care if it takes a while, overnight or whatever, I just want it to stop failing.

Any help would be much appreciated.

2

Oddly enough I let the uncooked run again while I was writing all of that up. Left for a long lunch with some friends and came back to it being 100% done. I’m not sure if the swapfile finally started working or if it was just a low use coincidence of timing, but…

image
image936×550 87.1 KB

done, finally!

Maybe if someone has an answer, it can help someone else or me on the next go round.

3

On ubuntu:

Sudo -i

On Debian

Su -

1 Like
4

Thanks. Not sure if I did it right or not.

image
image966×329 55.4 KB

5

That doesn’t look right. Which instructions are you following?

And why are you creating swap inside the container? Swap is meant to be created on the host not inside the container.

6

Because I only marginally know what I’m doing. :wink:

So that’s at the ubuntu: base level and not the ubuntu:/var/discourse or ubuntu-app:/var/www/discourselevels?

Any ideas on the rebake throttling?

1 Like
7

I have never seen rebake being throttled, it is asynchronous, it queues the jobs which are processed in the background.

1 Like
8

OK, thank you. Mine finished the rebake for now, so I’ll consider myself lucky and stop messing with it. :smiley:

9

Warning :warning: : don’t do things blindly on a production site, there is a strong chance of things going wrong without any real way to recover them.

no, it’s not the ubuntu: level.

Did you do a ./launcher enter app before trying to create swap? If yes, that what you shouldn’t have done.

When you first log on to your server is the host, when you enter app, you’re performing actions inside the container.

the swap needs to be created on the host, the application lives inside the container, this is what you will access to run the rebake command.

10

OK, thanks for the guidance.

Our forum is just a group of friends and has good backups, so if I roast it for a week or so, we’ll just have to suffer through GroupMe. :wink:

11

This means that you already have a swapfile, it’s called /swapfile, and it’s in use.

If you want to make it bigger, you can stop your container, then unmount it, and then use that dd command to make it bigger, and then the mkswap command.

If someone else reads this, the way that I do it is to grep swap discourse-setup and look at the commands there. It’s a slightly easier version of what to do.

3 Likes
12

I often serve as a cautionary tail.

Thanks for the addition.

1 Like
13

I don’t know about you, but I was repeatedly going out of memory during a long rebake process. Creating a 2 GB swap improved (more posts got rebaked), but still OOM after a while. Increased swap to 4 GB, and was able to rebake all my posts.

I didn’t know anything about swap files besides this guide Create a swapfile for your Linux server, but chatGPT was knowledgeable enough to help me increase my swap without issues.

3 Likes
14

One issue for me is that I don’t have a good grasp on what needs to be done in which area. Inside or outside the container or inside or outside of the app. I try to keep good notes and have mental shortcuts for the things that I’ll use again. But just when I start to understand it, I’m done with what I need to do and I won’t touch it again for another 6 to 12 to 18 months. Then I have to kind of relearn it.

15

sounds like every system admin’s whole life story :joy:

my solution is to take notes then create a little how to here on meta

I make it a wiki so others can edit it if they find a mistake or oversight and next time I need to do this same thing…

1 Like