Can't complete build on Lightsail

Hi, it seems like I’m having the same problem as Rebuild app has stuck and Background saving terminated with success, except that I retried the build process many times, with and without rebooting.

I have a Lightsail instance with 2vCPU, 1GB RAM running Ubuntu 20.04. Manually edited db_shared_memory to 256MB and Unicorns to 4 based on other discussions here. I tried discourse-setup, discourse-doctor and launcher rebuild app, all of them seem to have the same behaviour.

I see
WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect. in my log but trying the configs in MKJ's Opinionated Discourse Deployment Configuration didn’t seem to help.

My build process hangs at

...[Babel: select-kit > applyPatches]
...[@embroider/compat/app]
...[@embroider/webpack]
...[@embroider/webpack]
107:M 22 Jan 2025 14:37:30.565 * 1 changes in 3600 seconds. Saving...
107:M 22 Jan 2025 14:37:31.648 * Background saving started by pid 854
854:C 22 Jan 2025 14:37:34.940 * DB saved on disk
854:C 22 Jan 2025 14:37:35.092 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
107:M 22 Jan 2025 14:37:35.341 * Background saving terminated with success

Any suggestions of what I can do to solve this? Thanks

How much swap do you have? You might increase it to 3 or 4GB.

How long have you left it hanging at that prompt?

1GB isn’t much; is anything else running on that VM?

It’s not the problem, but why 20.04? Its EOL is in 4 months.

4 Likes

Sorry, 22.04. Nothing else running on it. I left it hanging for more than an hour for sure. Am trying to increase the swap now, thanks for the suggestion.

On a side note, I’m in that scenario where my VM has marginally under 1GB RAM and I had to comment out the memory check. Is this why discourse-setup outputs this?

Found 0GB of memory and 4 physical CPU cores
setting db_shared_buffers = 0MB
setting UNICORN_WORKERS = 0
containers/app.yml memory parameters updated.

It could be that the converting of the memory to GB and the way that it counts CPU cores is wrong for some reason.

1GB is really tight these days. I’d really recommend 2GB.

2 Likes

It is a very small community for the time being and funding comes from academia, so I’m trying my best to keep it under $10/month. If it proves impossible I’ll scale it up, thanks!

2 Likes

There are plenty under 10 USD

4 Likes

I’m not familiar with Lightsail, but with Hetzner you have 2 vCPU, 4 GB RAM and 40 GB NVMe for 5€/month. They have servers in US too.

3 Likes

Thanks for the suggestions. I’m somewhat tied to AWS since everything else for this project is hosted/managed there, but will definitely consider moving somewhere else should it come to it.

3 Likes

So increasing the swap got me farther than ever, but my ssh connection timed out before the build finished (after a couple hours running).

The last identifiable output between a thousand Babel: discourse-plugins > applyPatches was [WARN] (broccoli-terser-sourcemap) Minifying "assets/browser-detect.js" took: 43325ms (more than 20,000ms). Giving it more swap would help here or should I just look into more RAM?

You can try using tmux or screen so that you can re-connect to the session.

My guess is that you ran out of RAM/memory and adding more swap might get you farther, but I’d recommend again that you just get more RAM. It might be possible to resize it, leave the disk size the same, rebuild with the more ram, and then resize the VM to the smaller one once it’s running.

I don’t know if that’s possible or easy with Lightsail

How much swap do you have?

1 Like

Hi, I had 2GB swap. I ended up getting a chunkier VM in Lightsail and it built without a problem. I don’t think resizing up/down is possible with this service. Thanks again for the help!

P.S: I still got the memory overcommit warning but did nothing about it. Should I enable it nevertheless?

1 Like

If you’d gone to 4GB it might have worked.

I haven’t paid attention to that lately. I’m pretty sure that my tooling enables it and the error message doesn’t go away. It’s a very good bet that most people don’t see or ignore that message completely. Your mileage may vary.

1 Like

I see it regularly, but since my installs work, I just ignore it :smile:

I suppose so, but I saw

and figured that even if it worked it would still mean crazy long build times and $5 was worth the ease of mind. Hopefully our community grows to justify the leap too :slight_smile:

1 Like

2GB is the party line, but builds are taking increasing amounts of ram, so with only 1GB I’ll often create bigger swap.

That’s what I’ve been trying to tell you, but in some parts of the world $5 is lots more than it is in other parts! :wink:

1 Like

I took the hints!

1 Like

I still can’t wrap my head around the fact that I’ll look for every cent I can save when renting a server, whereas I’ll gladly pay 20$ for a game I won’t play on Steam, or buy a 15$ pizza knowing it won’t even be that good. :lolsob:

2 Likes