Rebuild taking ~3hours

I appreciate that, but rebuilding the app takes about 3 hours!
And this is on a massive VPS.
Wouldn’t it be a cool thing to be able to remove plugins just like we can add and remove themes?

Perhaps something to consider as feature in future?
Thanks!!

Sorry?! It should take ~20 minutes?!

1 Like

:person_shrugging: not here
Original install took 2 hours.

Now, the ./launcher rebuild app is gone as far:

Ensuring launcher is up to date
Fetching origin
Updating Launcher...
Updating eeefdde..30be122
Fast-forward
 image/base/install-imagemagick             | 5 ++++-
 launcher                                   | 2 +-
 templates/web.letsencrypt.ssl.template.yml | 2 +-
 templates/web.template.yml                 | 6 +++---
 4 files changed, 9 insertions(+), 6 deletions(-)
Launcher updated, restarting...
Ensuring launcher is up to date
Fetching origin
Launcher is up-to-date
Stopping old container
+ /usr/bin/docker stop -t 60 app
app
Unable to find image 'discourse/base:2.0.20220818-0047' locally
2.0.20220818-0047: Pulling from discourse/base
1efc276f4ff9: Pulling fs layer
1b880e64942b: Pulling fs layer
794f6dc9a2ff: Pulling fs layer
1efc276f4ff9: Verifying Checksum
1efc276f4ff9: Download complete
1efc276f4ff9: Pull complete
794f6dc9a2ff: Verifying Checksum
794f6dc9a2ff: Download complete
1b880e64942b: Verifying Checksum
1b880e64942b: Download complete
1b880e64942b: Pull complete
794f6dc9a2ff: Pull complete
Digest: sha256:7734701087766821ffb2ddcef423754798bd345c2ac0d550131c6e6905c268e8
Status: Downloaded newer image for discourse/base:2.0.20220818-0047

And after the last line, it just keeps blinking (process is ongoing)
That is about since 30 minutes now.
Last time I did that, it took full fledged 160 minutes.

Server Total Memory: 4.657GiB, Kernel Version: 5.15.0-43-generic, Operating System: Ubuntu 22.04 LTS, 50GB SSD drive, 2.5 Cores, 5 Threads Intel® Xeon® CPUs


Not sure what could be wrong with that, other than it takes very long :confused:

3 hours sounds very wrong indeed. You need to investigate this issue and resolve that as first priority.

There are ‘visual pauses’ in the build logging, this is normal, but that delay is not.

My hunch is somehow you are starting to use swap during build, but I’m no SA. Are you running anything else on that machine?

Downloads require unzipping, and that is computationally expensive, but its the same images for everyone.

For reference I just rebuilt a site just now and it took 13 minutes, 46 seconds on a 2GB 2vCPU machine on https://scaleway.com

4 Likes

Nope, this is a VPS that is solely for this discourse instance. And I run many other VPS with the same company, which all run super smooth (but they are not discourse instances, they are other apps such as custom PHP or WP/CP CMS instances etc).

I am not aware of any swap usage, and I cannot see any spikes in resource usages that would even justify
Now it is at building.... - it seems a bit faster than last time, yet, it is still clearly over 20 minutes

I will ask the server providers if they can spot anything unusual on their ends, however, I recall they personally told me that their own Discourse instance took about 2 hours as well to build. (PS These VPS are on Hetzner machines rented over a third party, I doubt I can get anything better.)

In any case, my suggestion stands that it would be amazing to be able to add, and remove plugins like we can themes. Perhaps it is something to consider?

I really don’t know anything, but AFAIK it is not possible, because Discourse is a lot of compiled javascripts. When there is need to add or remove something that changes how core is acting and working there must do rebuilding.

Very much same situation than with Varnish for example.

1 Like

Ok…

Btw I checked in with server folks and indeed we hit 100% memory

5gb ??? 5gb ram is a bit a lot to consume for anything really.
The discourse requirements say it need 1gb ram!

And it is now stuck on below:

104:M 04 Oct 2022 07:19:27.251 # Redis is now ready to exit, bye bye...
sha256:662695076506add39a630c2169b8b618f0121386951e93c6ffd2a6473107ae55
f4a95a1e69d5375e6ea30dfb22576209d249e5bc67b04a6fa09df289b1441888
Removing old container
+ /usr/bin/docker rm app
app

Thus I cannot even upgrade the server as the process would be interrupted.

really, I do not think this is a server issue at all. If we need 1GB, then 5GB should be just more than enough.
Something is wrong with this, massively wrong.
I understand others must have a better experience (looking at @merefield who said to upgrade on a 2GB…) yet it is not working for us as it should.

Anyway, it is off topic this thread I guess.

1 Like

I just rebuilt another site, 4GB, 3vCPU, again < 15 mins (ie the extra memory/CPU didn’t actually help that much in my case, but still far from hours!).

The only I just notice is that my VPS uses vfs instead of the suggested aufs or overlay.
Yet, according to this Can't run ./launcher rebuild app - Docker storage driver is unsupported - #45 by david, it shouldn’t be anything important, and thus we run the launcher with --skip-prereqs, because otherwise we cant even run Discourse.

I do wonder if there is more to that storage driver requirement

1 Like

That is a bit… over positive. For testing purposes it can be enough, and it can be a little bit tight even then. 2GB is much closer to reality.

4 GB is enough for bigger instances, though. But amount and power of cores plays important role too.

And still… if you have 5 GB and rebuilding takes that long time there must be something wrong.

I’m on DigitalOcean/4 GB/2 intel vcpu and rebuilding takes something around 25 minutes.

What is your system exacly, plus app.yml — is there something custom?

Hi, the only custom really is what I mentioned before (vfs instead of aufs/overaly)
The rest is vanilla.

Server:
Server Total Memory: 4.657GiB, Kernel Version: 5.15.0-43-generic, Operating System: Ubuntu 22.04 LTS, 50GB SSD drive, 2.5 Cores, 5 Threads Intel® Xeon® CPUs

yaml is original apart of these add-on plugins:

discourse-assign
discourse-chat
discourse-checklist
discourse-docs
discourse-reactions
discourse-solved
discourse-surveys
discourse-voting
docker_manager
styleguide

I don’t see from that data why rebuilding would take that long time.

I would spin on same sized ubuntu-VPS somewhere else. Just to find out if the issue comes from hosting company. It costs to you few bucks and hour or two job.

Lots of memory is needed to precompile the javascript. Try adding swap.

The overlay

I think that test may be there for a reason. This could be your issue.

1 Like

Why 5 GB RAM with vanilla Discourse would need more swap if smaller ones are totally happy what those automatically got?

Because it’s easier than doing what you really need to do.

Adding swap might help, but it’s not your biggest issue.

3 Likes

I assume I should have tried this before deploying the discourse container: Webdock : How to change the Docker storage driver

Apparently it is possible to change the storage driver - contrary to what the host instructed me back when we deployed the container (which was to either edit the loader file or ignore the check)

Duh, now it is too late I’d assume.

Thanks anyway, if this is the cause of the issues, I guess it is user-wrongdoing. Mea culpa :slight_smile:

I wouldn’t think so.

Also the two container setup has reduced down time since you csv rebuild the new web container while the old one continues to run.

1 Like

How large is your site? I would add swap regardless to address the memory issue.

A blank 1GB instance with 1GB swap performs fine for a small test community but as soon as it grows that won’t cut it. The swap absolutely makes a difference.

If rebuilding on a “massive” VPS, siting on a datacenter with a fat pipe to the internet takes 3 hours, and rebuilding my install over at https://discourse-on-a-pi.falco.dev that runs on a credit card sized board in my desk, connected to the internet via a standard home plan in a third world country takes only 14 minutes (+4 minutes when there is a new container image) there is something wrong in the product they are selling you…

15 Likes

How is the memory usage on the server? If you’re at or near capacity before you start rebuilding, you’ll have a lot of swapping and that will certainly make everything take excessively long.

If that is the case, I would suggest you temporarily disable anything else that might be running in the server if possible.

3 Likes