Bootstrap / rebuild without cloning everything over again?

Support
1

The Discourse instance is located behind the GFW, thus we use a SOCKS5 proxy for Git. We have a couple of plugins installed, so either rebuilding or bootstrapping the app clones all of these repos over and over again. Unfortunately the cloning results in timeout regularly, so the whole process starts from the very beginning, even though the most recent code base is already cloned. I have spend 40+ attempts and wasted about five hours. The last barrier is a yarn subprocess inside the container, which then usually times out, resulting in a failed upgrade.

Is there any way, how I could structure the app.yml, so that at least I do not invoke the whole plugin cloning process? Cloning into the docker-manager code and the discourse code base comes at a 50/50 chance, with the follow up cloning at about 1/3 success rate. I don’t know what causes the yarn subprocess to fail, yet at the moment it seems not to be possible to bring Discourse back to life with given methods.

Of course I was stupid enough to invoke launcher destroy app as I wanted to bootstrap it manually, so I can’t even do a launcher enter app for trying to run the yarn command manually. Anybody got any ideas? Thanks for your input.

Any chance to upgrade the postgres-client version?
2

I’m not expert in these mattters, but I think that the solution is a caching proxy for the stuff that you’re downloading.

There is a web.china.template.yml, are you using that?

3

Of course. I use the Chinese Ruby source in all Rails applications, glad to have that at least. What puzzles me: All the repos (including those of the plugins) are now already cloned, just this yarn subprocess invokes the following:

INFO -- : > cd /var/www/discourse && [ ! -d 'node_modules' ] || su discourse -c 'yarn install --production && yarn cache clean'

For whatever reason the GFW doesn’t like too many git clone processes after another, and at one point terminates the connection. If I could only run the launcher with a flag that says something like “OK dude, code is here, no need to clone… just bootstrap for now”, :grinning:

Edit: It’s unbelievable. Just as we type, my 78th attempt finally worked out after 11 hours. I resorted to sshuttle, which also took ~12 attempts but for whatever reason thh GFW had mercy with my poor soul.

4

That’s what your own caching proxy would do (or so I’d think). I’d check out squid. Then you’d see that launcher pulled from your mirror rather than the actual source.

launcher doesn’t have the code because it’s cloning it into a new container with none of the code, so that’s why it’s pulling down all of the code. Every. Single. Time.

5

Yes, we have some experience with Squid to some certain extent. This needs a general solution anyway, as I can’t spend hours with a manual Discourse upgrade every time. For a few months using a regular socks5 proxy had worked out, yet it’s a constant game of whack-a-mole with the GFW and accessing GitHub has become a drag since the beginning of October. No wonder there are tons of clones on this gitee.com site.

Thanks for the explanation about the launcher, I am rather stupid when it comes to Docker and had assumed it clones the git repos locally, and then writes them over to some container.

I will definitely look into the Squid options, as this might help with my second source of pain as well: fonts.googleapis.com

1 Like
6

That shouldn’t be git clones but install from the NPM package registry. For sure there may be an equivalent of bundle config mirror.https://rubygems.org https://gems.ruby-china.com/ for NPM/Yarn.

1 Like
7

Oh, directly after that followed some subprocess that cloned something and failed. Unfortunately I can’t reproduce the error log now. It had pulled something from GitHub definitely, as the error message was thge same TLS handshake error / timeout. It’s not important now though. Usually npm for whatever reason never has timeouts here. The GFW is long and full of mysteries!

2 Likes
8

Maybe it was

1 Like
9

You are the man! Exactly that broke my upgrade.

2 Likes
10

That is pretty darn stable, any chance we can push it to the NPM registry now @pmusaraj ?

3 Likes
11

Oh yes, definitely, let’s do that.

2 Likes
12

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.