The Discourse instance is located behind the GFW, thus we use a SOCKS5 proxy for Git. We have a couple of plugins installed, so either rebuilding or bootstrapping the app clones all of these repos over and over again. Unfortunately the cloning results in timeout regularly, so the whole process starts from the very beginning, even though the most recent code base is already cloned. I have spend 40+ attempts and wasted about five hours. The last barrier is a yarn subprocess inside the container, which then usually times out, resulting in a failed upgrade. Is there any way, how I could structure the app.yml, so that at least I do not invoke the whole plugin cloning process? Cloning into the docker-manager code and the discourse code base comes at a 50/50 chance, with the follow up cloning at about 1/3 success rate. I don’t know what causes the yarn subprocess to fail, yet at the moment it seems not to be possible to bring Discourse back to life with given methods. Of course I was stupid enough to invoke launcher destroy app as I wanted to bootstrap it manually, so I can’t even do a launcher enter app for trying to run the yarn command manually. Anybody got any ideas? Thanks for your input.

[image] leoniscsem: If I could only run the launcher with a flag that says something like “ OK dude, code is here, no need to clone… just bootstrap for now ”, :grinning: That’s what your own caching proxy would do (or so I’d think). I’d check out squid. Then you’d see that launcher pulled from your mirror rather than the actual source. launcher doesn’t have the code because it’s cloning it into a new container with none of the code, so that’s why it’s pulling down all of the code. Every. Single. Time.

I’m not expert in these mattters, but I think that the solution is a caching proxy for the stuff that you’re downloading. There is a web.china.template.yml, are you using that?

[image] pfaffman: There is a web.china.template.yml , are you using that? Of course. I use the Chinese Ruby source in all Rails applications, glad to have that at least. What puzzles me: All the repos (including those of the plugins) are now already cloned, just this yarn subprocess invokes the following: INFO -- : > cd /var/www/discourse && [ ! -d 'node_modules' ] || su discourse -c 'yarn install --production && yarn cache clean' For whatever reason the GFW doesn’t like too many git clone processes after another, and at one point terminates the connection. If I could only run the launcher with a flag that says something like “OK dude, code is here, no need to clone… just bootstrap for now”, :grinning: Edit: It’s unbelievable. Just as we type, my 78th attempt finally worked out after 11 hours. I resorted to sshuttle, which also took ~12 attempts but for whatever reason thh GFW had mercy with my poor soul.

Yes, we have some experience with Squid to some certain extent. This needs a general solution anyway, as I can’t spend hours with a manual Discourse upgrade every time. For a few months using a regular socks5 proxy had worked out, yet it’s a constant game of whack-a-mole with the GFW and accessing GitHub has become a drag since the beginning of October. No wonder there are tons of clones on this gitee.com site. Thanks for the explanation about the launcher, I am rather stupid when it comes to Docker and had assumed it clones the git repos locally, and then writes them over to some container. I will definitely look into the Squid options, as this might help with my second source of pain as well: fonts.googleapis.com

[image] leoniscsem: just this yarn subprocess invokes the following: INFO -- : > cd /var/www/discourse && [ ! -d 'node_modules' ] || su discourse -c 'yarn install --production && yarn cache clean' For whatever reason the GFW doesn’t like too many git clone processes after another, That shouldn’t be git clones but install from the NPM package registry. For sure there may be an equivalent of bundle config mirror.https://rubygems.org https://gems.ruby-china.com/ for NPM/Yarn.

Oh, directly after that followed some subprocess that cloned something and failed. Unfortunately I can’t reproduce the error log now. It had pulled something from GitHub definitely, as the error message was thge same TLS handshake error / timeout. It’s not important now though. Usually npm for whatever reason never has timeouts here. The GFW is long and full of mysteries!

Maybe it was package.json main "typescript": "^5.9.3" }, "scripts": { "dev": "bin/dev", "lint": "concurrently \"pnpm:lint:*(!fix)\" --names \"lint:\"", "lint:fix": "concurrently \"pnpm:lint:*:fix\" --names \"fix:\"", "lint:css": "pnpm stylelint 'app/assets/stylesheets/**/*.scss' $(script/list_bundled_plugins '/assets/stylesheets/**/*.scss') 'themes/**/*.scss'", "lint:css:fix": "pnpm stylelint --fix 'app/assets/stylesheets/**/*.scss' $(script/list_bundled_plugins '/assets/stylesheets/**/*.scss') 'themes/**/*.scss'", "lint:js": "eslint ./frontend $(script/list_bundled_plugins) ./themes --cache --no-error-on-unmatched-pattern --max-warnings 0", "lint:js:fix": "eslint --fix ./frontend $(script/list_bundled_plugins) ./themes --no-error-on-unmatched-pattern", "lint:prettier": "pnpm pprettier --list-different 'app/assets/stylesheets/**/*.scss' 'frontend/**/*.{js,gjs,css}' $(script/list_bundled_plugins '/assets/stylesheets/**/*.scss') $(script/list_bundled_plugins '/{assets,admin/assets,test}/javascripts/**/*.{js,gjs}') 'themes/**/*.{js,gjs,scss}'", "lint:prettier:fix": "pnpm prettier -w --no-error-on-unmatched-pattern 'app/assets/stylesheets/**/*.scss' 'frontend/**/*.{js,gjs,css}' $(script/list_bundled_plugins '/assets/stylesheets/**/*.scss') $(script/list_bundled_plugins '/{assets,admin/assets,test}/javascripts/**/*.{js,gjs}') 'themes/**/*.{js,gjs,scss}'", "lint:types": "ember-tsc -b", "types:generate": "pnpm --dir=frontend/discourse-i18n ember-tsc && pnpm --dir=frontend/discourse-types generate-external", "types:watch": "pnpm types:generate && pnpm ember-tsc -b --watch --preserveWatchOutput", "lttf:ignore": "lint-to-the-future ignore", "lttf:output": "lint-to-the-future output -o ./lint-progress/", "lint-progress": "pnpm lttf:output && npx html-pages ./lint-progress --no-cache", "playwright-install": "playwright install --no-shell chromium", "build": "pnpm --dir=frontend/discourse build", "start": "pnpm --dir=frontend/discourse start"

[image] Falco: "moment-timezone-names-translations": "https://github.com/discourse/moment-timezone-names-translations", You are the man! Exactly that broke my upgrade.

Oh yes, definitely, let’s do that.

Bootstrap / rebuild without cloning everything over again?

Product Support

Falco (Falco) January 10, 2022, 5:55pm 10

That is pretty darn stable, any chance we can push it to the NPM registry now @pmusaraj ?

Topic		Replies	Views
Upgrade from within China fails due to git issues Self-hosting	18	4340	July 12, 2023
Proxy during discourse bootstrap Self-hosting	2	1639	June 1, 2018
Install Discourse on an isolated CentOS 7 server Sysadmins how-to , install	23	11925	July 5, 2022
Can't install reaction plugin Self-hosting reactions	20	1152	June 11, 2024
Continuous Issues when Rebuilding Discourse Self-hosting	5	791	March 19, 2024

Bootstrap / rebuild without cloning everything over again?

Related topics