Could this be fixed before the next release, please! I’m testing deployment of new VPSs (different IP every time) and it still fails two out of three times.
2 Likes
Note that we just shipped a quick fix which would prevent the rebuild from failing when downloading the maxmind databases fail.
Once Discourse 3.3 has been released, we will relook into how we can make download Maxmind databases less error prone.
4 Likes
Hm, it still fails for me (clean install, not rebuild)?
I, [2024-07-30T05:19:47.753779 #1] INFO -- : > cd /var/www/discourse && su discourse -c 'SKIP_EMBER_CLI_COMPILE=1 bundle exec rake themes:update assets:precompile'
Purging temp files
Bundling assets
I, [2024-07-30T05:19:58.764890 #1059] INFO -- : Writing /var/www/discourse/public/assets/break_string-cc617154cd957804f2f6a1f3bc68258c9cdca3d4b9a322bf777d145fed04790e.js
I, [2024-07-30T05:19:58.787966 #1059] INFO -- : Writing /var/www/discourse/public/assets/service-worker-77c5660ded6e2c94a6a5ce2594411b46d0e15a1f5b9065e57b85652d7cbe3ead.js
I, [2024-07-30T05:19:58.797458 #1059] INFO -- : Writing /var/www/discourse/public/assets/locales/i18n-3b40e842fd72b9bcc74ea83e094c823cd9ca535e4ecc5e78722e6f99d3656137.js
I, [2024-07-30T05:19:58.807019 #1059] INFO -- : Writing /var/www/discourse/public/assets/scripts/discourse-test-listen-boot-9b14a0fc65c689577e6a428dcfd680205516fe211700a71c7adb5cbcf4df2cc5.js
rake aborted!
Zlib::BufError: buffer error (Zlib::BufError)
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sprockets-3.7.3/lib/sprockets/cache/file_store.rb:100:in `<<'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sprockets-3.7.3/lib/sprockets/cache/file_store.rb:100:in `set'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sprockets-3.7.3/lib/sprockets/cache.rb:212:in `set'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sprockets-3.7.3/lib/sprockets/cache.rb:136:in `set'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sprockets-3.7.3/lib/sprockets/loader.rb:243:in `store_asset'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sprockets-3.7.3/lib/sprockets/loader.rb:185:in `load_from_unloaded'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sprockets-3.7.3/lib/sprockets/loader.rb:60:in `block in load'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sprockets-3.7.3/lib/sprockets/loader.rb:317:in `fetch_asset_from_dependency_cache'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sprockets-3.7.3/lib/sprockets/loader.rb:44:in `load'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sprockets-3.7.3/lib/sprockets/cached_environment.rb:20:in `block in initialize'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sprockets-3.7.3/lib/sprockets/cached_environment.rb:47:in `load'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sprockets-3.7.3/lib/sprockets/base.rb:66:in `find_asset'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sprockets-3.7.3/lib/sprockets/base.rb:73:in `find_all_linked_assets'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sprockets-3.7.3/lib/sprockets/manifest.rb:134:in `block in find'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sprockets-3.7.3/lib/sprockets/manifest.rb:133:in `each'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sprockets-3.7.3/lib/sprockets/manifest.rb:133:in `find'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sprockets-3.7.3/lib/sprockets/manifest.rb:186:in `compile'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sprockets-rails-3.5.1/lib/sprockets/rails/task.rb:67:in `block (3 levels) in define'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sprockets-3.7.3/lib/rake/sprocketstask.rb:147:in `with_logger'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sprockets-rails-3.5.1/lib/sprockets/rails/task.rb:66:in `block (2 levels) in define'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/exe/rake:27:in `<top (required)>'
/usr/local/bin/bundle:25:in `load'
/usr/local/bin/bundle:25:in `<main>'
Tasks: TOP => assets:precompile
(See full trace by running task with --trace)
I, [2024-07-30T05:20:01.382260 #1] INFO -- : Downloading MaxMindDB...
I, [2024-07-30T05:20:01.438870 #1] INFO -- : Terminating async processes
I, [2024-07-30T05:20:01.444360 #1] INFO -- : Sending INT to HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-
cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main pid: 57
2024-07-30 05:20:01.448 UTC [57] LOG: received fast shutdown request
I, [2024-07-30T05:20:01.449248 #1] INFO -- : Sending TERM to exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf p
id: 120
120:signal-handler (1722316801) Received SIGTERM scheduling shutdown...
2024-07-30 05:20:01.512 UTC [57] LOG: aborting any active transactions
120:M 30 Jul 2024 05:20:01.523 # User requested shutdown...
120:M 30 Jul 2024 05:20:01.526 * Saving the final RDB snapshot before exiting.
2024-07-30 05:20:01.566 UTC [57] LOG: background worker "logical replication launcher" (PID 72) exited with exit code 1
120:M 30 Jul 2024 05:20:01.586 * DB saved on disk
120:M 30 Jul 2024 05:20:01.589 # Redis is now ready to exit, bye bye...
2024-07-30 05:20:01.598 UTC [67] LOG: shutting down
2024-07-30 05:20:02.090 UTC [57] LOG: database system is shut down
FAILED
--------------------
Pups::ExecError: cd /var/www/discourse && su discourse -c 'SKIP_EMBER_CLI_COMPILE=1 bundle exec rake themes:update assets:precompile' fa
iled with return #<Process::Status: pid 1057 exit 1>
Location of failure: /usr/local/lib/ruby/gems/3.3.0/gems/pups-1.2.1/lib/pups/exec_command.rb:132:in `spawn'
exec failed with the params {"cd"=>"$home", "tag"=>"precompile", "hook"=>"assets_precompile", "cmd"=>["su discourse -c 'SKIP_EMBER_CLI_COMPILE=1 bundle exec rake themes:update assets:precompile'"]}
bootstrap failed with exit code 1
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.
./discourse-doctor may help diagnose the problem.
c9da27502f7c288ec5465ca82b4830123a3c59003527b8f288457c594576adaa
x86_64 arch detected.
Just to confirm, you followed discourse/docs/INSTALL-cloud.md at main · discourse/discourse · GitHub and have valid credentials for maxmind?
Yes, standard install, valid maxmind account_id and license_key in app.yml
Just to confirm, you’re on the tests-passed branch and not stable? I’ve just tested it with an invalid maxmind key and am not seeing any errors.
I’m not sure if the issue itself is anymore about wrong credentials, but rebuilding has tendency fail with right credentials too.
2 Likes
Yes
Exactly, I’m using a valid key! (second attempt just now, new VPS, different IP, still fails with the same error).
1 Like
I got the error while updating docker from admin. I wonder if the problem is caused by the created files. This error always occurs in the same way and we do not get an error when we close maxmind. There is a problem between these two structures and something is affecting each other:
I’m starting to think that Maxmind is a red herring here. The backtrace doesn’t point to Maxmind at all and is indicating an error being raised in sprockets. I’ll have to dig further to see if I can reproduce this on a droplet.
3 Likes
I think the same. Maxmind may not be a problem, but its activity brings problems in some areas to the forefront.
1 Like
Are you running this on digitalocean? If so, can I know the number of vCPUs and RAM you’ve selected?
No, I have my own server and PostgreSQL is on a separate server. I made a note here, you can review it:
How much RAM and CPU does your server have? I’m trying to reproduce this and have been rebuilding many times on my droplet but can’t hit the error.
25gb ram main server
20gb ram postgresql server
8 cpu
cpu cores 4
db_shared_buffers: "4096MB"
UNICORN_WORKERS: 8
UNICORN_SIDEKIQ_MAX_RSS: 1500
## Plugins go here
## see https://meta.discourse.org/t/19157 for details
hooks:
after_code:
- exec:
cd: $home/plugins
cmd:
- git clone https://github.com/discourse/docker_manager.git
- git clone https://github.com/discourse/discourse-question-answer.git
- git clone https://github.com/discourse/discourse-solved.git
- git clone https://github.com/discourse/discourse-gamification.git
- git clone https://github.com/discourse/discourse-restricted-replies.git
- git clone https://github.com/discourse/discourse-saved-searches.git
- git clone https://github.com/discourse/discourse-follow.git
# - git clone https://github.com/discourse/discourse-yearly-review.git
- git clone https://github.com/discourse/discourse-voting.git
- git clone https://github.com/discourse/discourse-no-bump.git
- git clone https://github.com/discourse/discourse-math.git
- git clone https://github.com/discourse/discourse-akismet.git
- git clone https://github.com/discourse/discourse-adplugin.git
- git clone https://github.com/spirobel/discourse-matheditor.git
- git clone https://github.com/discourse/discourse-google-one-tap.git
- git clone https://github.com/discourse/discourse-ai.git
# - git clone https://github.com/paviliondev/discourse-custom-wizard.git
- git clone https://github.com/discourse/discourse-reactions.git
- git clone https://github.com/discourse/discourse-subscriptions.git
- git clone https://github.com/discourse/discourse-assign.git
# - git clone https://github.com/discourse/discourse-policy.git
- git clone https://github.com/discourse/discourse-data-explorer.git
# - git clone https://github.com/merefield/discourse-chatbot.git
# after_assets_precompile:
# - exec:
# cd: $home
# cmd:
# - sudo -E -u discourse bundle exec rake s3:upload_assets
# - sudo -E -u discourse bundle exec rake s3:expire_missing_assets
# - sudo -E -u discourse bundle exec rake uploads:migrate_to_s3
I had the issue today on a machine with (I think) 16gb of ram (but other things are running) and 12 or more threads (I can get exact numbers tomorrow). It does seem like maxmind is a red herring, but every time, disabling maxmind always solves it (that’s what I did today) .
But now you and I both have tried adding rescue on the maxmind function and it’s not fixing it.
Maybe it’s something happening in the precompile stuff before?
2 Likes
I harbor doubts about this: assets:precompile
By following this sequence of events and examining the incidents that occurred between them, we may gain some insight:
Bundling assets
…
Downloading MaxMindDB…
… (Perhaps the mistake lies somewhere in between)
assets:precompile
I just realised that we download the Maxmind databases in a background thread while running assets:precompile at the same time. Based on the backtraces I am seeing, I have a hypothesis that the background thread which we use to download the Maxmind database is messing with assets:precompile somehow.
I’m going to ship the fix above first and not dive too deep into the root cause since we are going to be moving downloading of Maxmind databases out of the build process.
5 Likes
That sounds like a great idea. I’ll try turning maxmind back on on a few sites tomorrow.
Can I test this now? Because when the update from the admin gave an error, I wanted to do it from ssh and got an error there too. If you tell me how to do it, I can do the first try.
1 Like