Rebuilding always fails when the MAXMIND daily limit is exhausted

JammyDodger · July 8, 2024, 8:11am

I’ve just tried a rebuild now and it failed at this point:

 - dist/javascripts/media-optimization-worker.js: 5.01 kB (1.74 kB gzipped)
 - dist/javascripts/pikaday/1.8.2/pikaday.js: 42.54 kB (9.67 kB gzipped)
 - dist/javascripts/squoosh/mozjpeg_enc.js: 39.03 kB (10.81 kB gzipped)
 - dist/javascripts/squoosh/squoosh_resize.js: 4.53 kB (1.28 kB gzipped)
Done in 355.08s.

I, [2024-07-08T07:59:42.015855 #1]  INFO -- : > cd /var/www/discourse && su discourse -c 'SKIP_EMBER_CLI_COMPILE=1 bundle exec rake themes:update assets:precompile'
Purging temp files
Bundling assets
I, [2024-07-08T07:59:57.247206 #1532]  INFO -- : Writing /var/www/discourse/public/assets/break_string-cc617154cd957804f2f6a1f3bc68258c9cdca3d4b9a322bf777d145fed04790e.js
I, [2024-07-08T07:59:57.264957 #1532]  INFO -- : Writing /var/www/discourse/public/assets/service-worker-9764e1cd771b38dbe65a0d1e42d89cb2cb5f72b266ab7316e55d3371cb0ac870.js
I, [2024-07-08T07:59:57.271269 #1532]  INFO -- : Writing /var/www/discourse/public/assets/locales/i18n-3b40e842fd72b9bcc74ea83e094c823cd9ca535e4ecc5e78722e6f99d3656137.js
I, [2024-07-08T07:59:57.277082 #1532]  INFO -- : Writing /var/www/discourse/public/assets/scripts/discourse-test-listen-boot-9b14a0fc65c689577e6a428dcfd680205516fe211700a71c7adb5cbcf4df2cc5.js
I, [2024-07-08T07:59:59.103776 #1532]  INFO -- : Writing /var/www/discourse/public/assets/locales/ar-dfed7a58f30378bc60d7a2cc8d6347524f68b272ae012f0232604f730e442f91.js
I, [2024-07-08T08:00:00.112555 #1532]  INFO -- : Writing /var/www/discourse/public/assets/locales/be-e12ac4c99df2289f422efd371dd3da766754aecb1189ea763fe003376aca9028.js
rake aborted!
Zlib::BufError: buffer error (Zlib::BufError)

kuaza · July 8, 2024, 2:38pm

I have a question. Are you using S3? I suspect that too.

JammyDodger · July 8, 2024, 3:38pm

No No S3 for me.

Roi · July 16, 2024, 7:22pm

I don’t know if this only happens when the daily limit is exhausted. But it happens all the time here for 2-3 months now. Also other people seem to have the same problem:

Maybe it would be a good idea to provide the Maxmind files from a local directory? I download this anyway for other uses, so it is already there.

And maybe it would be even better to provide the Maxmind files in a shared directory into the Discourse container to always have the up-to-date version? As I said I download it anyway and update the files once a day.

kuaza · July 16, 2024, 7:29pm

I think the problem is not related to the limit. In other words, it also gives an error when a wrong key or account ID is entered. This is how we can solve the problem here: in case of error, make it use the old database and continue rebuilding it. Of course, it is very important to determine the main cause of this problem.

I tried to create an example two days ago and it gave an error. It had nothing to do with the limit because it was my first rebuild that day. Then I created a new key and tried it and it worked. There is a situation here: when you create a key, it takes a while for it to become active. If you re-create it immediately, it gives an error.

interesting

Roi · July 16, 2024, 7:39pm

Well, my keys are old, I never changed them after starting to use them. Why is it working when you create a new key? You said it takes a while until it gets active. So it should thrown an error then?

That’s a good idea. Or offering the files from a local directory, as I suggested. Everything optional. But it really bugs me that rebuilding is like a lottery for many weeks now…

tgxworld · July 16, 2024, 10:38pm

This is against Maxmind’s terms of usage which is why we need to make everyone provide the API keys to download the files individually.

Assigning myself to have a look at what the problem is when the daily limit has been reached.

pfaffman · July 17, 2024, 12:54am

I’ve had the rebuild fail and then built the image with the maxmind settings turned off. And then inside the container, add the settings back and successfully run the rake task.

Maybe it’s possible to rebuild without downloading the database and then have the database download when the container boots?

Also, i added a PR that should allow the bootstrap to succeed (that has been merged) even if the download fails, but it’s still causing the bootstrap to fail.

tgxworld · July 17, 2024, 12:58am

Yup I think you’re right here. Maxmind isn’t a critical feature so there is no need for us to cause bootstrap to fail when we can’t download stuff.

Roi · July 17, 2024, 8:15am

I think you did not understand what I tried to say. Let me try again:

I have a script which downloads the Maxmind files every few days. With my own API keys of course. The files are used on the server for several things like AWstats web statistics, WordPress plugins and so on.

So I have the files on the machine anyway. Why download the files (again) when I rebuild the Discourse container?

That’s a brilliant idea.

pfaffman · July 17, 2024, 2:32pm

Yeah. It would be great to be able to have those in persistent storage, especially if it could be shared across instances. I’ve got a handful of Discourse sites on the same machine behind traefik, so if they could all share the same mmdb then it’d save keeping and downloading a bunch of separate copies. Even on a standard install, they could persiste between builds. Maybe that’s possible already? Maybe just mount /var/www/discourse/vendor/data somewhere persistent?

Aha. I think that’s what GlobalSetting.maxmind_backup_path is for. So I think that if you have a maxminddb for some reason you can set DISCOURSE_MAXMIND_BACKUP_PATH to something that’s available in the container.

Also, why do we need mmdb to precompile assets?

github.com

discourse/discourse/blob/main/lib/tasks/assets.rake#L330


      
            end
          end
          
          task "assets:precompile:theme_transpiler": "environment" do
            DiscourseJsProcessor::Transpiler.build_theme_transpiler
          end
          
          # Run these tasks **before** Rails' "assets:precompile" task
          task "assets:precompile": %w[
                 assets:precompile:before
                 maxminddb:refresh
                 assets:precompile:theme_transpiler
               ]
          
          # Run these tasks **after** Rails' "assets:precompile" task
          Rake::Task["assets:precompile"].enhance do
            Rake::Task["assets:precompile:compress_js"].invoke
            Rake::Task["assets:precompile:css"].invoke
          end

Roi · July 18, 2024, 12:38pm

So is this already working? Will setting up DISCOURSE_MAXMIND_BACKUP_PATH in app.yml (as internal link from inside the container or absolute link on the Docker host?) disable a download from Maxmind during rebuild?

pfaffman · July 18, 2024, 12:54pm

It’s in the code. I have never used it. It looks like if you include a path there it’ll find it. I can’t remember, but I think it’s a path to the directory.

Roi · July 18, 2024, 2:04pm

Okay, I could try. I have several instances and one is for testing only.

Can I see if the files were taken locally or were downloaded?

So something like /var/discourse/maxmind on the Docker host?

pfaffman · July 18, 2024, 2:11pm

Sorry. I’m not quite sure. It looks like it wants a directory, and you can make it whatever you want, so maybe you’d add a a volume in your app.yml like

  - volume:
      host: /data/
      guest: /data

and DISCOURSE_maxmind_backup_path=/data/mmdb (fixing case for the env variable)

here’s the code

github.com

discourse/discourse/blob/6344e3f9379d048fef2c486423dcd4d2a7a6f443/lib/tasks/maxminddb.rake#L49


      
          end
          
          maxmind_thread = nil
          
          task "maxminddb:refresh": "environment" do
            refresh_days = GlobalSetting.refresh_maxmind_db_during_precompile_days
            next if refresh_days.to_i <= 0
          
            mmdb_time = get_mmdb_time(DiscourseIpInfo.path)
          
            if GlobalSetting.maxmind_backup_path.present?
              backup_mmdb_time = get_mmdb_time(GlobalSetting.maxmind_backup_path)
              puts "Detected MaxMindDB backup (downloaded: #{backup_mmdb_time}) at #{GlobalSetting.maxmind_backup_path}"
              mmdb_time ||= backup_mmdb_time
            end
          
            if backup_mmdb_time && backup_mmdb_time >= mmdb_time
              copy_maxmind(GlobalSetting.maxmind_backup_path, DiscourseIpInfo.path)
              mmdb_time = backup_mmdb_time
            end

github.com

discourse/discourse/blob/6344e3f9379d048fef2c486423dcd4d2a7a6f443/lib/tasks/maxminddb.rake#L13


      
          GEOLITE_DBS ||= %w[GeoLite2-City GeoLite2-ASN]
          
          desc "downloads MaxMind's GeoLite2-City databases"
          task "maxminddb:get" => "environment" do
            GEOLITE_DBS.each do |name|
              puts "Downloading MaxMindDb's #{name}..."
              DiscourseIpInfo.mmdb_download(name)
            end
          end
          
          def get_mmdb_time(root_path)
            mmdb_time = nil
          
            GEOLITE_DBS.each do |name|
              path = File.join(root_path, "#{name}.mmdb")
          
              if File.exist?(path)
                mmdb_time = File.mtime(path)
              else
                mmdb_time = nil
                break

Roi · July 18, 2024, 2:35pm

Thank you again. I created /var/discourse/shared/discourse_test/data/mmdb and here is what I did to app.yml:

  ## The maxmind geolocation IP address key for IP address lookup
  ## see https://meta.discourse.org/t/-/137387/23 for details
  DISCOURSE_MAXMIND_ACCOUNT_ID: 123456
  DISCOURSE_MAXMIND_LICENSE_KEY: abcdefg
  DISCOURSE_MAXMIND_BACKUP_PATH: /data/mmdb

## The Docker container is stateless; all data is stored in /shared
volumes:
  - volume:
      host: /var/discourse/shared/discourse_test
      guest: /shared
  - volume:
      host: /var/discourse/shared/discourse_test/log/var-log
      guest: /var/log
  - volume:
      host: /var/discourse/shared/discourse_test/data
      guest: /data

I added DISCOURSE_MAXMIND_BACKUP_PATH and I added a third volume.

Is the directory for DISCOURSE_MAXMIND_BACKUP_PATH correct? The path is the path inside the container?

Do I need to remove DISCOURSE_MAXMIND_ACCOUNT_ID and DISCOURSE_MAXMIND_LICENSE_KEY now?

Sorry, too excited and also a bit unclear.

Roi · July 18, 2024, 3:55pm

Okayyyyyy

Detected MaxMindDB backup (downloaded: 2024-07-17 23:15:02 +0000) at /data/mmdb
Copying MaxMindDB from /data/mmdb to /var/www/discourse/vendor/data
Skip downloading MaxMindDB as it was last downloaded at 2024-07-17 23:15:02 +0000

I think this is what I (or better “we”) wanted to see.

pfaffman · July 18, 2024, 6:12pm

I think you could have skipped the extra volume and done

DISCOURSE_MAXMIND_BACKUP_PATH: /shared/data

But if you have some other process that’s keeping that database up to date elsewhere, then you could mount that directory so you’d have just one local copy on your drive and would need to update just that one copy.

Roi · July 19, 2024, 4:54pm

I think I will leave it that way. In my eyes it is a more clear configuration which also can be still understood in a few months. Also it might be more generic and for more use cases than just mine.

I’m copying the three mmdb files from Maxmind into that directory when downloading them. I just adjusted the script I am using (btw, for download I am using geoipupdate, which is also available as a package for Debian (and most likely other Linux distributions)).

Just finished rebuild four different Discourse containers, all without any error! This did not happen for months here. So great!

kuaza · July 26, 2024, 11:42pm

This problem still persists. I will explain the incident in detail:
I made an update from the admin, it stopped halfway. So I started a recompile from the ssh panel. Boom gave an error, sample error is below.

Then I created a new maxmind key and recompiled it, it gave an error again, same error. (interesting, maybe the key was not activated).

Then I disabled the maxmind setting in the app.yml file. I recompiled it and the compilation was successful.

The add-ons I used are shown in the image, other things I used:

Cloudflare R2
separate postgresql server
cloudflare

I can’t figure out what’s causing the problem anymore.

Topic		Replies	Views
Update stuck on Downloading MaxMindDB support	2	138	June 5, 2024
Server load issues (after rebuild hanged) installation	4	182	May 19, 2024
Rebuild app hangs dev	3	1020	May 24, 2017
Error while updating support	2	368	November 10, 2020
Understanding MaxMind privacy for our users support maxmind	2	508	December 23, 2020

Rebuilding always fails when the MAXMIND daily limit is exhausted

Related Topics