# Herbouwen mislukt altijd wanneer de dagelijkse limiet van MAXMIND is uitgeput
**URL:** https://meta.discourse.org/t/rebuilding-always-fails-when-the-maxmind-daily-limit-is-exhausted/312775
**Category:** Bug
**Tags:** maxmind, dev-ops, fixed
**Created:** [19 juni 2024 om 15:13 UTC](https://meta.discourse.org/t/rebuilding-always-fails-when-the-maxmind-daily-limit-is-exhausted/312775 "2024-06-19T15:13:54Z")
**Posts on this page:** 20
**Page:** 2
### Author:  [@JammyDodger](https://meta.discourse.org/u/JammyDodger)
#### Post date: [8 juli 2024 om 08:11 UTC](https://meta.discourse.org/t/rebuilding-always-fails-when-the-maxmind-daily-limit-is-exhausted/312775/23 "2024-07-08T08:11:51Z")
I’ve just tried a rebuild now and it failed at this point:
```plaintext
- dist/javascripts/media-optimization-worker.js: 5.01 kB (1.74 kB gzipped)
- dist/javascripts/pikaday/1.8.2/pikaday.js: 42.54 kB (9.67 kB gzipped)
- dist/javascripts/squoosh/mozjpeg_enc.js: 39.03 kB (10.81 kB gzipped)
- dist/javascripts/squoosh/squoosh_resize.js: 4.53 kB (1.28 kB gzipped)
Done in 355.08s.
I, [2024-07-08T07:59:42.015855 #1] INFO -- : > cd /var/www/discourse && su discourse -c 'SKIP_EMBER_CLI_COMPILE=1 bundle exec rake themes:update assets:precompile'
Purging temp files
Bundling assets
I, [2024-07-08T07:59:57.247206 #1532] INFO -- : Writing /var/www/discourse/public/assets/break_string-cc617154cd957804f2f6a1f3bc68258c9cdca3d4b9a322bf777d145fed04790e.js
I, [2024-07-08T07:59:57.264957 #1532] INFO -- : Writing /var/www/discourse/public/assets/service-worker-9764e1cd771b38dbe65a0d1e42d89cb2cb5f72b266ab7316e55d3371cb0ac870.js
I, [2024-07-08T07:59:57.271269 #1532] INFO -- : Writing /var/www/discourse/public/assets/locales/i18n-3b40e842fd72b9bcc74ea83e094c823cd9ca535e4ecc5e78722e6f99d3656137.js
I, [2024-07-08T07:59:57.277082 #1532] INFO -- : Writing /var/www/discourse/public/assets/scripts/discourse-test-listen-boot-9b14a0fc65c689577e6a428dcfd680205516fe211700a71c7adb5cbcf4df2cc5.js
I, [2024-07-08T07:59:59.103776 #1532] INFO -- : Writing /var/www/discourse/public/assets/locales/ar-dfed7a58f30378bc60d7a2cc8d6347524f68b272ae012f0232604f730e442f91.js
I, [2024-07-08T08:00:00.112555 #1532] INFO -- : Writing /var/www/discourse/public/assets/locales/be-e12ac4c99df2289f422efd371dd3da766754aecb1189ea763fe003376aca9028.js
rake aborted!
Zlib::BufError: buffer error (Zlib::BufError)
```
---
### Author:  [@kuaza](https://meta.discourse.org/u/kuaza)
#### Post date: [8 juli 2024 om 14:38 UTC](https://meta.discourse.org/t/rebuilding-always-fails-when-the-maxmind-daily-limit-is-exhausted/312775/24 "2024-07-08T14:38:00Z")
I have a question. Are you using S3? I suspect that too.
---
### Author:  [@JammyDodger](https://meta.discourse.org/u/JammyDodger)
#### Post date: [8 juli 2024 om 15:38 UTC](https://meta.discourse.org/t/rebuilding-always-fails-when-the-maxmind-daily-limit-is-exhausted/312775/25 "2024-07-08T15:38:04Z")
No :baymax_no: No S3 for me.
---
### Author:  [@Roi](https://meta.discourse.org/u/Roi)
#### Post date: [16 juli 2024 om 19:22 UTC](https://meta.discourse.org/t/rebuilding-always-fails-when-the-maxmind-daily-limit-is-exhausted/312775/26 "2024-07-16T19:22:00Z")
I don’t know if this only happens when the daily limit is exhausted. But it happens all the time here for 2-3 months now. Also other people seem to have the same problem:
> [@"Zlib::BufError: buffer error" - Discourse update fails](https://meta.discourse.org/t/zlib-buffer-error-discourse-update-fails/311347):
>
> Plugin name is 'discourse-education-group-custom-fields', but plugin directory is named 'discourse-group-custom-fields' Purging temp files Bundling assets I, [2024-06-10T06:41:22.964825 #1211] INFO -- : Writing /var/www/discourse/public/assets/break\_string-cc617154cd957804f2f6a1f3bc68258c9cdca3d4b9a322bf777d145fed04790e.js I, [2024-06-10T06:41:22.986328 #1211] INFO -- : Writing /var/www/discourse/public/assets/ser…
Maybe it would be a good idea to provide the Maxmind files from a local directory? I download this anyway for other uses, so it is already there.
And maybe it would be even better to provide the Maxmind files in a shared directory into the Discourse container to always have the up-to-date version? As I said I download it anyway and update the files once a day.
---
### Author:  [@kuaza](https://meta.discourse.org/u/kuaza)
#### Post date: [16 juli 2024 om 19:29 UTC](https://meta.discourse.org/t/rebuilding-always-fails-when-the-maxmind-daily-limit-is-exhausted/312775/27 "2024-07-16T19:29:54Z")
I think the problem is not related to the limit. In other words, it also gives an error when a wrong key or account ID is entered. This is how we can solve the problem here: in case of error, make it use the old database and continue rebuilding it. Of course, it is very important to determine the main cause of this problem.
I tried to create an example two days ago and it gave an error. It had nothing to do with the limit because it was my first rebuild that day. Then I created a new key and tried it and it worked. There is a situation here: when you create a key, it takes a while for it to become active. If you re-create it immediately, it gives an error.
interesting 🙂
---
### Author:  [@Roi](https://meta.discourse.org/u/Roi)
#### Post date: [16 juli 2024 om 19:39 UTC](https://meta.discourse.org/t/rebuilding-always-fails-when-the-maxmind-daily-limit-is-exhausted/312775/28 "2024-07-16T19:39:01Z")
Well, my keys are old, I never changed them after starting to use them. Why is it working when you create a new key? You said it takes a while until it gets active. So it should thrown an error then?
> [@kuaza](#):
>
> This is how we can solve the problem here: in case of error, make it use the old database and continue rebuilding it.
That’s a good idea. Or offering the files from a local directory, as I suggested. Everything optional. But it really bugs me that rebuilding is like a lottery for many weeks now…
---
### Author:  [@tgxworld](https://meta.discourse.org/u/tgxworld)
#### Post date: [16 juli 2024 om 22:38 UTC](https://meta.discourse.org/t/rebuilding-always-fails-when-the-maxmind-daily-limit-is-exhausted/312775/29 "2024-07-16T22:38:31Z")
> [@Roi](#):
>
> And maybe it would be even better to provide the Maxmind files in a shared directory into the Discourse container to always have the up-to-date version?
This is against Maxmind’s terms of usage which is why we need to make everyone provide the API keys to download the files individually.
Assigning myself to have a look at what the problem is when the daily limit has been reached.
---
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [17 juli 2024 om 00:54 UTC](https://meta.discourse.org/t/rebuilding-always-fails-when-the-maxmind-daily-limit-is-exhausted/312775/31 "2024-07-17T00:54:21Z")
I’ve had the rebuild fail and then built the image with the maxmind settings turned off. And then inside the container, add the settings back and successfully run the rake task.
Maybe it’s possible to rebuild without downloading the database and then have the database download when the container boots?
Also, i added a PR that should allow the bootstrap to succeed (that has been merged) even if the download fails, but it’s still causing the bootstrap to fail.
---
### Author:  [@tgxworld](https://meta.discourse.org/u/tgxworld)
#### Post date: [17 juli 2024 om 00:58 UTC](https://meta.discourse.org/t/rebuilding-always-fails-when-the-maxmind-daily-limit-is-exhausted/312775/32 "2024-07-17T00:58:33Z")
Yup I think you’re right here. Maxmind isn’t a critical feature so there is no need for us to cause bootstrap to fail when we can’t download stuff.
---
### Author:  [@Roi](https://meta.discourse.org/u/Roi)
#### Post date: [17 juli 2024 om 08:15 UTC](https://meta.discourse.org/t/rebuilding-always-fails-when-the-maxmind-daily-limit-is-exhausted/312775/33 "2024-07-17T08:15:47Z")
> [@tgxworld](#):
>
> > [@Roi](#):
> >
> > And maybe it would be even better to provide the Maxmind files in a shared directory into the Discourse container to always have the up-to-date version?
>
> This is against Maxmind’s terms of usage which is why we need to make everyone provide the API keys to download the files individually.
I think you did not understand what I tried to say. Let me try again:
I have a script which downloads the Maxmind files every few days. With my own API keys of course. The files are used on the server for several things like AWstats web statistics, WordPress plugins and so on.
So I have the files on the machine anyway. Why download the files (again) when I rebuild the Discourse container?
> [@pfaffman](#):
>
> Maybe it’s possible to rebuild without downloading the database and then have the database download when the container boots?
That’s a brilliant idea. 🙂
---
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [17 juli 2024 om 14:32 UTC](https://meta.discourse.org/t/rebuilding-always-fails-when-the-maxmind-daily-limit-is-exhausted/312775/34 "2024-07-17T14:32:23Z")
> [@Roi](#):
>
> So I have the files on the machine anyway. Why download the files (again) when I rebuild the Discourse container?
Yeah. It would be great to be able to have those in persistent storage, especially if it could be shared across instances. I’ve got a handful of Discourse sites on the same machine behind traefik, so if they could all share the same mmdb then it’d save keeping and downloading a bunch of separate copies. Even on a [standard install](https://meta.discourse.org/t/142537?silent=true), they could persiste between builds. Maybe that’s possible already? Maybe just mount `/var/www/discourse/vendor/data` somewhere persistent?
Aha. I think that’s what `GlobalSetting.maxmind_backup_path` is for. So I think that if you have a maxminddb for some reason you can set `DISCOURSE_MAXMIND_BACKUP_PATH` to something that’s available in the container.
Also, why do we need mmdb to precompile assets?
>
---
### Author:  [@Roi](https://meta.discourse.org/u/Roi)
#### Post date: [18 juli 2024 om 12:38 UTC](https://meta.discourse.org/t/rebuilding-always-fails-when-the-maxmind-daily-limit-is-exhausted/312775/35 "2024-07-18T12:38:51Z")
> [@pfaffman](#):
>
> Aha. I think that’s what `GlobalSetting.maxmind_backup_path` is for. So I think that if you have a maxminddb for some reason you can set `DISCOURSE_MAXMIND_BACKUP_PATH` to something that’s available in the container.
So is this already working? Will setting up `DISCOURSE_MAXMIND_BACKUP_PATH` in app.yml (as internal link from inside the container or absolute link on the Docker host?) disable a download from Maxmind during rebuild?
---
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [18 juli 2024 om 12:54 UTC](https://meta.discourse.org/t/rebuilding-always-fails-when-the-maxmind-daily-limit-is-exhausted/312775/36 "2024-07-18T12:54:31Z")
It’s in the code. I have never used it. It looks like if you include a path there it’ll find it. I can’t remember, but I think it’s a path to the directory.
---
### Author:  [@Roi](https://meta.discourse.org/u/Roi)
#### Post date: [18 juli 2024 om 14:04 UTC](https://meta.discourse.org/t/rebuilding-always-fails-when-the-maxmind-daily-limit-is-exhausted/312775/37 "2024-07-18T14:04:54Z")
Okay, I could try. I have several instances and one is for testing only.
Can I see if the files were taken locally or were downloaded?
> [@pfaffman](#):
>
> I can’t remember, but I think it’s a path to the directory.
So something like `/var/discourse/maxmind` on the Docker host?
---
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [18 juli 2024 om 14:11 UTC](https://meta.discourse.org/t/rebuilding-always-fails-when-the-maxmind-daily-limit-is-exhausted/312775/38 "2024-07-18T14:11:27Z")
Sorry. I’m not quite sure. It looks like it wants a directory, and you can make it whatever you want, so maybe you’d add a a volume in your `app.yml` like
```plaintext
- volume:
host: /data/
guest: /data
```
and `DISCOURSE_maxmind_backup_path=/data/mmdb` (fixing case for the env variable)
here’s the code
>
>
---
### Author:  [@Roi](https://meta.discourse.org/u/Roi)
#### Post date: [18 juli 2024 om 14:35 UTC](https://meta.discourse.org/t/rebuilding-always-fails-when-the-maxmind-daily-limit-is-exhausted/312775/39 "2024-07-18T14:35:17Z")
Thank you again. I created `/var/discourse/shared/discourse_test/data/mmdb` and here is what I did to `app.yml`:
```plaintext
## The maxmind geolocation IP address key for IP address lookup
## see https://meta.discourse.org/t/-/137387/23 for details
DISCOURSE_MAXMIND_ACCOUNT_ID: 123456
DISCOURSE_MAXMIND_LICENSE_KEY: abcdefg
DISCOURSE_MAXMIND_BACKUP_PATH: /data/mmdb
## The Docker container is stateless; all data is stored in /shared
volumes:
- volume:
host: /var/discourse/shared/discourse_test
guest: /shared
- volume:
host: /var/discourse/shared/discourse_test/log/var-log
guest: /var/log
- volume:
host: /var/discourse/shared/discourse_test/data
guest: /data
```
I added `DISCOURSE_MAXMIND_BACKUP_PATH` and I added a third volume.
Is the directory for `DISCOURSE_MAXMIND_BACKUP_PATH` correct? The path is the path inside the container?
Do I need to remove `DISCOURSE_MAXMIND_ACCOUNT_ID` and `DISCOURSE_MAXMIND_LICENSE_KEY` now?
Sorry, too excited and also a bit unclear. 😉
---
### Author:  [@Roi](https://meta.discourse.org/u/Roi)
#### Post date: [18 juli 2024 om 15:55 UTC](https://meta.discourse.org/t/rebuilding-always-fails-when-the-maxmind-daily-limit-is-exhausted/312775/40 "2024-07-18T15:55:02Z")
Okayyyyyy 😃
```plaintext
Detected MaxMindDB backup (downloaded: 2024-07-17 23:15:02 +0000) at /data/mmdb
Copying MaxMindDB from /data/mmdb to /var/www/discourse/vendor/data
Skip downloading MaxMindDB as it was last downloaded at 2024-07-17 23:15:02 +0000
```
I think this is what I (or better “we”) wanted to see. 😃
---
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [18 juli 2024 om 18:12 UTC](https://meta.discourse.org/t/rebuilding-always-fails-when-the-maxmind-daily-limit-is-exhausted/312775/41 "2024-07-18T18:12:28Z")
I think you could have skipped the extra volume and done
```plaintext
DISCOURSE_MAXMIND_BACKUP_PATH: /shared/data
```
But if you have some other process that’s keeping that database up to date elsewhere, then you could mount that directory so you’d have just one local copy on your drive and would need to update just that one copy.
---
### Author:  [@Roi](https://meta.discourse.org/u/Roi)
#### Post date: [19 juli 2024 om 16:54 UTC](https://meta.discourse.org/t/rebuilding-always-fails-when-the-maxmind-daily-limit-is-exhausted/312775/42 "2024-07-19T16:54:07Z")
I think I will leave it that way. In my eyes it is a more clear configuration which also can be still understood in a few months. Also it might be more generic and for more use cases than just mine.
I’m copying the three mmdb files from Maxmind into that directory when downloading them. I just adjusted the script I am using (btw, for download I am using `geoipupdate`, which is also available as a package for Debian (and most likely other Linux distributions)).
Just finished rebuild four different Discourse containers, all without any error! This did not happen for months here. So great! 🙂
---
### Author:  [@kuaza](https://meta.discourse.org/u/kuaza)
#### Post date: [26 juli 2024 om 23:42 UTC](https://meta.discourse.org/t/rebuilding-always-fails-when-the-maxmind-daily-limit-is-exhausted/312775/43 "2024-07-26T23:42:54Z")
This problem still persists. I will explain the incident in detail:
I made an update from the admin, it stopped halfway. So I started a recompile from the ssh panel. Boom gave an error, sample error is below.
Then I created a new maxmind key and recompiled it, it gave an error again, same error. (interesting, maybe the key was not activated).
Then I disabled the maxmind setting in the app.yml file. I recompiled it and the compilation was successful.
**The add-ons I used are shown in the image, other things I used:**
- Cloudflare R2
- separate postgresql server
- cloudflare
I can’t figure out what’s causing the problem anymore.
[Previous page](https://meta.discourse.org/t/rebuilding-always-fails-when-the-maxmind-daily-limit-is-exhausted/312775.md?page=1)
[Next page](https://meta.discourse.org/t/rebuilding-always-fails-when-the-maxmind-daily-limit-is-exhausted/312775.md?page=3)