Upgrade / Rebuilds Fail due to MaxMind DB EOL

Guys,

Hope somebody can help here. Ive not had a problem for over 12 months with upgrades to my forum that is running in a docker container on EC2

Ive tried:

  • launcher cleanup but no issues with disk space
  • latest git pull
  • multiple app rebuilds
  • Even tried a discourse-setup and was going to try a DB restore :frowning:
  • Server reboot
  • apt-get update; apt-get upgrade

Here is the tail of the log

Done compressing all JS files : 88.88 secs

rake aborted!

SocketError: Failed to open TCP connection to geolite.maxmind.com:443 (getaddrinfo: No address associated with hostname)

/var/www/discourse/lib/final_destination.rb:411:in `safe_session'

/var/www/discourse/lib/final_destination.rb:362:in `safe_get'

/var/www/discourse/lib/final_destination.rb:131:in `get'

/var/www/discourse/lib/file_helper.rb:51:in `download'

/var/www/discourse/lib/discourse_ip_info.rb:30:in `mmdb_download'

/var/www/discourse/lib/tasks/assets.rake:220:in `block (3 levels) in <top (required)>'

/var/www/discourse/lib/tasks/assets.rake:219:in `each'

/var/www/discourse/lib/tasks/assets.rake:219:in `block (2 levels) in <top (required)>'

Caused by:

SocketError: getaddrinfo: No address associated with hostname

/var/www/discourse/lib/final_destination.rb:411:in `safe_session'

/var/www/discourse/lib/final_destination.rb:362:in `safe_get'

/var/www/discourse/lib/final_destination.rb:131:in `get'

/var/www/discourse/lib/file_helper.rb:51:in `download'

/var/www/discourse/lib/discourse_ip_info.rb:30:in `mmdb_download'

/var/www/discourse/lib/tasks/assets.rake:220:in `block (3 levels) in <top (required)>'

/var/www/discourse/lib/tasks/assets.rake:219:in `each'

/var/www/discourse/lib/tasks/assets.rake:219:in `block (2 levels) in <top (required)>'

Tasks: TOP => assets:precompile

(See full trace by running task with --trace)

I, [2019-12-30T18:56:27.608998 #1] INFO -- : Downloading MaxMindDB...

Compressing Javascript and Generating Source Maps

I, [2019-12-30T18:56:27.633923 #1] INFO -- : Terminating async processes

I, [2019-12-30T18:56:27.637383 #1] INFO -- : Sending INT to HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/10/bin/postmaster -D /etc/postgresql/10/main pid: 49

I, [2019-12-30T18:56:27.639770 #1] INFO -- : Sending TERM to exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf pid: 166

166:signal-handler (1577732187) Received SIGTERM scheduling shutdown...

2019-12-30 18:56:27.639 UTC [49] LOG: received fast shutdown request

2019-12-30 18:56:27.654 UTC [49] LOG: aborting any active transactions

166:M 30 Dec 2019 18:56:27.666 # User requested shutdown...

166:M 30 Dec 2019 18:56:27.694 * Saving the final RDB snapshot before exiting.

2019-12-30 18:56:27.702 UTC [49] LOG: worker process: logical replication launcher (PID 58) exited with exit code 1

2019-12-30 18:56:27.711 UTC [53] LOG: shutting down

2019-12-30 18:56:27.819 UTC [49] LOG: database system is shut down

166:M 30 Dec 2019 18:56:27.885 * DB saved on disk

166:M 30 Dec 2019 18:56:27.886 # Redis is now ready to exit, bye bye...

FAILED

--------------------

Pups::ExecError: cd /var/www/discourse && su discourse -c 'bundle exec rake assets:precompile' failed with return #<Process::Status: pid 507 exit 1>

Location of failure: /pups/lib/pups/exec_command.rb:112:in `spawn'

exec failed with the params {"cd"=>"$home", "hook"=>"assets_precompile", "cmd"=>["su discourse -c 'bundle exec rake assets:precompile'"]}

fcf17d793c27c4e87616420ead222cc3f4a9fc163f239a5542b1c9a092579b30

** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.

./discourse-doctor may help diagnose the problem.

Really hope somebody can shed some light on this.

Thanks all.

6 Likes

Can you please add this to the env section of your app.yml file:

DISCOURSE_REFRESH_MAXMIND_DB_DURING_PRECOMPILE_DAYS: 30

This is a temporary fix while we are working on the underlying issue.

10 Likes

Hmm. Looking quickly at MaxMind, this looks like a real problem. @Roman_Rizzi and @david made recent MaxMind commits.

Looks like Falco is on this. I’ll stop now.

From: Significant Changes to Accessing and Using GeoLite2 Databases | MaxMind Blog

https://status.maxmind.com/

5 Likes

Thanks @Falco @pfaffman for the quick replies guys.

I’ll add the env variable and try a rebuild again

5 Likes

Quick update:

Thanks @Falco all fine now :+1:

For anyone else with this issue the syntax for app.yml is as follows:

DISCOURSE_REFRESH_MAXMIND_DB_DURING_PRECOMPILE_DAYS: 30

8 Likes

Worked for me! :rocket:

Thanks, Falco!

3 Likes

I pushed this change in core just now, so we have a few weeks to figure out what to do here.

9 Likes

What does that mean for new installs?

Can we allow a graceful failure if the download isn’t available? That would work for both cases.

1 Like

Our Docker image has a cached database. So new installs and old will work for the next weeks. And we will fix it before that cached database is too old.

8 Likes

@Falco Thanks for pointing me here from Unable to rebuild, 'bundle exec rake assets:precompile' failed.

I put it at top of app.yml but that didn’t work. I get the same error.

EDIT: oops, I needed it in the env section.

1 Like

I did this Joseph then re-read the instruction and it says add the line to the env: section. It worked for me after that.

2 Likes

Does this workaround impact the discourse site in any noticeable way for the end user?

Noticeable? No.

A very small number of users may have wrong place in their “Recently Used Devices” under https://meta.discourse.org/my/preferences/account

6 Likes

Unfortunately this workaround doesn’t work for the Bitnami discourse images - source. If anyone knows of a workaround for them, that would be great. :slight_smile:

2 Likes

You can either ask over at bitnami or follow the install cloud procedure.

5 Likes

Just to add to the comment from @pfaffman all third party images are totally unsupported here.

3 Likes

Reading the MaxMind announcement, it sounds like redistributing the database within the Docker image might have some severe legal implications as well. Might be worth looking into.

3 Likes

Yeah, this is just a band-aid to get people forums back up as we try and come up with a proper solution in the next few days. Really bad timing for this change, but I know they also had little time to react…

7 Likes

I would respectfully disagree that they had little time to react. The law was passed in June of 2018. It was amended in October of 2019. That’s plenty of time to do this with more than 12 days notice.

7 Likes

I meant the fellow engineers who claimed to have just a few weeks.

10 Likes