mentalstring reports an issue running Discourse on AWS Graviton (ARM64) instances due to a missing libjemalloc.so.1 symlink, which causes repeated errors in rails.stderr.log. The issue stems from version differences in jemalloc between x86 and ARM64 builds, with ARM64 shipping libjemalloc.so.2 instead. A workaround by overriding $RUBY_ALLOCATOR in web.template.yml resolves the error. merefield links to a related Raspberry Pi issue. Falco confirms they’re actively working on ARM64 container support and plan to use a universal .so symlink compatible with both architectures. mentalstring offers to test on Graviton2 and suggests dynamically selecting jemalloc versions based on PAGESIZE to reduce divergence between architectures. Falco confirms prior Graviton testing and notes the upcoming changes are retrocompatible. Falco announces the fix is deployed. mentalstring confirms successful builds and absence of jemalloc errors on Graviton2, and plans to evaluate ARM64 for production if stability holds.
I’m experimenting running Discourse in AWS’s graviton based instances as they are attractive price wise. I understand that aarch64 support is experimental, but the build message says to report issues, so this is it. I think many are running Discourse in AWS, so hopefully this can benefit others too.
I got an existing installation up and running in arm64 after a rebuild and some basic sanity check to make sure the frontend loads. However, a ./launcher logs web_only shows that rails.stderr.log is constantly being filled with these:
ERROR: ld.so: object '/usr/lib/libjemalloc.so.1' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
ERROR: ld.so: object '/usr/lib/libjemalloc.so.1' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
And, indeed, /usr/lib/libjemalloc.so.1 doesn’t exist on arm64 like it does on the x84 images, but /usr/lib/libjemalloc.so.2 does. I traced the difference of version in arm64 to this:
Side note: kudos for the good documentation of the change.
I think the error message comes from the specific mention to the non-existent /usr/lib/libjemalloc.so.1 done here:
A quick&dirty override of $RUBY_ALLOCATOR on templates/web.template.yml seems to have successfully stopped the error message, so it appears rails (puma?) is now running with the proper libjemalloc.
Having said that, I don’t know if a proper fix may need more changes than this and if such change may have other implications for production systems. Still, wanted to share in case it affects others trying to use Discourse with arm64 in AWS. Perhaps the Discourse team can take a look too?
As the link by @merefield shows, I’m actively working on the aarch64 container image and ran out of time for it last Friday.
I think I finally covered all the places, and the next step is changing that ENV for the main symlink that ends in .so which will work on both x64 and ARM64.
Oh, wow, lucky timing for me then. I’ve been meaning to try arm64 for a long while now, but only got around doing it today, just as you have been working on that.
Happy to help test when you are ready, if you’d like the changes to be tested with a graviton2 instance.
Speaking of which, since the custom jemalloc version can break things (but still needed for some Raspberry Pi setup), would it make sense to make the version decision based on looking up the particular PAGESIZE on the machine the image is being built? The reason I suggest this is because the graviton2 instance I’m testing with has PAGESIZE=4k. A newer jemalloc version would still be needed for some cases, but it could perhaps reduce the differences between arm64 and x64 image setups.
I did test it on Graviton a couple of years ago, and it works just fine. The work I’m doing now is to also make it compatible with newer ARM platforms with non-standard PAGESIZE. The changes are all retrocompatible with old platforms, so nothing should change for those.
Can confirm that it’s building fine without local changes in a graviton2 and that there’s no jemalloc errors on the logs that I can see. Thanks for addressing this.
Will continue playing with it as we may move our production server to arm64 if it is stable enough. If I bump into something else, I’ll report it.