I don’t have any direct experience to offer but I would strongly advise against coercing docker into allowing the overlay2 driver to be used on top of btrfs. It’s explicitly unsupported and the overlay2 driver could be making assumptions about the filesystem that may or may not be true for btrfs, plausibly leading to various unexpected results. You really don’t want unexpected results at filesystem level.
Low level filesystem operations will be taken care of by docker and the btrfs storage driver. If that’s a mature and well maintained combination, which is not known-buggy as devicemapper was, there’s a good chance it will just work fine.
The reason btrfs isn’t supported in Discourse, as I understand it, isn’t because of some expectation that it will fail, it’s simply because they don’t have enough information to tell people that it will work.
They don’t have any (or enough) incentive to run their own servers on btrfs so the only way for them to get that information is from people in the community trying it out in production and reporting back after extended periods of time. That hasn’t happened yet, so it remains unsupported.
If you find yourself in this situation in future, you can always reset the change, update, then apply the change again before running launcher:
cd /var/discourse
git reset --hard
git pull
sed -i 's/Storage Driver: (/Storage Driver: (btrfs|/' launcher
./launcher rebuild app