You can try to use multiple container files instead of a single standalone app.yml.
But let’s be honest, almost no one needs to rebuild containers frequently, right?
If you need to install plugins, it is not necessary to rebuild the container. I have unofficial ways to install plugins without rebuilding.