Well I did manage to do it.
I’m not sure why I encountered such blocking issues… There is still a bit of jumping but it’s just a few pixels so don’t see it if you’re not trying to see it by scrolling extrêmely slowly.
I also have custom content on top of the header so it may have played a role in the issue I encountered.