Replying to my own post, after searching for a while I stumbled across this post where the same behavior is described:
The jittery on mobile seems to be exclusively related to the option to placed ads every nth post. Putting this to zero does in fact stop the problem, but it would be much nicer if the plugin authors could fix the issue instead.
The UI proposal is to add max size parameters to the Discourse settings UI, instead of the current selection menu where you have a handful of fixed sizes available. Would de-clutter the UI, as responsive would be the default, with the possibility to limit the height.
Are you sure you haven’t tried using CSS
I suck at CSS, but to my knowledge it has been tried a number of times. Currently working around the issue by using a fixed size for mobile layout.