This is the “expected” behavior of a HTML content editable - inside and outside an inline node is considered a single position.
The inline code is a great example of an alternative: it shows a fake cursor at that single position, so when you arrow navigate, it’s the same position with different marks, and slightly shifted visually.
I believe an ideal state will be supporting that same behavior for other types of content.