Still thinking about this - whatever it is will probably need to take the specific posts locator into account when considering anything.
Perhaps that existing rectangle with the post number in it could remain in that spot even after being pressed so that people can still type a post number in, but in between them in that space somewhere there could be a simple up/down indicator.
Now, it’s ugly and dodgy, but something like this where the post/date text moves slightly right so that lefties can use the existing scroll structure and righties can use another floating object (arrow/line/no idea) and both can see the number they’re scrolling to.