OK, i just created an account because i had the same problem and found the solution!
Like Bertrand said; you can set the reply_to_post_number all day long but in the UI it is not shown as a regular reply. When i tried to create a reply however in the same way but with my admin account it DID work! This indicated it had something to do with permissions.
I found out that if i changed the Security settings of the Category the post belonged to, so that the default Everyone group had all rights (so also Create rights which was unchecked by default for me) it now WORKS! Not sure why this right is needed as Create already works (i can create a post), and the Reply right was already checked but anyway… it WORKS 