With the need for such fine grained access, you might need to consider if a regular public topic is the best solution. Maybe you can tell a bit more about your specific use case?
A good starting point would indeed be my private-replies plugin. I has a function called can_see_post_if_author_among which currently takes a topic. You should modify it to take a post. You should store the allowed users with the post in a custom field and then check the viewing user against that list.
The plugin also overrides the access control logic for raw posts, search, topic views and user profile activity.
That said, I expect you to run into quite some issues from both a performance and a UX point of view and especially if you have no experience with Discourse plugin development, this can be quite an undertaking, so I would again recommend you to consider if this is the best approach for your use case.