I believe the correct fix here is to not require capitalized group name in URL params. Done via:
https://github.com/discourse/discourse/commit/af6d0342b642feac3a5e359d13c854d2a80145b6
Note that browsing to /u/username/messages/group/groupname will fire an AJAX request to /topics/private-messages-group/username/groupname, so what @angus did above to debug this issue does look correct.
Thanks for reporting this issue @RGJ 