This was one of the first changes I made with my theme.
Completely agree with shifting it below topic, it reduces clutter and “sort by category” really makes very little sense even though Josh sometimes uses it 
… I am definitely not against ninja ?order=tag and ?order=assigned and ?order=category but it is totally in the ninja department.