Just came across this while I was about to create a new post on the subject. (Good news, I did find this topic using the search feature as is).
But I think that @geek 's idea is still worth considering doing as a first step @sam . Better algorithms would be great, but I think you’ll want another UI regardless in the end. After the first “show more” click, if you could search on an infinitely scrolling page using the same UI as all the topic list pages, you’d have a lot more information to tell if the topic is what you’re looking for before you click it. (first post date, last post date, number of posts, etc).
Plus, while I see the value in the current design, where you can search without changing contexts, being able to keep a tab open of your search results while you explore them is helpful and natural to everyone who has been trained over the years to use a search engine like Gooooogle.
Really think its worth considering doing the search results topic list page as step 1, and then algorithmic improvements as an ongoing project (it can always get better, just ask Google)