Sphinx search for Discourse


(Katie Hunter) #1

Hi, i am not aware if whether or not Discourse uses sphinx for its search function http://sphinxsearch.com but it is very powerful and light. I know on my forum which uses Vbulletin, i had to integrate the sphynix search which works really well than the Mysql default search which comes with VB =) b/c we have around 5 million posts and 1.7+ million members. Vb Mysql default search was not the right approach for big to growing communities.

Does Discourse uses it for their search function? If not, i would really recommend it and if there is any plans for it?


(rafalp) #2

AFAIK Discourse uses PostgreSQL’s superawesome text search features:


(Katie Hunter) #3

Hi, why not Sphinx search ? I am not familiar with PostgreSQL but would it not face issues with big communities and high activity? If so, then i recommend Sphinx, it is light and doesn’t add stress on the server. It is great actually =)

Is there any future plan for it? allowing two options may be ? One that can use the Sphinx and another to use PostgreSQL. I am not sure if Sphinx works with Ruby. I am not familiar with this part.

I only found this GitHub - pat/thinking-sphinx: Sphinx plugin for ActiveRecord/Rails


(rafalp) #4

I understand that for somebody coming from MySQL world notion that you can use database search facilities exclusively may sound… crazy, but thats not fair “standard” to judge other database engines fulltext search features.


(Katie Hunter) #5

Well, like i said, i am not familiar with it =) i don’t know if it will perform well under big communities. I would like to hear @codinghorror opinion on this and whether there is a future plan for it. It doesn’t hurt to have a setting in the admin page to choose the search type that you like to use.

VB has that function too, it provides you with the type of search in a drop down menu that you would like to use but they don’t support Sphinx, i had to look around and find a 3rd party plugin which gets installed and importing the plugin to work with vb. But Vb5 are supporting it though but it is currently in beta and not that great yet.


(Kane York) #6

People have been complaining that the search isn’t quite up to snuff recently, maybe this should be looked into?

Here’s some stuff I’ve found so far:

http://sphinxsearch.com/wiki/doku.php?id=sphinx_docs
https://rubygems.org/gems/sphinx


(Sam Saffron) #7

The immediate issue is not search quality.

The amount of space we use to display search results is just too tight, we need a full page “topic list” like search results page.

After that is done and if that fails we can discuss alternative approaches.


(DAB Lucide) #8

Just throwing this out there for some fun, but what if the search results were displayed in a way that would allow the user to still navigate around but having full search results displayed? Quick example:

Basically, the search bar and results pop out on a side panel that can be opened and closed at any time while maintaining the search results while the user continues to navigate discourse. Similar to how users can navigate while still having their reply/topic creation panel open. The user could also scroll up and down this panel, when they move their mouse within the panel, to view more of the listed results. I also like this idea because it keeps the results from covering up anything else on the page. Just thought it’d be fun :stuck_out_tongue: Not sure about it conflicting with the reply/topic creation panel though. :o

Noticed how discourse looks unbalanced in my example. Maybe throw something on the left side as well?


(Bill Ayakatubby) #9

I see the issue as being the lack of excerpts, not the results pane being too small.


(Katie Hunter) #10

This looks like a better idea and style wise could be integrated well =)


(Jeff Atwood) #11

You don’t think “lack of excerpts” and “results pane being small” are related? Maybe a little?


(Katie Hunter) #12

I still encourage giving the option to use Sphinx Search.

Vbulletin 5.x even though in beta, their new software is using Sphinx, VB 4.x didn’t have it and the quality of search wasn’t fast nor great. Many had to use a 3rd party plugin to use it, it is also very light on servers. Vb learned its lesson though, because of the many complains coming from big forums with thousands of topics.


(Michael - DiscourseHosting.com) #13

What problem is this going to solve exactly? Search is fast enough on Discourse.


(Katie Hunter) #14

Well, we haven’t tried to use Discourse on big active boards. But I am a believer in using what is best out there instead of using what is just working nicely. If something is popular and known to be the best out there, then I would always use it.


(Neil Lalonde) #15

Adding a dependency on Sphinx doesn’t seem necessary to me. Our complaint-driven development process hasn’t found that postgresql’s search is lacking. Sphinx is nice, but using it and maintaining it has a cost (search config, managing sphinx processes, re-indexing after deploys, etc.).


(Jens) #16

That is exactly what we’ve started with our search results as a route plugin

We’ve also added categories as facets on the left:

There’s an extra link top right in todays search results to go to the serp:

It works so far incl sorting, but UI is still pretty rough.
Any thoughts?

Jens


(Katie Hunter) #17

I use a Sphinx plugin for vb 4.x and it wasn’t complicated to install server side, then install the plugin and run re-index. Then after that, everything ran on its own. I haven’t tested Vb 5.x though which comes already with it.

It might not be urgent to use it atm but it is a very nice and essential feature in my opinion for Discourse. Many webmasters love to use Sphinx search because they know how powerfull and light it is when it comes to indexing millions of data and searching.


(Jens) #18

Others love Elasticsearch, or Solr, or Google Sitesearch - plugin territory, I guess.


(Neil Lalonde) #19

But why? How is postgresql’s search failing us?


(Michael - DiscourseHosting.com) #20

You can’t compare MySQL based boards where search is slow and painful to Discourse. One of the reasons it uses Postgresql is to work around those limitations.