Full Text Searching with Solr and Sunspot

This post is part of our Exploring Solr and Sunspot series.

Full text searching can be a tricky subject. Luckily there are a number of great tools out there that are much better than doing "... content like '%cars%' ...". One excellent tool that we have used on numerous projects is Solr along with the Ruby library Sunspot.

What is Solr?

Solr is not technically a full text search engine itself, but instead is an HTTP layer wrapped around the Lucene engine. But for our purposes we can think of Solr as a search engine with a web API. The first thing you’ll notice with Solr is that it’s built in java, but don’t let that turn you off. One of my favorite things about Solr is that you can download it and start running out of the box without having to install or compile anything.

If you are courageous you can access the Solr API directly, fortunately there is a Ruby library Sunspot that will do all the heavy lifting for us.

What is Sunspot and how does it work?

When working with Rails, all you need to worry about is sunspot_rails. You won’t need to download or install anything with solr because the gem includes the solr libraries as well as rake tasks to start and stop the server.

What sunspot_rails does is hook into the ActiveRecord lifecycle callbacks and send updates to the solr server behind the scenes. Then you can perform a search which will first query the solr server then load the ActiveRecord instances for you, making the search feel transparent.

Having fun with Sunspot

Enough explaining, lets get our hands dirty. First start by adding the following gems to your bundler Gemfile:

# add this to your bundler Gemfile
gem 'sunspot_rails'

# install it by running
> bundle install

Solr runs as a separate process so you will need to start up the Solr server before you begin developing.

# to start the server
> rake sunspot:solr:start

# to stop the server
> rake sunspot:solr:stop

When you start the server you may notice files being copied into RAILS_ROOT/solr, that’s because Solr/Lucene stores the search indexes in the file system. That way if you restart the server, you’ll retain the search indexes.

Let’s create a scaffold for a Book:

> rails generate scaffold book title:string isbn:string
> rake db:migrate

We’re going to make a few small tweaks to the Book model to make it searchable and to offer a search feature. First add the following the app/models/book.rb

class Book < ActiveRecord::Base
  validates_presence_of :title, :isbn

  searchable do
    text :title
  end
end

What this does is define the fields sunspot will watch and communicate to solr. Lets also add a search action to the codebase. Add the following to config/routes.rb

resources :books do
  collection do
    get :search
  end
end

Add the following to app/controllers/books_controller.rb

# GET /books/search
# GET /books/search.xml
def search
  @books = Book.search do
    keywords params[:query]
  end.results

  respond_to do |format|
    format.html { render :action => "index" }
    format.xml  { render :xml => @books }
  end
end

And add the following to app/views/books/index.html.erb

<%= form_tag search_books_path, :method => :get do %>
<p>
  <%= text_field_tag :query, params[:query] %> <%= submit_tag "Search!" %>
</p>
<% end %>

Now add a few books and perform a search! For instance:

Searching for 'Great' and viewing results

Cool huh?

Updating indexed fields and re-indexing

Another great feature of Sunspot/Solr is changing the fields you’re indexing. In our case we have both a title and an isbn, so let’s search off of any of them. Update app/models/book.rb

class Book < ActiveRecord::Base
  validates_presence_of :title, :isbn

  searchable do
    text :title, :isbn
  end
end

And from this point on, Sunspot will include the isbn with the search indexes. But that doesn’t help with our existing data. Not to fear, sunspot has a handy rake task that will let you re-index your existing data again:

> rake sunspot:reindex # this will go through all your searchable models and re-index them with solr

Wrapping up

That was pretty slick and pretty easy eh?

You may have noticed that by default Sunspot/Solr doesn’t search within words. For instance I can search for “Great” but not “Gre”. That’s due to the quick default indexer. In a later blog post I’ll describe how to tweak the Solr config to allow you to search within words.

For those who want to play with a working codebase, I’ve uploaded a working demo onto github. There are a whole slew of fun things you can do with sunspot. I’d recommend checking out the site and the wiki for advanced usage.

Zach Moazeni

Comments:


Post a Comment

(optional)
(optional — will be included as a link.)
  1. Thanks for sharing!

    Tatum
    Tatum
    March 08, 2011 at 23:57 PM
  2. No need for SunSpot-rails anymore. Just sunspot !

    March 11, 2011 at 8:50 AM
  3. @alain What do you mean? The gems are separated with the rails specific hooks in sunspot_rails.

    https://github.com/outoftime/sunspot/wiki/adding-sunspot-search-to-rails-in-5-minutes-or-less

    You can certainly use sunspot without sunspot_rails for non-Rails projects.

    March 11, 2011 at 9:36 AM
  4. alain & Zach: it does look like sunspot_rails has been merged into the main sunspot repo, but the sunspot_rails gem still exists.

    March 19, 2011 at 14:29 PM
  5. Thanks, this is exactly what I was looking for.

    Jim
    Jim
    April 02, 2011 at 21:05 PM
  6. @zach, do you have any thoughts on searching “through” a model a-la relationships?  For example, searching through ‘Comments’ on a Post?

    As best I can tell, in the above case, some magic wizardry must be done on the Post to enable this because of the fact that Solr is document oriented.

    diebels727
    diebels727
    May 05, 2011 at 15:05 PM
  7. @diebels727 that’s a great question. I’ll put together a blog post on that instead of shoving it here in the comments. But check out the lambda syntax inside of the searchable block.

    text :body {|instance| … }

    May 05, 2011 at 16:16 PM
  8. Does anyone know how to teach solr to return results from ‘#term’?
    I need to have different set results for ‘term’ and ‘#term’

    January 23, 2012 at 15:17 PM
  9. I dont think this works ! I have tried it and i keep getting this error ActiveRecord::RecordNotFound in BooksController#show

    Peter
    Peter
    February 17, 2012 at 4:08 AM
  10. So I followed pretty much exactly (just changing variable names and such) and get the following error: undefined method `searchable’ for #

    Then, I refresh the page again and the following error occurs:
    undefined method `key?’ for nil:NilClass

    In my terminal server I get:
    Started GET “/” for 127.0.0.1 at 2012-03-17 17:48:58 -0400

    NoMethodError (undefined method `key?’ for nil:NilClass):
      actionpack (3.2.1) lib/action_controller/metal/hide_actions.rb:36:in `visible_action?’….

    Any suggestions?

    Austin
    Austin
    March 17, 2012 at 17:52 PM
  11. @Peter and @Austin – check out the codebase I pushed up to https://github.com/collectiveidea/playing-with-sunspot – I just tested it out and it still works. I also updated the readme with more detailed instructions on how to get it running.

    March 19, 2012 at 11:05 AM
  12. Dear Zach Moazeni:

    I had follow all about your examples. well done.
    it works like charm, but I need this program run with nginx .

    then I get the error message:

    Errno::ECONNREFUSED in KojenadultsController#search

    Connection refused – connect(2)

    this program is on my github, could you help me how to make it run with nginx? thx

    git://github.com/idarfan/kojenadults.git

    idarfan
    idarfan
    April 09, 2012 at 9:02 AM
  13. I can’t charge this application on my browser. I have a problem, this exception is thrown for me: NoMethodError in BooksController#index. 
    I have the exactly code that you post.. Except that I use Json intead of xml for the render.. Please Help me

    Paola
    Paola
    May 20, 2012 at 20:56 PM
  14. @Paola: you should check out the example codebase on https://github.com/collectiveidea/playing-with-sunspot that may help tell the difference.

    @idarfan: I’m not sure about solr running behind nginx. Doesn’t seem necessary.

    May 21, 2012 at 9:50 AM
  15. Je suis toujours en ligne pour enquêter conseils qui peuvent m’être utile. Merci collectiveidea.com

    June 11, 2012 at 10:18 AM
  16. Do you know how to update the index instead of just reindexing every time?

    Mel
    Mel
    August 21, 2012 at 16:07 PM
  17. @Mel: You have to pass :auto_index for searchable in your model and set it to true.

    limonka
    limonka
    August 24, 2012 at 11:39 AM
  18. Heyyy I’m building a website by RoR ..and i use Sunspot and solr Search Engine …i needed to put autocomplete functionality but i coudn’t .. tried to use the gem sunspot_autocomplete …but i got many errors …is this gem updated and i can use it normally or it’s out dated ? thanks a lot man 

    Mazen
    Mazen
    September 09, 2012 at 19:31 PM
  19. If I search pro “Grea” the result is blank, how to solve this??

    Tiago
    Tiago
    February 26, 2014 at 13:28 PM