Full Text Searching with Solr and Sunspot

This post is part of our Exploring Solr and Sunspot series.

Full text searching can be a tricky subject. Luckily there are a number of great tools out there that are much better than doing "... content like '%cars%' ...". One excellent tool that we have used on numerous projects is Solr along with the Ruby library Sunspot.

What is Solr?

Solr is not technically a full text search engine itself, but instead is an HTTP layer wrapped around the Lucene engine. But for our purposes we can think of Solr as a search engine with a web API. The first thing you’ll notice with Solr is that it’s built in java, but don’t let that turn you off. One of my favorite things about Solr is that you can download it and start running out of the box without having to install or compile anything.

If you are courageous you can access the Solr API directly, fortunately there is a Ruby library Sunspot that will do all the heavy lifting for us.

What is Sunspot and how does it work?

When working with Rails, all you need to worry about is sunspot_rails. You won’t need to download or install anything with solr because the gem includes the solr libraries as well as rake tasks to start and stop the server.

What sunspot_rails does is hook into the ActiveRecord lifecycle callbacks and send updates to the solr server behind the scenes. Then you can perform a search which will first query the solr server then load the ActiveRecord instances for you, making the search feel transparent.

Having fun with Sunspot

Enough explaining, lets get our hands dirty. First start by adding the following gems to your bundler Gemfile:

# add this to your bundler Gemfile
gem 'sunspot_rails'

# install it by running
> bundle install

Solr runs as a separate process so you will need to start up the Solr server before you begin developing.

# to start the server
> rake sunspot:solr:start

# to stop the server
> rake sunspot:solr:stop

When you start the server you may notice files being copied into RAILS_ROOT/solr, that’s because Solr/Lucene stores the search indexes in the file system. That way if you restart the server, you’ll retain the search indexes.

Let’s create a scaffold for a Book:

> rails generate scaffold book title:string isbn:string
> rake db:migrate

We’re going to make a few small tweaks to the Book model to make it searchable and to offer a search feature. First add the following the app/models/book.rb

class Book < ActiveRecord::Base
  validates_presence_of :title, :isbn

  searchable do
    text :title
  end
end

What this does is define the fields sunspot will watch and communicate to solr. Lets also add a search action to the codebase. Add the following to config/routes.rb

resources :books do
  collection do
    get :search
  end
end

Add the following to app/controllers/books_controller.rb

# GET /books/search
# GET /books/search.xml
def search
  @books = Book.search do
    keywords params[:query]
  end.results

  respond_to do |format|
    format.html { render :action => "index" }
    format.xml  { render :xml => @books }
  end
end

And add the following to app/views/books/index.html.erb

<%= form_tag search_books_path, :method => :get do %>
<p>
  <%= text_field_tag :query, params[:query] %> <%= submit_tag "Search!" %>
</p>
<% end %>

Now add a few books and perform a search! For instance:

Cool huh?

Updating indexed fields and re-indexing

Another great feature of Sunspot/Solr is changing the fields you’re indexing. In our case we have both a title and an isbn, so let’s search off of any of them. Update app/models/book.rb

class Book < ActiveRecord::Base
  validates_presence_of :title, :isbn

  searchable do
    text :title, :isbn
  end
end

And from this point on, Sunspot will include the isbn with the search indexes. But that doesn’t help with our existing data. Not to fear, sunspot has a handy rake task that will let you re-index your existing data again:

> rake sunspot:reindex # this will go through all your searchable models and re-index them with solr

Wrapping up

That was pretty slick and pretty easy eh?

You may have noticed that by default Sunspot/Solr doesn’t search within words. For instance I can search for “Great” but not “Gre”. That’s due to the quick default indexer. In a later blog post I’ll describe how to tweak the Solr config to allow you to search within words.

For those who want to play with a working codebase, I’ve uploaded a working demo onto github. There are a whole slew of fun things you can do with sunspot. I’d recommend checking out the site and the wiki for advanced usage.