Large-scale Search

Document Length and Relevance Ranking

In Part 1, we made the argument that the one to two orders of magnitude of difference in document length between HathiTrust books and the documents used in standard test collections affects all aspects of relevance ranking. 

When we first started working on large scale search we confronted the issue of whether to index pages or complete books as our fundamental unit of indexing.[i]   We had some concerns about indexing on the page level.  We knew we would need to scale to 10-20 million books and at an average of 300 pages per book that comes out to about 6 billion pages.  At that time we did not think that Solr would scale to 6 billion pages.[ii]  If we indexed by page, we also wanted to be able

Forty days forty nights: Re-indexing 7+ million books (part 1)

Forty days and forty nights; That’s how long we estimated it would take to re-index all 7+ million volumes in HathiTrust. Because of this forty day turnaround time, when we found a problem with our current indexing, we were reluctant to do a complete re-index. Whenever feasible we would just re-index the affected materials.

We just released a new feature in our full-text Large Scale Search. When you do a search,you will see check boxes next to each search result. You can select items you want from the search results and create a personal collection. This should make it much easier to do repeated searches and explore a targeted subset of the HathiTrust volumes. If you are not logged in, the collection will be temporary. If you log in you can save the collection permanently.

On November 19, 2009, we put new hardware into production to provide full-text searching against about 4.6 million volumes.  Currently we have about 5.3 million volumes.  The average response time is about  3 seconds,  90% of queries take under 4 seconds, 9% of queries take between 4 seconds and 24 seconds, and 1% of queries take longer than 24 seconds.

This is the list of 400 (actually 415) common words used for our current Solr configuration as described in Tuning Search Performance.

Before we implemented the CommonGrams Index, our slowest query with the standard index was “the lives and literature of the beat generation” which took about 2 minutes  for the 500,000 volume index.  When we implemented the CommonGrams index, that query took only 3.6 seconds. 

This is a brief note on the  current hardware and software environment we are using for Solr testing.

Solr Servers

• Two Dell PowerEdge 1950 blades
• 2 x Dual Core Intel Xeon 3.0 GHz 5160 Processors
• 8GB - 32GB RAM depending on the test configuration
• Red Hat Enterprise Linux 5.3 (kernel: 2.6.18 PAE)
• Java(TM) SE Runtime Environment (build: 1.6.0_11-b03)
• Solr 1.3
• Tomcat 5.5.26

Storage Server

Since we finished the work described in the Large Scale Search Report we have made some changes to our test protocol and upgraded our Solr implementions to Solr 1.3. We  have completed some testing with increased memory and some preliminary load testing.

The new test protocol has these features