Available Indexes
About › News and Publications › Blogs › Large-scale Search › Current Hardware Used for Testing

Current Hardware Used for Testing

Submitted by Tom Burton-West on July 24, 2009

This is a brief note on the  current hardware and software environment we are using for Solr testing.

Solr Servers

  • Two Dell PowerEdge 1950 blades
  • 2 x Dual Core Intel Xeon 3.0 GHz 5160 Processors
  • 8GB - 32GB RAM depending on the test configuration
  • Red Hat Enterprise Linux 5.3 (kernel: 2.6.18 PAE)
  • Java(TM) SE Runtime Environment (build: 1.6.0_11-b03)
  • Solr 1.3
  • Tomcat 5.5.26

Storage Server

  • Dell PowerEdge 1650
  • 2 x Intel Pentium III 1.2GHz Processors
  • 4GB RAM
  • Fedora Core (kernel: 2.6.17)
  • 2 x ACNC JetStor 416S SCSI/SATA hardware RAID 6 (1 3TB, 14 x 250GB SATA; 1 6TB, 14 x 500GB SATA)
  • LVM2 volume striping across RAIDs

Network

  • NFS over a dedicated/private GbE network

We will be installing new hardware for the production system soon and will be able to run  a few tests before we switch it into production.

Comments

Submitted by Anonymous (not verified) on January 9, 2010
How did you set up the 2 shards 2 machine 16GB test? I assume you have split your index into two shards with one shard, one Solr server, and 16G of RAM per machine. Then you executed distributed queries similar to http://localhost:8983/solr/select?shards=localhost:8983/solr,localhost:7... see http://wiki.apache.org/solr/DistributedSearch Is that correct?

Submitted by David R (not verified) on February 23, 2010
That`s a lot of power on your machine configuration for use at Solr testing...Pariuri SportiveRezultate LiveClasamente Fotbal Statistici FotbalBiletul Zilei

Submitted by Christal (not verified) on March 25, 2010
This does seem like overkill for what you want to achieve, doesn't it?

Submitted by Tom Burton-West on March 25, 2010

If you look at the testing results, you will see that indeed this is not a lot of hardware for the size of the indexes we need. With the testing hardware we were not able to sustain a throughput of 2 queries per second and prior to the implementation of CommonGrams we had queries that took 2 minutes.

http://www.hathitrust.org/blogs/large-scale-search/update-testing-memory-and-load-tests

For comparison, take a look at our current production hardware, which is significantly more powerful than what we used in testing.
In particular we are using more RAM and significantly faster storage.
http://www.hathitrust.org/blogs/large-scale-search/new-hardware-searching-5-million-volumes-full-text

Tom

Submitted by rodhes (not verified) on January 28, 2011
I assume you have split your index into two shards with one shard, one Solr server.

Submitted by Tom Burton-West on January 28, 2011

Yes, for our early tests with two shards we put one shard on each host under its own tomcat/Solr instance. Currently we are testing with 3 shards on one host under one solr server to simulate our production environment which currently has 4 machines with each machine running one solr instance under tomcat with 3 shards.

Tom

Add new comment

You are browsing an archive of the HathiTrust website. In July 2023, we launched a new site at www.hathitrust.org.