Apache Solr 2014: A Year In Review

The last few years have seen Apache Solr move at a reasonably aggressive pace. 2013 marked the debut of SolrCloud: a newly architected, truly distributed mode of Solr. 2014 saw Solr, specifically SolrCloud, mature in terms of greater ease of use, more predictable scalability and more reliability in production environments with each new release.

Continue reading the post on the official Lucidworks blog here:

http://lucidworks.com/blog/solr-2014-year-review/

Third Bangalore Apache Solr/Lucene Meetup

We just had the third Bangalore Apache Solr/Lucene meetup this last weekend. It's good fun to see the community grow to 200+. Actually, as I type this, we're already 3 short of the 250 mark.

As per the requests from a reasonable number of members, we had a "Solr 101" by Varun Thacker form Unbxd. Right before he leaves for his talk at Lucene Revolution Europe 2013, he did get a lot of attendees both introduced and interested in Solr.

His talk was followed up by a talk on a "DIY Bookmarks manager using Lucene" by Umar Shah from SimplyPhi, Bangalore. It was a nice demo and I'm sure this would have again motivated people to try out similar DIY stuff using either Lucene/Solr.

This was followed with a talk by Ganesh M from Dell Sonicwall. Ganesh gave a quick talk about "Building Custom Analyzers in Lucene". A short and quick take on a complex but interesting part of Lucene certainly got the advanced users interested.

After his talk, with the last talk, I spoke about "MoreLikeThis in Solr/Lucene". I started off with some history on the current MLT stuff and how it really works along with what are the kind of issues that people are known to run into when using this feature. I also spoke about a MoreLikeThis QueryParser for Solr that I've been working on as a part of my official work at LucidWorks. We plan to Open Source it as soon as I have time to put it out and document it a bit.

This may well be the last meetup I'd organize and attend in Bangalore for the year. Good luck to Varun and Shalin for organizing this going forward.

Thanks to Microsoft Accelerator for giving us the venue to host the meetup yet again. It's one of the most centrally located and well equipped spaces in Bangalore for such meetups.

In case you'd like to join the meetup group and be a part of the active community, here's where to do that: http://www.meetup.com/Bangalore-Apache-Solr-Lucene-Group/ .

Photographs from Bangalore Apache Solr/Lucene Meetup#2

Bangalore Apache Solr/Lucene Meetup, a set on Flickr.

Photographs from the 2nd meetup we organized last Saturday.

Second Bangalore Apache Solr/Lucene Meetup

After having a rather good first meetup, I, along with Shalin Mangar and Varun Thacker organised the second one for the Bangalore Apache Solr/Lucene community on the 27th July, 2013. The meetup group has gone up from 85 odd members, around the time of the first meetup, a month and a half ago, to almost 200.

Though we weren't able to get the same venue as last time i.e. Microsoft Accelerator, due to availability issues, we'd like to thank Flipkart for providing us with more than what we bargained for. From a nice space to host the event to constant flow of breakfast and coffee.

Talking about the event, the audience, like last time was a mixed bag. Though there were more than a few people who were completely new to Solr and actually very few who were advanced Solr users, it was good to have as many people take interest in the Apache Lucene/Solr project.

Shalin Shekhar Mangar from LucidWorks, started the talks for the day with a short talk on 'Introduction to Solr' that enabled newbies in the audience to relate to most of the things that were spoken through the rest of the day. He also  demoed people a working Solr instance, the admin interface and the document upload from UI (new in Solr 4.4).

This was followed by a talk by Venkatesh Prasanna from Infosys Technologies. He spoke about 'Knowledge Management at Infosys' and how they use Solr and Nutch within Infosys.

Flipkart didn't just give us a venue but also a talk about what happens in Flipkart Search, along with some insight on e-commerce search challenges in general. Umesh Prasad and Thejus VM spoke about their architecture and why the vector space model doesn't work for them.

The last talk of the day by Jaideep Dhok from Inmobi kept people interested until we called it a day. Talking about "DIY Percolator using Lucene", he highlighted possibilities, use cases and also demoed his code.

All in all, it was a productive day and I assume everyone found it worth waking up early, on a Saturday morning.

In case you're interested in attending these events, feel free to join the meetup group [here]. Also, if you would want to present at the next meetup, kindly drop me a line.

P.S: The presentations from the meetup should up soon. Also, we are figuring out if the video recording (specially audio) quality was good enough to be uploaded and shared. If we end up posting those, I'll share the links at the meetup page.

Shard Splitting in SolrCloud

It's been around two years since my last post and after this hiatus, I intend to be more regular at it.
I've been a part of a couple of organizations since then. After being a part of the small team that launched the first ever AWS Search Service, CloudSearch, I am back to the open source world.

I joined LucidWorks in Jan '13, the primary company backing and supporting the Apache Solr project. The last few months have been spent trying to absorb the changes that the Apache Lucene/Solr project went through over an year and a half, when I was busy with A9. I've also actively worked and contributed on the Shard Splitting feature (SOLR-3755) for SolrCloud, with Shalin Shekhar Mangar since then.

It certainly feels good to be back with this stuff and to begin with, here's my post on Shard Splitting in SolrCloud : http://searchhub.org/2013/06/19/shard-splitting-in-solrcloud/.

Lucene Vs Sphinx - A Showdown on a large dataset

There has long existed a battle among the "pro java" and "pro c" developer community. Not that I'd want to strictly be associated with one, but would always say the robustness/exception handling/stability vector have a better cosine value with java as compared to c [With all respect to C, for being the progenitor for Java]. Let me not move off on the tangent and rather more towards the core here. Last few weeks were spent trying to benchmark 'java' lucene as a search engine with 'C' sphinx. Though not exactly in their vanilla form, and a lot of modifications to both, we finally ran a lot of tests on both the engines.

Keeping a common playground with the following specifications:
Processor(s) : Intel Quad Core X 2
RAM : 24G
Operating System : RHEL 32 Bit
Document Corpus : 18 million+ Documents
Source Size : 90G [RDBMS Table]


On a corpus of approximately 18 million records, indexed and not all of them stored. Multiple field queries with varying boost values and some good level of complexity. Here is the result sheet:

Lucene
Index Size : 20G

Concurrency : 30 [5*6 Daemons, 2G each]
Total Searches : 64931
Slow Query Count (>=10 secs) : 3803 ( 5.86%)
Total Duration (secs) : 238094.574
Mean Duration : 3.667
Mode Duration : 0.835
Minimum Duration : 0.001
Maximum Duration : 1174.757
Duration Standard Deviation : 15.441
Search Time (secs) Distribution :-
[0,0.25) : 2770 ( 4.27%)
[0.25,0.5) : 5666 ( 8.73%)
[0.5,1) : 13515 (20.81%)
[1,1.5) : 10928 (16.83%)
[1.5,2) : 7330 (11.29%)
[2,3) : 8476 (13.05%)
[3,5) : 7222 (11.12%)
[5,10) : 5221 ( 8.04%)
[10,20) : 2335 ( 3.60%)
[20,+inf) : 1468 ( 2.26%)

Concurrency : 5 [1 Daemon * 2G]
Total Searches : 225906
Slow Query Count (>=10 secs) : 972 ( 0.43%)
Total Duration (secs) : 186700.646
Mean Duration : 0.826
Mode Duration : 0.003
Minimum Duration : 0.001
Maximum Duration : 467.647
Duration Standard Deviation : 2.864
Duration (secs) Bins :-
[0,0.25) : 64621 (28.61%)
[0.25,0.5) : 58947 (26.09%)
[0.5,1) : 56894 (25.18%)
[1,1.5) : 19836 ( 8.78%)
[1.5,2) : 9397 ( 4.16%)
[2,3) : 7941 ( 3.52%)
[3,5) : 4810 ( 2.13%)
[5,10) : 2488 ( 1.10%)
[10,20) : 684 ( 0.30%)
[20,+inf) : 288 ( 0.13%)



Sphinx
Index Size: 60G

Concurrency: 30
Total Searches : 244431
Slow Query Count (>=10 secs) : 27479 (11.24%)
Total Duration (secs) : 1243474.213
Mean Duration : 5.087
Mode Duration : 0.007
Minimum Duration : 0.001
Maximum Duration : 1869.063
Duration Standard Deviation : 17.833
Average Queries : 2.783
Duration (secs) Bins :-
[0,0.25) : 51186 (20.94%)
[0.25,0.5) : 27798 (11.37%)
[0.5,1) : 32372 (13.24%)
[1,1.5) : 20490 ( 8.38%)
[1.5,2) : 16915 ( 6.92%)
[2,3) : 21833 ( 8.93%)
[3,5) : 23550 ( 9.63%)
[5,10) : 22808 ( 9.33%)
[10,20) : 14540 ( 5.95%)
[20,+inf) : 12939 ( 5.29%)

Concurrency: 5
Total Searches : 226528
Slow Query Count (>=10 secs) : 9895 ( 4.37%)
Total Duration (secs) : 453296.517
Mean Duration : 2.001
Mode Duration : 0.007
Minimum Duration : 0.001
Maximum Duration : 164.713
Duration Standard Deviation : 4.543
Average Queries : 2.773
Duration (secs) Bins :-
[0,0.25) : 71001 (31.34%)
[0.25,0.5) : 36500 (16.11%)
[0.5,1) : 32799 (14.48%)
[1,1.5) : 20416 ( 9.01%)
[1.5,2) : 16385 ( 7.23%)
[2,3) : 13951 ( 6.16%)
[3,5) : 13330 ( 5.88%)
[5,10) : 12251 ( 5.41%)
[10,20) : 7563 ( 3.34%)
[20,+inf) : 2332 ( 1.03%)

As per the analysis, for the dataset analyzed, Lucene was found to win convincingly over its rival.
More details on the same to come soon!

P.S: Though lucene works great for a lot of cases, so does sphinx. Here, lucene seemed to have an upper hand