Indexing MongoDB Data for Solr

Solr is the popular, blazing fast open source enterprise search platform from the Apache Lucene project.

MongoDB (from "humongous") is a scalable, high-performance, open source, document-oriented data store. 

I was happy using MongoDB and my very own search engine written using/extending lucene, until the trunks for Solr and Lucene were merged. This merge translated to Solr using the same release of lucene that I was using, unlike the past when there was some disconnect between the two. I realized that a lot of what I was trying to build was available through Solr.

Though Solr is used by a lot of organizations (which can be found here) and I'm sure that at least a few of them using Mongo, for some reason there was/is no straight forward out of the box import handler for data stored in MongoDB.

This made me search for a framework/module/plug to do the same, but in vain.

All said and done, here's a way that I finally was able to index my mongodb data into Solr.

I've used SolrJ to access my Solr instance and a mongo connector to connect to Mongo. Having written my own sweet layer that has access to both the elements of the app, I have been able to inject data as required.

--snip--

public SolrServer getSolrServer(String solrHost, String solrPort) throws MalformedURLException {
    String urlString = "http://"+solrHost+":"+solrPort+"/solr";
    return new CommonsHttpSolrServer(urlString);

}

--/snip--

Fire the mongo query, iterate and add to the index

--snip--

SolrServer server = getSolrServer(..); //Get a server instance

DBCursor curr = ..; //Fire query @ mongo, get the cursor

while (curr.hasNext()) { //iterate over the result set
        BasicDBObject record = (BasicDBObject) curr.next();
        //Do some magic, get a document bean       

        server.addBean(doc);

}

server.commit();

--/snip--

This will get you started on your track to index mongo data into a running Solr instance.

Also, remember to configure Solr correctly for this to run smooth.

Download Resources:

• Mongo Drivers
• SolrJ

Searching off the RAM

Search engines are a lot about precision, recall and speed. These three factors pretty much define the quality of a search engine. I'd only talk about the last point here, speed. The time taken to search for a search engine is such a critical factor that an improvement of a few hundred milliseconds is of extreme importance to anyone associated with developing/designing search engines.
More often than not, as a short term gain, all of us look at putting in more money on the hardware to improve a system's performance. Though this might look like a solution, its bound to fail if you try to run away from actually fixing the application architecture, which happens to be the root cause for poorly performing applications generally.
For those who have already done whatever it takes to optimize the search process, here are a few ways that are generally used to host the search index on the RAM, in order to improve the search speed.
You may mount a tmpfs/ramfs on your machine and copy the index on it. You may then open index reader/searcher on this copy of the index. This would help in reducing the I/O latency and improve the search speed.
The difference between using tmpfs vs ramfs are:

• ramfs will grow dynamically unlike tmpfs which is static and initialized with a fixed size.
• tmpfs uses swap memory whereas ramfs doesn't.

I have personally used tmpfs and it works rather efficiently.
One thing to remember is that both tmpfs and ramfs are volatile. They both get erased on a system restart and hence you'd need to re-mount and copy the index on system startup.

Mounting tmpfs: 
mkdir -p /mnt/tmp 
mount -t tmpfs -o size=20m tmpfs /mnt/tmp

Mounting ramfs:
mkdir -p /mnt/ram 
mount -t ramfs -o size=20m ramfs /mnt/ram

The other approach in case you're using lucene is to have the IndexReader use a Ram Directory. In this case you'd need a JVM instance with enough memory to load all of the index contents into memory and have more for processing search queries.
Also, that may translate to 'requiring'a 64-bit JVM so that it could use pseudo unlimited address space.

--Snip--

NIOFSDirectory nDir = new NIOFSDirectory(new File(indexDir));
RAMDirectory directory = new RAMDirectory(nDir);
IndexReader ir = IndexReader.open(directory);
IndexSearcher indexSearcher = new IndexSearcher(ir);

--Snip--

I haven't been really fond of using this approach as it forces a 64-bit architecture to be used but it may surely work as there's no overhead of using and manipulating the data manually as in the case of maintaining a tmpfs. The cleanups etc...

These are the basic 2 techniques to be used if you want your index to be fed off the RAM. It is a frequent question on the lucene users mailing list, so perhaps people can now stop asking that question... well.. almost...
All said and done.. don't stop optimizing the engine/app if your search is slow.. 99% of the times.. that is where it has to be handled.

Dealing with High Dimensional Data

I believe that data is best represented as a vector. For those who haven't heard this before, well let me start with a very basic example of 'how' this is done.

Lets assume a corpus of documents which have only 2 unique words(A,B) in its dictionary (If that was hard to follow, comment and I shall follow up with what that means). Now a document containing only 'A' is a unit vector along the direction of 'A' and so with a document containing only a single occurrence of 'B'. Documents with 'x' As and 'y' Bs can hence be represented as :

x a + y b (a and b are unit vectors along A and B).

When the corpus comprises of documents wherein there are a lot of terms with a very low document frequency, it is referred to as high dimensional data. An example would be a list of proper nouns e.g. hotel names.

High dimensional data, poses a lot of issues primarily due to its sparseness in the vector space. The sparseness of data makes a lot of tasks like clustering and tagging challenging. In order to process this data, more often than not, there is a need for reducing the dimension of the documents (sparseness). I'll discuss a relatively easy way to reduce the dimension of such data.

Given a corpus of high dimensional data, create document vectors for each of them. Create a term frequency matrix for the corpus and follow it up with dropping off all terms that occur in less than 10% (might vary as per the corpus/dataset) documents. Statistically this should remove around 60% of the documents

Also, removing the terms that occur in more than 80% of the documents would lead to removing a considerable ratio of terms that are redundant and too frequent. Such terms are generally tagged as stop-words and removed under all normal data/text processing algorithms.

The residue that remains now is of a considerably reduced dimension. This is a straightforward way of projecting the original data on a multi dimensional plane. A plane comprising of all dimensions that were reduced.

This data can now be consumed for any processing viz. clustering, classification etc..

Posts on how to cluster and various clustering techniques would soon follow.. unlike this one which took ages!

Google Intelligence!

Google seems to be the definitive word when it comes to household usage for the term, intelligent! Detecting the intent and context has always been a tough nut to crack but it seems like the search major has its own way to handle it.

So when you type in 'facebook is' or 'twitter is' @ Google's search box, this is exactly the kind of suggestions it'd throw at you. Try it out at yourself.

Source for idea/thought: Cleartrip official blog

Lucid Gaze - Tough Nut!

A One Stop Solution to a lot of statistical analysis for lucene internals is the all new lucid gaze for Lucene. Perhaps it has been around for a while for Solar and I'm left to wonder.. dude...where's the documentation? There aren't many places on the information superhighway where I could spot info on how to use lucid gaze. A Google for the same would prove my point.

After I did figure out how to, it seemed like a good tool, easy to use(after the eureka moment, at least for me). Here's how I analyzed various things using Lucid Gaze by Lucid Imagination.

Pre-requisites:

• lucene core jar from Lucid imagination [Here]

Write the indexing/search logic. Open the Reader/Writer/Searcher as usual.

Create a RamUsageEstimator:

• RamUsageEstimator estimator = new RamUsageEstimator();

At point where you'd want to analyze, do a

• estimator.estimateRamUsage(ir);

Where ir is an IndexReader/IndexWriter/IndexSearcher.

__Snip__

Stats s;

s = LuceneCore.getIndexStats(); //For getting IndexStats

s = LuceneCore.getStoreStats(); //For getting StoreStats

s = LuceneCore.getSearchStats(); //For getting SearchStats

s = LuceneCore.getAnalysisStats(); //For getting AnalysisStats

__Snip Ends__

Once the above step is done, s is populated with a Stats Object containing the Index/Store/Search/Analysis Stats (as per the function call).

__Snip__

HashMap h = (HashMap) s.getCurrentCounters(); // Retrieve counters accumulated since last #resetStats().

__Snip Ends__

This HashMap is populated with current stat counters. The key to these are found in the javadoc.

The following is the kind of output expected on iterating through the entire HashMap.

__Code__

IndexReader ir = IndexReader.open(indexName);

RamUsageEstimator estimator = new

estimator.estimateRamUsage(ir);

Stats s;

HashMap h;

s=LuceneCore.getIndexStats();

h = (HashMap) s.getCurrentCounters();

for(String key:h.keySet())

System.out.println("indexStats: "+key+"/"+h.get(key));

s = LuceneCore.getIndexStats();

h = (HashMap) s.getCurrentCounters();

for(String key:h.keySet())

System.out.println("storeStats: "+key+"/"+h.get(key));

s=LuceneCore.getSearchStats();

h = (HashMap) s.getCurrentCounters();

for(String key:h.keySet())

System.out.println("searchStats: "+key+"/"+h.get(key));

s=LuceneCore.getAnalysisStats();

h = (HashMap) s.getCurrentCounters();

for(String key:h.keySet())

System.out.println("analysisStats: "+key+"/"+h.get(key));

ir.close();

__Code Ends__

__Output__

indexStats: iw_adT/2000180

indexStats: ir_C/0

indexStats: ir_newC/1

indexStats: iw_C/1

indexStats: iw_segs/0

indexStats: iw_adC/15

indexStats: iw_buf/0

indexStats: iw_segC/1

indexStats: ir_ram/0

indexStats: iw_newC/1

indexStats: iw_ram/10487

storeStats: iw_adT/2000180

storeStats: ir_C/0

storeStats: ir_newC/1

storeStats: iw_C/1

storeStats: iw_segs/0

storeStats: iw_adC/15

storeStats: iw_buf/0

storeStats: iw_segC/1

storeStats: ir_ram/0

storeStats: iw_newC/1

storeStats: iw_ram/10487

analysisStats: toks/1

analysisStats: tss/30

__Output Ends__

This gives pretty much all of the desired information to optimize the search/index or any other process involving a lucene index Reader/Writer/Searcher.

Thanks to the developers @ Lucid Imagination for coming up with this.

Thanks to Jayant , Nitish for the help. :)

Download Lucid Gaze for Lucene Here [@Lucid Imagination]

Generic Code? Extensible Code?

It has been a question that pops up every time (at-least I) write code. 'How generic should this be?' By Generic, I mean the power to (re)use the same piece of code, without changing anything 'inside' the code and only changing a configuration file (xml or whatever is the implementation choice).

More often than not I end up just trying to write a code so generic, that the purpose it was primarily built for (whatever application) is complicated. Correct, everything now is in the conf file, but at the same time writing the conf file in itself is a task wherein the only kind of people who would want to write that conf instead of rewriting an application specific code would be those who are 'programming challenged'.
I've realized, perhaps if only a few questions are answered prior to writing the so generic code, the developer/designer would be at such a level of ease.
* Who asked for it?
* Would someone else ever use it? Really? Or is it just a mere assumption that someday the world would run on it?
* Assuming that the world might run on it someday, do I need to write code for all of that right now? Can I just write what I want, optimize it for what is required at the moment and a little more and then just let it be? On the lines of the early ages of internet [Design it now, and let it get corrected as it goes on, with the future users correcting it themselves]

There are a lot of other questions which should be answered before the attempt to write 'the universal machine' is made. All attempts to write code are generally towards writing a universal machine which would do all we can think of, all we can imagine, and all that the machine would be able to imagine years from now on! :)
Lets write for 'now' and write it well designed.
Let them extend it rather than configure it....

Lucene Vs Sphinx - A Showdown on a large dataset

There has long existed a battle among the "pro java" and "pro c" developer community. Not that I'd want to strictly be associated with one, but would always say the robustness/exception handling/stability vector have a better cosine value with java as compared to c [With all respect to C, for being the progenitor for Java]. Let me not move off on the tangent and rather more towards the core here. Last few weeks were spent trying to benchmark 'java' lucene as a search engine with 'C' sphinx. Though not exactly in their vanilla form, and a lot of modifications to both, we finally ran a lot of tests on both the engines.

Keeping a common playground with the following specifications:
Processor(s) : Intel Quad Core X 2
RAM : 24G
Operating System : RHEL 32 Bit
Document Corpus : 18 million+ Documents
Source Size : 90G [RDBMS Table]


On a corpus of approximately 18 million records, indexed and not all of them stored. Multiple field queries with varying boost values and some good level of complexity. Here is the result sheet:

Lucene
Index Size : 20G

Concurrency : 30 [5*6 Daemons, 2G each]
Total Searches : 64931
Slow Query Count (>=10 secs) : 3803 ( 5.86%)
Total Duration (secs) : 238094.574
Mean Duration : 3.667
Mode Duration : 0.835
Minimum Duration : 0.001
Maximum Duration : 1174.757
Duration Standard Deviation : 15.441
Search Time (secs) Distribution :-
[0,0.25) : 2770 ( 4.27%)
[0.25,0.5) : 5666 ( 8.73%)
[0.5,1) : 13515 (20.81%)
[1,1.5) : 10928 (16.83%)
[1.5,2) : 7330 (11.29%)
[2,3) : 8476 (13.05%)
[3,5) : 7222 (11.12%)
[5,10) : 5221 ( 8.04%)
[10,20) : 2335 ( 3.60%)
[20,+inf) : 1468 ( 2.26%)

Concurrency : 5 [1 Daemon * 2G]
Total Searches : 225906
Slow Query Count (>=10 secs) : 972 ( 0.43%)
Total Duration (secs) : 186700.646
Mean Duration : 0.826
Mode Duration : 0.003
Minimum Duration : 0.001
Maximum Duration : 467.647
Duration Standard Deviation : 2.864
Duration (secs) Bins :-
[0,0.25) : 64621 (28.61%)
[0.25,0.5) : 58947 (26.09%)
[0.5,1) : 56894 (25.18%)
[1,1.5) : 19836 ( 8.78%)
[1.5,2) : 9397 ( 4.16%)
[2,3) : 7941 ( 3.52%)
[3,5) : 4810 ( 2.13%)
[5,10) : 2488 ( 1.10%)
[10,20) : 684 ( 0.30%)
[20,+inf) : 288 ( 0.13%)



Sphinx
Index Size: 60G

Concurrency: 30
Total Searches : 244431
Slow Query Count (>=10 secs) : 27479 (11.24%)
Total Duration (secs) : 1243474.213
Mean Duration : 5.087
Mode Duration : 0.007
Minimum Duration : 0.001
Maximum Duration : 1869.063
Duration Standard Deviation : 17.833
Average Queries : 2.783
Duration (secs) Bins :-
[0,0.25) : 51186 (20.94%)
[0.25,0.5) : 27798 (11.37%)
[0.5,1) : 32372 (13.24%)
[1,1.5) : 20490 ( 8.38%)
[1.5,2) : 16915 ( 6.92%)
[2,3) : 21833 ( 8.93%)
[3,5) : 23550 ( 9.63%)
[5,10) : 22808 ( 9.33%)
[10,20) : 14540 ( 5.95%)
[20,+inf) : 12939 ( 5.29%)

Concurrency: 5
Total Searches : 226528
Slow Query Count (>=10 secs) : 9895 ( 4.37%)
Total Duration (secs) : 453296.517
Mean Duration : 2.001
Mode Duration : 0.007
Minimum Duration : 0.001
Maximum Duration : 164.713
Duration Standard Deviation : 4.543
Average Queries : 2.773
Duration (secs) Bins :-
[0,0.25) : 71001 (31.34%)
[0.25,0.5) : 36500 (16.11%)
[0.5,1) : 32799 (14.48%)
[1,1.5) : 20416 ( 9.01%)
[1.5,2) : 16385 ( 7.23%)
[2,3) : 13951 ( 6.16%)
[3,5) : 13330 ( 5.88%)
[5,10) : 12251 ( 5.41%)
[10,20) : 7563 ( 3.34%)
[20,+inf) : 2332 ( 1.03%)

As per the analysis, for the dataset analyzed, Lucene was found to win convincingly over its rival.
More details on the same to come soon!

P.S: Though lucene works great for a lot of cases, so does sphinx. Here, lucene seemed to have an upper hand

Go Dutch! @ Info Edge

Nice read....

http://teche-go-dutch.blogspot.com/

Intelligent Video Surveillance

In the light of the bombing in New Delhi, I thought of steering my thoughts (actually seeded by Manisha) to a solution to the important problem of monitoring an over populous city. If we look at it and analyze, we could compare it to a typical search problem. Excessive amounts of data, limited time to process and high level of accuracy.
If we could design a search system, so potent and so intelligent so that it detects and notifies anything/anyone that it thinks is worth a mention, and integrate it with new media to flash it over the cell phones and billboards around the area it is detected in, we would have an amazing (though ideal) system. And trust me, unlike all ideal systems, this is do-able.
Getting into the technicalities of the issue, question 1 is.... what do we already have in place? A terrorist database with photographs and other details, a police force (though not adequate in volumes to monitor and act, both at the same time)
So taking all that data, and first of all digitizing it (which I believe the government would already have done) is the way to start it off. Once the data is digitized, it has to be preprocessed (to prepare it for indexing), exactly the way all data is treated for search engines.
Once we structure the data and order it accordingly(which includes, and primarily includes images), we are ready to index it. Now image indexing is the pivotal thing here as the images are immense and numerous.
What after we are ready with the indexed data? (which happens to be a lot of images).
We need to build an image search engine. Ok.. So how does it differ from a Google image search or a Yahoo image search? Unlike those search engines which are a function of text and not RGB values this has to be one that matches an image for an image ( and similar images).
In other words, this runs a search on an input 'image' and not a keyword search to pull up all images tagged with the keyword.
This is the most important part as it involves bit stream matches, needs an algorithm that knows how to filter out noise over time (so that its noise removal works better with each passing day). Also understands that a person could wear a helmet/scarf and still would have to be detected.
Also, there could be voice matchers that match the voice to make sure that the person is the same and build a mechanism to learn about human voice modulations and variations.
There's a lot more that the search engine could work on and handle (I'm sure more people thinking on it would get better ideas..at least more ideas.. on the issue)
Another question, which again is a pivotal one, would a human match such stuff? as in.. would the input to the system come from a human being? I say... could be.. but rarely... as mostly.... it would have to be integrated with cameras.. 'Intelligent Video Surveillance' cameras. With the current age of technology and canon having its amazing multiple face detection technology, we are almost there on this front. An integration of the technology with the frames (on a sampled basis as all could not be handled) from video cameras could perform a search for faces using the already defined engine. This search would be an ongoing process and as soon as something fishy or known is detected by the system, it could raise an alarm.
We could then integrate it with video devices in police stations and control rooms to flash the captured and detected 'may-be terrorist images' which go in as a lead to the existing police forces.

This I believe would help the country , the police force and the law enforcement agencies like nothing else(at least as far as the current issue is concerned).
There's a need for better technology by the government agencies, with the terrorism taking a new age format that is highly dependent on technology.
Hope someone reads this.. or thinks about it.. someone who has the power for this implementation at national level.

--
P.S : Comments would be appreciated!

An Ideal Agent

Don't we just feel amazing (and willing to pay that extra buck) in case we have are offered personalized service? I guess we do and so we have personal managers. Each time a HNI (High Net-worth Individual) steps into a bank, he is sold a service with a 'personal assistant/agent'.
Why do only the select few HNIs get this kind of benefit and satisfaction? Perhaps its the cost factor. An agent costs money(salary) and the financial institution can not afford to have an agent for a non HNI.
When it seems so obvious, the need for a personalized service seems the right way, what stops us from having a pseudo-personal agent? An artificial agent? Some one over then internet? The agent knows what we do, where we went for our last vacation, who is it that we went with, our last job, our girlfriend (and her birthday) and just does what we humans are ideally expected to do!
Doesn't seem impossible but then is there something we should fear?
I already sense the so called 'what happens to my personal space' and invasion of privacy issues? I say when you could share your wealth information with a human being, what harm could a machine cause? Just that it would help you remember your girlfriends` (or wifes`) birthday on time and might as well help you with other stuff. I really understand the security risks involved with it, but how many of us fear storing our emails in our gmail accounts considering the fact that Google's privacy document (that each gmail user signs) lets them store a copy of all emails sent/received by you (for so called security purposes). If we are not scared storing mails why the hype about everything else?
What are the odds that you wouldn't want this kind of an agent considering that it'd help you each time you log on to the internet by flashing your kind of news, informing about your fav. sports star (say sachin or michael schumacher) or your fav actor (might be SRK for instance).It could also tell you about the time you spent last month visiting all the wrong websites while planning for your vacation(and so would keep you away from such sites).

I'm certainly in favour of having an ideal agent taking care of all of us.... considering our choices. Imagine girls and a machine telling them talking to them all the while they surf.... talking their language....'hey.... doesn't that daimond ring at abc.com look so awesome?' and then you'd check to realize that it exactly matches your choice.... (just because the machine can't think by itself.. it only knows how to think your way.. and by now it has started to understand your likes and your taste....). All of this though, might have one impact.... we might just start liking machines more than human beings... and then we'd (or the biotech guys) be designing 'intelligent' human beings..... :)