Lucid Gaze - Tough Nut!

A One Stop Solution to a lot of statistical analysis for lucene internals is the all new lucid gaze for Lucene. Perhaps it has been around for a while for Solar and I'm left to wonder.. dude...where's the documentation? There aren't many places on the information superhighway where I could spot info on how to use lucid gaze. A Google for the same would prove my point.

After I did figure out how to, it seemed like a good tool, easy to use(after the eureka moment, at least for me). Here's how I analyzed various things using Lucid Gaze by Lucid Imagination.

Pre-requisites:

• lucene core jar from Lucid imagination [Here]

Write the indexing/search logic. Open the Reader/Writer/Searcher as usual.

Create a RamUsageEstimator:

• RamUsageEstimator estimator = new RamUsageEstimator();

At point where you'd want to analyze, do a

• estimator.estimateRamUsage(ir);

Where ir is an IndexReader/IndexWriter/IndexSearcher.

__Snip__

Stats s;

s = LuceneCore.getIndexStats(); //For getting IndexStats

s = LuceneCore.getStoreStats(); //For getting StoreStats

s = LuceneCore.getSearchStats(); //For getting SearchStats

s = LuceneCore.getAnalysisStats(); //For getting AnalysisStats

__Snip Ends__

Once the above step is done, s is populated with a Stats Object containing the Index/Store/Search/Analysis Stats (as per the function call).

__Snip__

HashMap h = (HashMap) s.getCurrentCounters(); // Retrieve counters accumulated since last #resetStats().

__Snip Ends__

This HashMap is populated with current stat counters. The key to these are found in the javadoc.

The following is the kind of output expected on iterating through the entire HashMap.

__Code__

IndexReader ir = IndexReader.open(indexName);

RamUsageEstimator estimator = new

estimator.estimateRamUsage(ir);

Stats s;

HashMap h;

s=LuceneCore.getIndexStats();

h = (HashMap) s.getCurrentCounters();

for(String key:h.keySet())

System.out.println("indexStats: "+key+"/"+h.get(key));

s = LuceneCore.getIndexStats();

h = (HashMap) s.getCurrentCounters();

for(String key:h.keySet())

System.out.println("storeStats: "+key+"/"+h.get(key));

s=LuceneCore.getSearchStats();

h = (HashMap) s.getCurrentCounters();

for(String key:h.keySet())

System.out.println("searchStats: "+key+"/"+h.get(key));

s=LuceneCore.getAnalysisStats();

h = (HashMap) s.getCurrentCounters();

for(String key:h.keySet())

System.out.println("analysisStats: "+key+"/"+h.get(key));

ir.close();

__Code Ends__

__Output__

indexStats: iw_adT/2000180

indexStats: ir_C/0

indexStats: ir_newC/1

indexStats: iw_C/1

indexStats: iw_segs/0

indexStats: iw_adC/15

indexStats: iw_buf/0

indexStats: iw_segC/1

indexStats: ir_ram/0

indexStats: iw_newC/1

indexStats: iw_ram/10487

storeStats: iw_adT/2000180

storeStats: ir_C/0

storeStats: ir_newC/1

storeStats: iw_C/1

storeStats: iw_segs/0

storeStats: iw_adC/15

storeStats: iw_buf/0

storeStats: iw_segC/1

storeStats: ir_ram/0

storeStats: iw_newC/1

storeStats: iw_ram/10487

analysisStats: toks/1

analysisStats: tss/30

__Output Ends__

This gives pretty much all of the desired information to optimize the search/index or any other process involving a lucene index Reader/Writer/Searcher.

Thanks to the developers @ Lucid Imagination for coming up with this.

Thanks to Jayant , Nitish for the help. :)

Download Lucid Gaze for Lucene Here [@Lucid Imagination]

Lucene Vs Sphinx - A Showdown on a large dataset

There has long existed a battle among the "pro java" and "pro c" developer community. Not that I'd want to strictly be associated with one, but would always say the robustness/exception handling/stability vector have a better cosine value with java as compared to c [With all respect to C, for being the progenitor for Java]. Let me not move off on the tangent and rather more towards the core here. Last few weeks were spent trying to benchmark 'java' lucene as a search engine with 'C' sphinx. Though not exactly in their vanilla form, and a lot of modifications to both, we finally ran a lot of tests on both the engines.

Keeping a common playground with the following specifications:
Processor(s) : Intel Quad Core X 2
RAM : 24G
Operating System : RHEL 32 Bit
Document Corpus : 18 million+ Documents
Source Size : 90G [RDBMS Table]


On a corpus of approximately 18 million records, indexed and not all of them stored. Multiple field queries with varying boost values and some good level of complexity. Here is the result sheet:

Lucene
Index Size : 20G

Concurrency : 30 [5*6 Daemons, 2G each]
Total Searches : 64931
Slow Query Count (>=10 secs) : 3803 ( 5.86%)
Total Duration (secs) : 238094.574
Mean Duration : 3.667
Mode Duration : 0.835
Minimum Duration : 0.001
Maximum Duration : 1174.757
Duration Standard Deviation : 15.441
Search Time (secs) Distribution :-
[0,0.25) : 2770 ( 4.27%)
[0.25,0.5) : 5666 ( 8.73%)
[0.5,1) : 13515 (20.81%)
[1,1.5) : 10928 (16.83%)
[1.5,2) : 7330 (11.29%)
[2,3) : 8476 (13.05%)
[3,5) : 7222 (11.12%)
[5,10) : 5221 ( 8.04%)
[10,20) : 2335 ( 3.60%)
[20,+inf) : 1468 ( 2.26%)

Concurrency : 5 [1 Daemon * 2G]
Total Searches : 225906
Slow Query Count (>=10 secs) : 972 ( 0.43%)
Total Duration (secs) : 186700.646
Mean Duration : 0.826
Mode Duration : 0.003
Minimum Duration : 0.001
Maximum Duration : 467.647
Duration Standard Deviation : 2.864
Duration (secs) Bins :-
[0,0.25) : 64621 (28.61%)
[0.25,0.5) : 58947 (26.09%)
[0.5,1) : 56894 (25.18%)
[1,1.5) : 19836 ( 8.78%)
[1.5,2) : 9397 ( 4.16%)
[2,3) : 7941 ( 3.52%)
[3,5) : 4810 ( 2.13%)
[5,10) : 2488 ( 1.10%)
[10,20) : 684 ( 0.30%)
[20,+inf) : 288 ( 0.13%)



Sphinx
Index Size: 60G

Concurrency: 30
Total Searches : 244431
Slow Query Count (>=10 secs) : 27479 (11.24%)
Total Duration (secs) : 1243474.213
Mean Duration : 5.087
Mode Duration : 0.007
Minimum Duration : 0.001
Maximum Duration : 1869.063
Duration Standard Deviation : 17.833
Average Queries : 2.783
Duration (secs) Bins :-
[0,0.25) : 51186 (20.94%)
[0.25,0.5) : 27798 (11.37%)
[0.5,1) : 32372 (13.24%)
[1,1.5) : 20490 ( 8.38%)
[1.5,2) : 16915 ( 6.92%)
[2,3) : 21833 ( 8.93%)
[3,5) : 23550 ( 9.63%)
[5,10) : 22808 ( 9.33%)
[10,20) : 14540 ( 5.95%)
[20,+inf) : 12939 ( 5.29%)

Concurrency: 5
Total Searches : 226528
Slow Query Count (>=10 secs) : 9895 ( 4.37%)
Total Duration (secs) : 453296.517
Mean Duration : 2.001
Mode Duration : 0.007
Minimum Duration : 0.001
Maximum Duration : 164.713
Duration Standard Deviation : 4.543
Average Queries : 2.773
Duration (secs) Bins :-
[0,0.25) : 71001 (31.34%)
[0.25,0.5) : 36500 (16.11%)
[0.5,1) : 32799 (14.48%)
[1,1.5) : 20416 ( 9.01%)
[1.5,2) : 16385 ( 7.23%)
[2,3) : 13951 ( 6.16%)
[3,5) : 13330 ( 5.88%)
[5,10) : 12251 ( 5.41%)
[10,20) : 7563 ( 3.34%)
[20,+inf) : 2332 ( 1.03%)

As per the analysis, for the dataset analyzed, Lucene was found to win convincingly over its rival.
More details on the same to come soon!

P.S: Though lucene works great for a lot of cases, so does sphinx. Here, lucene seemed to have an upper hand

Intelligent Video Surveillance

In the light of the bombing in New Delhi, I thought of steering my thoughts (actually seeded by Manisha) to a solution to the important problem of monitoring an over populous city. If we look at it and analyze, we could compare it to a typical search problem. Excessive amounts of data, limited time to process and high level of accuracy.
If we could design a search system, so potent and so intelligent so that it detects and notifies anything/anyone that it thinks is worth a mention, and integrate it with new media to flash it over the cell phones and billboards around the area it is detected in, we would have an amazing (though ideal) system. And trust me, unlike all ideal systems, this is do-able.
Getting into the technicalities of the issue, question 1 is.... what do we already have in place? A terrorist database with photographs and other details, a police force (though not adequate in volumes to monitor and act, both at the same time)
So taking all that data, and first of all digitizing it (which I believe the government would already have done) is the way to start it off. Once the data is digitized, it has to be preprocessed (to prepare it for indexing), exactly the way all data is treated for search engines.
Once we structure the data and order it accordingly(which includes, and primarily includes images), we are ready to index it. Now image indexing is the pivotal thing here as the images are immense and numerous.
What after we are ready with the indexed data? (which happens to be a lot of images).
We need to build an image search engine. Ok.. So how does it differ from a Google image search or a Yahoo image search? Unlike those search engines which are a function of text and not RGB values this has to be one that matches an image for an image ( and similar images).
In other words, this runs a search on an input 'image' and not a keyword search to pull up all images tagged with the keyword.
This is the most important part as it involves bit stream matches, needs an algorithm that knows how to filter out noise over time (so that its noise removal works better with each passing day). Also understands that a person could wear a helmet/scarf and still would have to be detected.
Also, there could be voice matchers that match the voice to make sure that the person is the same and build a mechanism to learn about human voice modulations and variations.
There's a lot more that the search engine could work on and handle (I'm sure more people thinking on it would get better ideas..at least more ideas.. on the issue)
Another question, which again is a pivotal one, would a human match such stuff? as in.. would the input to the system come from a human being? I say... could be.. but rarely... as mostly.... it would have to be integrated with cameras.. 'Intelligent Video Surveillance' cameras. With the current age of technology and canon having its amazing multiple face detection technology, we are almost there on this front. An integration of the technology with the frames (on a sampled basis as all could not be handled) from video cameras could perform a search for faces using the already defined engine. This search would be an ongoing process and as soon as something fishy or known is detected by the system, it could raise an alarm.
We could then integrate it with video devices in police stations and control rooms to flash the captured and detected 'may-be terrorist images' which go in as a lead to the existing police forces.

This I believe would help the country , the police force and the law enforcement agencies like nothing else(at least as far as the current issue is concerned).
There's a need for better technology by the government agencies, with the terrorism taking a new age format that is highly dependent on technology.
Hope someone reads this.. or thinks about it.. someone who has the power for this implementation at national level.

--
P.S : Comments would be appreciated!