Monday, September 7, 2009

Lucid Gaze - Tough Nut!

A One Stop Solution to a lot of statistical analysis for lucene internals is the all new lucid gaze for Lucene. Perhaps it has been around for a while for Solar and I'm left to wonder.. dude...where's the documentation? There aren't many places on the information superhighway where I could spot info on how to use lucid gaze. A Google for the same would prove my point.
After I did figure out how to, it seemed like a good tool, easy to use(after the eureka moment, at least for me). Here's how I analyzed various things using Lucid Gaze by Lucid Imagination.

  • lucene core jar from Lucid imagination [Here]
Write the indexing/search logic. Open the Reader/Writer/Searcher as usual.
Create a RamUsageEstimator:
  • RamUsageEstimator estimator = new RamUsageEstimator();
At point where you'd want to analyze, do a
  • estimator.estimateRamUsage(ir);
Where ir is an IndexReader/IndexWriter/IndexSearcher.

Stats s;
s = LuceneCore.getIndexStats(); //For getting IndexStats
s = LuceneCore.getStoreStats(); //For getting StoreStats
s = LuceneCore.getSearchStats(); //For getting SearchStats
s = LuceneCore.getAnalysisStats(); //For getting AnalysisStats
__Snip Ends__

Once the above step is done, s is populated with a Stats Object containing the Index/Store/Search/Analysis Stats (as per the function call).

HashMap h = (HashMap) s.getCurrentCounters(); // Retrieve counters accumulated since last #resetStats().
__Snip Ends__
This HashMap is populated with current stat counters. The key to these are found in the javadoc.
The following is the kind of output expected on iterating through the entire HashMap.

IndexReader ir =;
RamUsageEstimator estimator = new
Stats s;
HashMap h;
h = (HashMap) s.getCurrentCounters();
for(String key:h.keySet())
System.out.println("indexStats: "+key+"/"+h.get(key));
s = LuceneCore.getIndexStats();
h = (HashMap) s.getCurrentCounters();
for(String key:h.keySet())
System.out.println("storeStats: "+key+"/"+h.get(key));
h = (HashMap) s.getCurrentCounters();
for(String key:h.keySet())
System.out.println("searchStats: "+key+"/"+h.get(key));
h = (HashMap) s.getCurrentCounters();
for(String key:h.keySet())
System.out.println("analysisStats: "+key+"/"+h.get(key));

__Code Ends__

indexStats: iw_adT/2000180
indexStats: ir_C/0
indexStats: ir_newC/1
indexStats: iw_C/1
indexStats: iw_segs/0
indexStats: iw_adC/15
indexStats: iw_buf/0
indexStats: iw_segC/1
indexStats: ir_ram/0
indexStats: iw_newC/1
indexStats: iw_ram/10487
storeStats: iw_adT/2000180
storeStats: ir_C/0
storeStats: ir_newC/1
storeStats: iw_C/1
storeStats: iw_segs/0
storeStats: iw_adC/15
storeStats: iw_buf/0
storeStats: iw_segC/1
storeStats: ir_ram/0
storeStats: iw_newC/1
storeStats: iw_ram/10487
analysisStats: toks/1
analysisStats: tss/30
__Output Ends__

This gives pretty much all of the desired information to optimize the search/index or any other process involving a lucene index Reader/Writer/Searcher.
Thanks to the developers @ Lucid Imagination for coming up with this.
Thanks to Jayant , Nitish for the help. :)

Download Lucid Gaze for Lucene Here [@Lucid Imagination]