Apache Solr 2014: A Year In Review

The last few years have seen Apache Solr move at a reasonably aggressive pace. 2013 marked the debut of SolrCloud: a newly architected, truly distributed mode of Solr. 2014 saw Solr, specifically SolrCloud, mature in terms of greater ease of use, more predictable scalability and more reliability in production environments with each new release.

Continue reading the post on the official Lucidworks blog here:

http://lucidworks.com/blog/solr-2014-year-review/

Asynchronous Collection API calls in Solr

I just wrote a blogpost about 'Asynchronous Collection API calls in Solr' on SearchHub. Here's the link for the same: https://lucidworks.com/blog/asynchronous-collection-api-calls-in-solr/.

SearchHub Blog: Multi level composite-id routing in SolrCloud

I just wrote a blog at SearchHub about "Multi level composite-id routing in SolrCloud". Here's a link to that: https://lucidworks.com/blog/multi-level-composite-id-routing-solrcloud/.

Third Bangalore Apache Solr/Lucene Meetup

We just had the third Bangalore Apache Solr/Lucene meetup this last weekend. It's good fun to see the community grow to 200+. Actually, as I type this, we're already 3 short of the 250 mark.

As per the requests from a reasonable number of members, we had a "Solr 101" by Varun Thacker form Unbxd. Right before he leaves for his talk at Lucene Revolution Europe 2013, he did get a lot of attendees both introduced and interested in Solr.

His talk was followed up by a talk on a "DIY Bookmarks manager using Lucene" by Umar Shah from SimplyPhi, Bangalore. It was a nice demo and I'm sure this would have again motivated people to try out similar DIY stuff using either Lucene/Solr.

This was followed with a talk by Ganesh M from Dell Sonicwall. Ganesh gave a quick talk about "Building Custom Analyzers in Lucene". A short and quick take on a complex but interesting part of Lucene certainly got the advanced users interested.

After his talk, with the last talk, I spoke about "MoreLikeThis in Solr/Lucene". I started off with some history on the current MLT stuff and how it really works along with what are the kind of issues that people are known to run into when using this feature. I also spoke about a MoreLikeThis QueryParser for Solr that I've been working on as a part of my official work at LucidWorks. We plan to Open Source it as soon as I have time to put it out and document it a bit.

This may well be the last meetup I'd organize and attend in Bangalore for the year. Good luck to Varun and Shalin for organizing this going forward.

Thanks to Microsoft Accelerator for giving us the venue to host the meetup yet again. It's one of the most centrally located and well equipped spaces in Bangalore for such meetups.

In case you'd like to join the meetup group and be a part of the active community, here's where to do that: http://www.meetup.com/Bangalore-Apache-Solr-Lucene-Group/ .

Collection Aliasing in SolrCloud

One of the many features that have come out for SolrCloud has been collection aliasing. As the name suggests, it aliases the collection, in other words gives another name or a pointer to a collection (which can be changed) behind the scenes.
Among other things the most important uses of aliasing a collection could the power to change an index without having to change or modify the client applications. It helps in disconnecting the view from the actual index. So let's see how can we practically use this feature for rather common stuff.

Collection aliasing command:
http://<hostname>:<port>/solr/admin/collections?

                                         action=CREATEALIAS&

                                         name=alias-name&
                                         collections=list-of-collections

Fig 1. myindex with a read and a write alias each

Firstly, it gives users the ability to reindex their content into another collection and then swap it out. If you begin with a setup as in Fig. 1, you'd have to follow the following steps:

• Switch the write alias to a new index.
• Start re-indexing using the index update client that you use. That way you never change the name/alias but behind the scenes, all the updates go to a new index.
• Once the re-indexing is complete, change the read alias to use the new index too.

Updating an existing alias:
An existing alias can be updated with just a fresh CREATEALIAS call with the new alias specifications.

Secondly, the collection aliasing command lets users to specify a single name for a set of collections. This comes in handy if the data has time windows e.g. month-wise. Every month can be a collection by itself and things like last-month, last-quarter can be aliases of appropriate months. It can also be useful when the data gets added e.g. in case of travel/geo search. A continent could be an alias, consisting of collections holding data for certain countries. As data for other countries from the continent comes in, you may create a new collection for those countries and add those to the existing continent aliases.

There's no limit as to what aliasing can be practically used for as far as use cases are concerned, but hope the ones mentioned above help you get an idea of what aliasing is broadly about.

Related Readings:
JIRA: https://issues.apache.org/jira/browse/SOLR-4497

Apache Solr Guide

Photographs from Bangalore Apache Solr/Lucene Meetup#2

Bangalore Apache Solr/Lucene Meetup, a set on Flickr.

Photographs from the 2nd meetup we organized last Saturday.

Second Bangalore Apache Solr/Lucene Meetup

After having a rather good first meetup, I, along with Shalin Mangar and Varun Thacker organised the second one for the Bangalore Apache Solr/Lucene community on the 27th July, 2013. The meetup group has gone up from 85 odd members, around the time of the first meetup, a month and a half ago, to almost 200.

Though we weren't able to get the same venue as last time i.e. Microsoft Accelerator, due to availability issues, we'd like to thank Flipkart for providing us with more than what we bargained for. From a nice space to host the event to constant flow of breakfast and coffee.

Talking about the event, the audience, like last time was a mixed bag. Though there were more than a few people who were completely new to Solr and actually very few who were advanced Solr users, it was good to have as many people take interest in the Apache Lucene/Solr project.

Shalin Shekhar Mangar from LucidWorks, started the talks for the day with a short talk on 'Introduction to Solr' that enabled newbies in the audience to relate to most of the things that were spoken through the rest of the day. He also  demoed people a working Solr instance, the admin interface and the document upload from UI (new in Solr 4.4).

This was followed by a talk by Venkatesh Prasanna from Infosys Technologies. He spoke about 'Knowledge Management at Infosys' and how they use Solr and Nutch within Infosys.

Flipkart didn't just give us a venue but also a talk about what happens in Flipkart Search, along with some insight on e-commerce search challenges in general. Umesh Prasad and Thejus VM spoke about their architecture and why the vector space model doesn't work for them.

The last talk of the day by Jaideep Dhok from Inmobi kept people interested until we called it a day. Talking about "DIY Percolator using Lucene", he highlighted possibilities, use cases and also demoed his code.

All in all, it was a productive day and I assume everyone found it worth waking up early, on a Saturday morning.

In case you're interested in attending these events, feel free to join the meetup group [here]. Also, if you would want to present at the next meetup, kindly drop me a line.

P.S: The presentations from the meetup should up soon. Also, we are figuring out if the video recording (specially audio) quality was good enough to be uploaded and shared. If we end up posting those, I'll share the links at the meetup page.

Shard Splitting in SolrCloud

It's been around two years since my last post and after this hiatus, I intend to be more regular at it.
I've been a part of a couple of organizations since then. After being a part of the small team that launched the first ever AWS Search Service, CloudSearch, I am back to the open source world.

I joined LucidWorks in Jan '13, the primary company backing and supporting the Apache Solr project. The last few months have been spent trying to absorb the changes that the Apache Lucene/Solr project went through over an year and a half, when I was busy with A9. I've also actively worked and contributed on the Shard Splitting feature (SOLR-3755) for SolrCloud, with Shalin Shekhar Mangar since then.

It certainly feels good to be back with this stuff and to begin with, here's my post on Shard Splitting in SolrCloud : http://searchhub.org/2013/06/19/shard-splitting-in-solrcloud/.

Indexing MongoDB Data for Solr

Solr is the popular, blazing fast open source enterprise search platform from the Apache Lucene project.

MongoDB (from "humongous") is a scalable, high-performance, open source, document-oriented data store. 

I was happy using MongoDB and my very own search engine written using/extending lucene, until the trunks for Solr and Lucene were merged. This merge translated to Solr using the same release of lucene that I was using, unlike the past when there was some disconnect between the two. I realized that a lot of what I was trying to build was available through Solr.

Though Solr is used by a lot of organizations (which can be found here) and I'm sure that at least a few of them using Mongo, for some reason there was/is no straight forward out of the box import handler for data stored in MongoDB.

This made me search for a framework/module/plug to do the same, but in vain.

All said and done, here's a way that I finally was able to index my mongodb data into Solr.

I've used SolrJ to access my Solr instance and a mongo connector to connect to Mongo. Having written my own sweet layer that has access to both the elements of the app, I have been able to inject data as required.

--snip--

public SolrServer getSolrServer(String solrHost, String solrPort) throws MalformedURLException {
    String urlString = "http://"+solrHost+":"+solrPort+"/solr";
    return new CommonsHttpSolrServer(urlString);

}

--/snip--

Fire the mongo query, iterate and add to the index

--snip--

SolrServer server = getSolrServer(..); //Get a server instance

DBCursor curr = ..; //Fire query @ mongo, get the cursor

while (curr.hasNext()) { //iterate over the result set
        BasicDBObject record = (BasicDBObject) curr.next();
        //Do some magic, get a document bean       

        server.addBean(doc);

}

server.commit();

--/snip--

This will get you started on your track to index mongo data into a running Solr instance.

Also, remember to configure Solr correctly for this to run smooth.

Download Resources:

• Mongo Drivers
• SolrJ

Searching off the RAM

Search engines are a lot about precision, recall and speed. These three factors pretty much define the quality of a search engine. I'd only talk about the last point here, speed. The time taken to search for a search engine is such a critical factor that an improvement of a few hundred milliseconds is of extreme importance to anyone associated with developing/designing search engines.
More often than not, as a short term gain, all of us look at putting in more money on the hardware to improve a system's performance. Though this might look like a solution, its bound to fail if you try to run away from actually fixing the application architecture, which happens to be the root cause for poorly performing applications generally.
For those who have already done whatever it takes to optimize the search process, here are a few ways that are generally used to host the search index on the RAM, in order to improve the search speed.
You may mount a tmpfs/ramfs on your machine and copy the index on it. You may then open index reader/searcher on this copy of the index. This would help in reducing the I/O latency and improve the search speed.
The difference between using tmpfs vs ramfs are:

• ramfs will grow dynamically unlike tmpfs which is static and initialized with a fixed size.
• tmpfs uses swap memory whereas ramfs doesn't.

I have personally used tmpfs and it works rather efficiently.
One thing to remember is that both tmpfs and ramfs are volatile. They both get erased on a system restart and hence you'd need to re-mount and copy the index on system startup.

Mounting tmpfs: 
mkdir -p /mnt/tmp 
mount -t tmpfs -o size=20m tmpfs /mnt/tmp

Mounting ramfs:
mkdir -p /mnt/ram 
mount -t ramfs -o size=20m ramfs /mnt/ram

The other approach in case you're using lucene is to have the IndexReader use a Ram Directory. In this case you'd need a JVM instance with enough memory to load all of the index contents into memory and have more for processing search queries.
Also, that may translate to 'requiring'a 64-bit JVM so that it could use pseudo unlimited address space.

--Snip--

NIOFSDirectory nDir = new NIOFSDirectory(new File(indexDir));
RAMDirectory directory = new RAMDirectory(nDir);
IndexReader ir = IndexReader.open(directory);
IndexSearcher indexSearcher = new IndexSearcher(ir);

--Snip--

I haven't been really fond of using this approach as it forces a 64-bit architecture to be used but it may surely work as there's no overhead of using and manipulating the data manually as in the case of maintaining a tmpfs. The cleanups etc...

These are the basic 2 techniques to be used if you want your index to be fed off the RAM. It is a frequent question on the lucene users mailing list, so perhaps people can now stop asking that question... well.. almost...
All said and done.. don't stop optimizing the engine/app if your search is slow.. 99% of the times.. that is where it has to be handled.

Dealing with High Dimensional Data

I believe that data is best represented as a vector. For those who haven't heard this before, well let me start with a very basic example of 'how' this is done.

Lets assume a corpus of documents which have only 2 unique words(A,B) in its dictionary (If that was hard to follow, comment and I shall follow up with what that means). Now a document containing only 'A' is a unit vector along the direction of 'A' and so with a document containing only a single occurrence of 'B'. Documents with 'x' As and 'y' Bs can hence be represented as :

x a + y b (a and b are unit vectors along A and B).

When the corpus comprises of documents wherein there are a lot of terms with a very low document frequency, it is referred to as high dimensional data. An example would be a list of proper nouns e.g. hotel names.

High dimensional data, poses a lot of issues primarily due to its sparseness in the vector space. The sparseness of data makes a lot of tasks like clustering and tagging challenging. In order to process this data, more often than not, there is a need for reducing the dimension of the documents (sparseness). I'll discuss a relatively easy way to reduce the dimension of such data.

Given a corpus of high dimensional data, create document vectors for each of them. Create a term frequency matrix for the corpus and follow it up with dropping off all terms that occur in less than 10% (might vary as per the corpus/dataset) documents. Statistically this should remove around 60% of the documents

Also, removing the terms that occur in more than 80% of the documents would lead to removing a considerable ratio of terms that are redundant and too frequent. Such terms are generally tagged as stop-words and removed under all normal data/text processing algorithms.

The residue that remains now is of a considerably reduced dimension. This is a straightforward way of projecting the original data on a multi dimensional plane. A plane comprising of all dimensions that were reduced.

This data can now be consumed for any processing viz. clustering, classification etc..

Posts on how to cluster and various clustering techniques would soon follow.. unlike this one which took ages!

Google Intelligence!

Google seems to be the definitive word when it comes to household usage for the term, intelligent! Detecting the intent and context has always been a tough nut to crack but it seems like the search major has its own way to handle it.

So when you type in 'facebook is' or 'twitter is' @ Google's search box, this is exactly the kind of suggestions it'd throw at you. Try it out at yourself.

Source for idea/thought: Cleartrip official blog

Lucid Gaze - Tough Nut!

A One Stop Solution to a lot of statistical analysis for lucene internals is the all new lucid gaze for Lucene. Perhaps it has been around for a while for Solar and I'm left to wonder.. dude...where's the documentation? There aren't many places on the information superhighway where I could spot info on how to use lucid gaze. A Google for the same would prove my point.

After I did figure out how to, it seemed like a good tool, easy to use(after the eureka moment, at least for me). Here's how I analyzed various things using Lucid Gaze by Lucid Imagination.

Pre-requisites:

• lucene core jar from Lucid imagination [Here]

Write the indexing/search logic. Open the Reader/Writer/Searcher as usual.

Create a RamUsageEstimator:

• RamUsageEstimator estimator = new RamUsageEstimator();

At point where you'd want to analyze, do a

• estimator.estimateRamUsage(ir);

Where ir is an IndexReader/IndexWriter/IndexSearcher.

__Snip__

Stats s;

s = LuceneCore.getIndexStats(); //For getting IndexStats

s = LuceneCore.getStoreStats(); //For getting StoreStats

s = LuceneCore.getSearchStats(); //For getting SearchStats

s = LuceneCore.getAnalysisStats(); //For getting AnalysisStats

__Snip Ends__

Once the above step is done, s is populated with a Stats Object containing the Index/Store/Search/Analysis Stats (as per the function call).

__Snip__

HashMap h = (HashMap) s.getCurrentCounters(); // Retrieve counters accumulated since last #resetStats().

__Snip Ends__

This HashMap is populated with current stat counters. The key to these are found in the javadoc.

The following is the kind of output expected on iterating through the entire HashMap.

__Code__

IndexReader ir = IndexReader.open(indexName);

RamUsageEstimator estimator = new

estimator.estimateRamUsage(ir);

Stats s;

HashMap h;

s=LuceneCore.getIndexStats();

h = (HashMap) s.getCurrentCounters();

for(String key:h.keySet())

System.out.println("indexStats: "+key+"/"+h.get(key));

s = LuceneCore.getIndexStats();

h = (HashMap) s.getCurrentCounters();

for(String key:h.keySet())

System.out.println("storeStats: "+key+"/"+h.get(key));

s=LuceneCore.getSearchStats();

h = (HashMap) s.getCurrentCounters();

for(String key:h.keySet())

System.out.println("searchStats: "+key+"/"+h.get(key));

s=LuceneCore.getAnalysisStats();

h = (HashMap) s.getCurrentCounters();

for(String key:h.keySet())

System.out.println("analysisStats: "+key+"/"+h.get(key));

ir.close();

__Code Ends__

__Output__

indexStats: iw_adT/2000180

indexStats: ir_C/0

indexStats: ir_newC/1

indexStats: iw_C/1

indexStats: iw_segs/0

indexStats: iw_adC/15

indexStats: iw_buf/0

indexStats: iw_segC/1

indexStats: ir_ram/0

indexStats: iw_newC/1

indexStats: iw_ram/10487

storeStats: iw_adT/2000180

storeStats: ir_C/0

storeStats: ir_newC/1

storeStats: iw_C/1

storeStats: iw_segs/0

storeStats: iw_adC/15

storeStats: iw_buf/0

storeStats: iw_segC/1

storeStats: ir_ram/0

storeStats: iw_newC/1

storeStats: iw_ram/10487

analysisStats: toks/1

analysisStats: tss/30

__Output Ends__

This gives pretty much all of the desired information to optimize the search/index or any other process involving a lucene index Reader/Writer/Searcher.

Thanks to the developers @ Lucid Imagination for coming up with this.

Thanks to Jayant , Nitish for the help. :)

Download Lucid Gaze for Lucene Here [@Lucid Imagination]

Generic Code? Extensible Code?

It has been a question that pops up every time (at-least I) write code. 'How generic should this be?' By Generic, I mean the power to (re)use the same piece of code, without changing anything 'inside' the code and only changing a configuration file (xml or whatever is the implementation choice).

More often than not I end up just trying to write a code so generic, that the purpose it was primarily built for (whatever application) is complicated. Correct, everything now is in the conf file, but at the same time writing the conf file in itself is a task wherein the only kind of people who would want to write that conf instead of rewriting an application specific code would be those who are 'programming challenged'.
I've realized, perhaps if only a few questions are answered prior to writing the so generic code, the developer/designer would be at such a level of ease.
* Who asked for it?
* Would someone else ever use it? Really? Or is it just a mere assumption that someday the world would run on it?
* Assuming that the world might run on it someday, do I need to write code for all of that right now? Can I just write what I want, optimize it for what is required at the moment and a little more and then just let it be? On the lines of the early ages of internet [Design it now, and let it get corrected as it goes on, with the future users correcting it themselves]

There are a lot of other questions which should be answered before the attempt to write 'the universal machine' is made. All attempts to write code are generally towards writing a universal machine which would do all we can think of, all we can imagine, and all that the machine would be able to imagine years from now on! :)
Lets write for 'now' and write it well designed.
Let them extend it rather than configure it....

Lucene Vs Sphinx - A Showdown on a large dataset

There has long existed a battle among the "pro java" and "pro c" developer community. Not that I'd want to strictly be associated with one, but would always say the robustness/exception handling/stability vector have a better cosine value with java as compared to c [With all respect to C, for being the progenitor for Java]. Let me not move off on the tangent and rather more towards the core here. Last few weeks were spent trying to benchmark 'java' lucene as a search engine with 'C' sphinx. Though not exactly in their vanilla form, and a lot of modifications to both, we finally ran a lot of tests on both the engines.

Keeping a common playground with the following specifications:
Processor(s) : Intel Quad Core X 2
RAM : 24G
Operating System : RHEL 32 Bit
Document Corpus : 18 million+ Documents
Source Size : 90G [RDBMS Table]


On a corpus of approximately 18 million records, indexed and not all of them stored. Multiple field queries with varying boost values and some good level of complexity. Here is the result sheet:

Lucene
Index Size : 20G

Concurrency : 30 [5*6 Daemons, 2G each]
Total Searches : 64931
Slow Query Count (>=10 secs) : 3803 ( 5.86%)
Total Duration (secs) : 238094.574
Mean Duration : 3.667
Mode Duration : 0.835
Minimum Duration : 0.001
Maximum Duration : 1174.757
Duration Standard Deviation : 15.441
Search Time (secs) Distribution :-
[0,0.25) : 2770 ( 4.27%)
[0.25,0.5) : 5666 ( 8.73%)
[0.5,1) : 13515 (20.81%)
[1,1.5) : 10928 (16.83%)
[1.5,2) : 7330 (11.29%)
[2,3) : 8476 (13.05%)
[3,5) : 7222 (11.12%)
[5,10) : 5221 ( 8.04%)
[10,20) : 2335 ( 3.60%)
[20,+inf) : 1468 ( 2.26%)

Concurrency : 5 [1 Daemon * 2G]
Total Searches : 225906
Slow Query Count (>=10 secs) : 972 ( 0.43%)
Total Duration (secs) : 186700.646
Mean Duration : 0.826
Mode Duration : 0.003
Minimum Duration : 0.001
Maximum Duration : 467.647
Duration Standard Deviation : 2.864
Duration (secs) Bins :-
[0,0.25) : 64621 (28.61%)
[0.25,0.5) : 58947 (26.09%)
[0.5,1) : 56894 (25.18%)
[1,1.5) : 19836 ( 8.78%)
[1.5,2) : 9397 ( 4.16%)
[2,3) : 7941 ( 3.52%)
[3,5) : 4810 ( 2.13%)
[5,10) : 2488 ( 1.10%)
[10,20) : 684 ( 0.30%)
[20,+inf) : 288 ( 0.13%)



Sphinx
Index Size: 60G

Concurrency: 30
Total Searches : 244431
Slow Query Count (>=10 secs) : 27479 (11.24%)
Total Duration (secs) : 1243474.213
Mean Duration : 5.087
Mode Duration : 0.007
Minimum Duration : 0.001
Maximum Duration : 1869.063
Duration Standard Deviation : 17.833
Average Queries : 2.783
Duration (secs) Bins :-
[0,0.25) : 51186 (20.94%)
[0.25,0.5) : 27798 (11.37%)
[0.5,1) : 32372 (13.24%)
[1,1.5) : 20490 ( 8.38%)
[1.5,2) : 16915 ( 6.92%)
[2,3) : 21833 ( 8.93%)
[3,5) : 23550 ( 9.63%)
[5,10) : 22808 ( 9.33%)
[10,20) : 14540 ( 5.95%)
[20,+inf) : 12939 ( 5.29%)

Concurrency: 5
Total Searches : 226528
Slow Query Count (>=10 secs) : 9895 ( 4.37%)
Total Duration (secs) : 453296.517
Mean Duration : 2.001
Mode Duration : 0.007
Minimum Duration : 0.001
Maximum Duration : 164.713
Duration Standard Deviation : 4.543
Average Queries : 2.773
Duration (secs) Bins :-
[0,0.25) : 71001 (31.34%)
[0.25,0.5) : 36500 (16.11%)
[0.5,1) : 32799 (14.48%)
[1,1.5) : 20416 ( 9.01%)
[1.5,2) : 16385 ( 7.23%)
[2,3) : 13951 ( 6.16%)
[3,5) : 13330 ( 5.88%)
[5,10) : 12251 ( 5.41%)
[10,20) : 7563 ( 3.34%)
[20,+inf) : 2332 ( 1.03%)

As per the analysis, for the dataset analyzed, Lucene was found to win convincingly over its rival.
More details on the same to come soon!

P.S: Though lucene works great for a lot of cases, so does sphinx. Here, lucene seemed to have an upper hand

Go Dutch! @ Info Edge

Nice read....

http://teche-go-dutch.blogspot.com/

Intelligent Video Surveillance

In the light of the bombing in New Delhi, I thought of steering my thoughts (actually seeded by Manisha) to a solution to the important problem of monitoring an over populous city. If we look at it and analyze, we could compare it to a typical search problem. Excessive amounts of data, limited time to process and high level of accuracy.
If we could design a search system, so potent and so intelligent so that it detects and notifies anything/anyone that it thinks is worth a mention, and integrate it with new media to flash it over the cell phones and billboards around the area it is detected in, we would have an amazing (though ideal) system. And trust me, unlike all ideal systems, this is do-able.
Getting into the technicalities of the issue, question 1 is.... what do we already have in place? A terrorist database with photographs and other details, a police force (though not adequate in volumes to monitor and act, both at the same time)
So taking all that data, and first of all digitizing it (which I believe the government would already have done) is the way to start it off. Once the data is digitized, it has to be preprocessed (to prepare it for indexing), exactly the way all data is treated for search engines.
Once we structure the data and order it accordingly(which includes, and primarily includes images), we are ready to index it. Now image indexing is the pivotal thing here as the images are immense and numerous.
What after we are ready with the indexed data? (which happens to be a lot of images).
We need to build an image search engine. Ok.. So how does it differ from a Google image search or a Yahoo image search? Unlike those search engines which are a function of text and not RGB values this has to be one that matches an image for an image ( and similar images).
In other words, this runs a search on an input 'image' and not a keyword search to pull up all images tagged with the keyword.
This is the most important part as it involves bit stream matches, needs an algorithm that knows how to filter out noise over time (so that its noise removal works better with each passing day). Also understands that a person could wear a helmet/scarf and still would have to be detected.
Also, there could be voice matchers that match the voice to make sure that the person is the same and build a mechanism to learn about human voice modulations and variations.
There's a lot more that the search engine could work on and handle (I'm sure more people thinking on it would get better ideas..at least more ideas.. on the issue)
Another question, which again is a pivotal one, would a human match such stuff? as in.. would the input to the system come from a human being? I say... could be.. but rarely... as mostly.... it would have to be integrated with cameras.. 'Intelligent Video Surveillance' cameras. With the current age of technology and canon having its amazing multiple face detection technology, we are almost there on this front. An integration of the technology with the frames (on a sampled basis as all could not be handled) from video cameras could perform a search for faces using the already defined engine. This search would be an ongoing process and as soon as something fishy or known is detected by the system, it could raise an alarm.
We could then integrate it with video devices in police stations and control rooms to flash the captured and detected 'may-be terrorist images' which go in as a lead to the existing police forces.

This I believe would help the country , the police force and the law enforcement agencies like nothing else(at least as far as the current issue is concerned).
There's a need for better technology by the government agencies, with the terrorism taking a new age format that is highly dependent on technology.
Hope someone reads this.. or thinks about it.. someone who has the power for this implementation at national level.

--
P.S : Comments would be appreciated!

An Ideal Agent

Don't we just feel amazing (and willing to pay that extra buck) in case we have are offered personalized service? I guess we do and so we have personal managers. Each time a HNI (High Net-worth Individual) steps into a bank, he is sold a service with a 'personal assistant/agent'.
Why do only the select few HNIs get this kind of benefit and satisfaction? Perhaps its the cost factor. An agent costs money(salary) and the financial institution can not afford to have an agent for a non HNI.
When it seems so obvious, the need for a personalized service seems the right way, what stops us from having a pseudo-personal agent? An artificial agent? Some one over then internet? The agent knows what we do, where we went for our last vacation, who is it that we went with, our last job, our girlfriend (and her birthday) and just does what we humans are ideally expected to do!
Doesn't seem impossible but then is there something we should fear?
I already sense the so called 'what happens to my personal space' and invasion of privacy issues? I say when you could share your wealth information with a human being, what harm could a machine cause? Just that it would help you remember your girlfriends` (or wifes`) birthday on time and might as well help you with other stuff. I really understand the security risks involved with it, but how many of us fear storing our emails in our gmail accounts considering the fact that Google's privacy document (that each gmail user signs) lets them store a copy of all emails sent/received by you (for so called security purposes). If we are not scared storing mails why the hype about everything else?
What are the odds that you wouldn't want this kind of an agent considering that it'd help you each time you log on to the internet by flashing your kind of news, informing about your fav. sports star (say sachin or michael schumacher) or your fav actor (might be SRK for instance).It could also tell you about the time you spent last month visiting all the wrong websites while planning for your vacation(and so would keep you away from such sites).

I'm certainly in favour of having an ideal agent taking care of all of us.... considering our choices. Imagine girls and a machine telling them talking to them all the while they surf.... talking their language....'hey.... doesn't that daimond ring at abc.com look so awesome?' and then you'd check to realize that it exactly matches your choice.... (just because the machine can't think by itself.. it only knows how to think your way.. and by now it has started to understand your likes and your taste....). All of this though, might have one impact.... we might just start liking machines more than human beings... and then we'd (or the biotech guys) be designing 'intelligent' human beings..... :)

Page Rank - My Version!

So what is 'Page Rank'?
There are a little too many answers to the question. Seems like everyone has a 'similar' version. In other terms, its the same style of cooking but a different , leading to a distinction in taste. I would not really try to define the ranking algorithm that was finely designed by the 2 Stanford University PhD students (and as one of the thought also goes, in the name of one of the designers of the algo).
I would like to have my thought on the algo added to the already existing ones, though this one is like a more generic one. I'd say that PageRank of a page(or a document), is the probability of a web searcher ending his search at the particular page looking for 'X' keywords assuming that he/she had infinite amount of time at his/her disposal.
Now the question is, how do we (or a machine) calculate that probability?
There are a lot of things that would have some relationship with this calculation. Starting off with the most heard of, 'back links'. Probability and logic have it, the more the links from other page to this page, the more are the chances of someone reaching the page. Thats logic and the probability is math.
The odds of terminating the search at a particular website also is a factor of the reliability factor of the page (or the website). The credibility is as important as anything because no one would want to call it a day with all the wrong information, would you in that position terminate it there? I'm sure 'No' is the answer (unless you have a deadline, meeting which is the priority for now as compared to reliability :P ). This is the reason why .gov, .edu, .google.co* and wikipedia are given the boost they are given while google ranks pages for your search.
Also, having spoken about the back linking of websites, there is a damping factor associated for each hop so that having traversed through 2 edges to reach a page would weigh less as compared to a single hop. The damping factor supposedly being 0.85.
Next question that would come to mind would be, what about the newbies? In that case, there's a default value (Supposedly 0.15) for them.
Those are the important components of page rank.
All said and done, we would still be concerned with the question of 'where do we start' and how do we get a back link etc rank until we first build the network? This boils down to the chicked and hen problem.. right!!! :)
Well, it is supposed to start off with default values at the start and then the process is repeated 'n' times (where 'n' is a very high number) to fine tune the ranks to get closer to their real values.

This is my version of the way the best 'generic' search engine on the planet works (and there's a lot more than this that is used for fine tuning search results and which is beyond the scope of this entry).

Search Engine and Artificial Intelligence!

Search Engines encompass a variety of sub domains or specialized fields of computer science. Starting from Information Retrieval (as they were in the primitive form) to being highly related to Artificially Intelligence (check out the few options mentioned below). Search engines are ideally to be self learning machines that can rightfully decipher the queries provided to them as Natural Languages, also understanding the intent and context of the searcher viz. differentiating between Jaguar - the car and Jaguar - the animal.

Anatomy of a Search Engine,
Generally indexing comprises of

• Porting/Crawling (to for a data source)
• Mapping, Cleanup
• Indexing

And searching

• Index reader
• Query formation
• Searcher (for selecting data)
• Scoring mechanism (for ordering data)

Other ideas that are of use while designing a search engine:

• Mapping
• Clustering
• Wisdom of crowd for
  • Relevance scoring
  • Spam filtering
  • Suggestion Engine
• Natural Language Processing for
  • Query formation
  • Intent determination
  • One box searching - G****E like!

There's a lot more that could be of use, but lets start off with this for the moment!