Über Grep

I have a 1 Billion record text file that currently has just 3 columns of data. It's 92 gigs uncompressed. On a fairly decent server it currently takes about 40 minutes to grep through the file and output the records I am interested in. I would like to speed that process up by orders of magnitude. Under 10 seconds would be nice. 3 seconds would be fantastic. I have explored various options such as Hadoop, noSQL, Map Reduce, etc. I need an expert to help me plan this project. I don't need a coder yet. This project is to explain the terrain, explore the options, and write up a document that will outline the plan and the estimated budget.

The data will be externalized as a service and an API design with key will be outlined along with schemas, flowcharts, system requirements, and other supporting professional documentation. I am open to putting this in the cloud, or buying the servers required for this endeavor. I would like to know the pros and cons of either approach from someone who has been there and done it, preferably more than once. Most likely, the author of this document will also get the job for implementing the project, but I cannot guarantee that.

I look forward to hearing from you.

Escritura técnica

Nº del proyecto: #3677451

Sobre el proyecto