|
In late 1997, I began work on a search engine called Pinnacle. Back then, search engines were in their infancy, and there was a huge opening for new technology in that field. I built a small engine in my spare time to demonstrate the novel ranking algorithms I had devised. Of course, this turned out to be far bigger project than I had ever imagined; one that I would continue working on until the present day.
The search engine took roughly 25 months of development time, which was essentially any spare time I could find. I had developed a very primitive search and an assortment of tools which I used to build the indexes of the engine, and a large percentage of the development time I devoted towards the project was in improving performance and optimizing the search algorithms for the most relevant search results.
My cousin Shone joined me on the project in late 1998, and began the process of financing development and registering the company as a corporation. During our humble beginnings we even did several contracts for various companies to contribute to the finances we needed for the project.
Originally, I employed human editors to help build the indexes in the fashion of the Open Directory Project. For this I had built software which allowed volunteers to extract all the links from Netscape and Internet Explorer bookmark files and send them to me to be included in the indexes. This approach unfortunately, did not exactly yield the number of documents I had hoped for, and although the quality of the index was excellent the scope of the search proved overly narrow due to the lack of content.
I quickly realized that I needed a spider, and after two weeks of work I developed Sven. Since I had written it very quickly, there were a number of problems; primarily that it was very slow (single-threaded). We considered Sven to be a bit of an experiment in the field of spiders, and as such it was only used to gather a small number of documents before I went back into development mode and wrote Origin; our first production spider. The main differences between Sven and Origin was that Origin could be stopped and restarted at will, and could be run in parallel. It also had a neat status screen which remained the centerpiece during our countless business discussions and late night programming sessions.
In early 1999, the project was moved Shone's apartment where we set up several servers and a rudimentary network to support the quickly growing search engine. As two young entrepreneurs, we had a difficult time affording the required hardware but fortunately, through Shone's efforts, we were able to build up several servers to support our perpetually growing project.
Within only a few months, our project was rapidly outgrowing our available hardware and we made the effort to redevelop the engine as a parallel implementation. Three AMD 400 MHz machines made up our parallel search cluster, and a file server running Novell, an NT web server and Shone's desktop machine rounded out our very first parallel search engine. We ran the engine in this exact configuration out of a spare room in Shone's apartment for nearly a year!
In early 2000, we were approached by a venture capital firm then known as Western Shores, or WSi Corporation. We signed on with WSi in April, 2000 and moved our servers and supporting hardware to a server room in their facilities shortly thereafter. Provided with a high speed Internet line and access to hardware we had only previously dreamed about, we were ready to take on the world.
The next month, we hired on Allan Scott, a friend of mine who I had worked with on several programming projects with while I was in high school. For several weeks, Allan and I debated over designs and implementation details, and gradually laid out the blueprints for modernizing our brainchild. We left the search engine to run in it's existing configuration for several months during development, and began assembling our test bed for the new implementation of the search.
We purchased three IBM Netfinity 1U servers, each with dual 600 MHz processors and dual 9.1G SCSI hard drives. We recycled some of our old machines for various tasks (servers, spiders, etc) and eventually managed to squeeze the jungle of machines and wires into a rack. Our first task was to build a new spider that was robust and also very fast; now that we had large amounts of hardware available to us, we could develop for a large, distributed platform with ease.
By June 2000, we had successfully created our test index of over one million documents and began work on the new search engine. The engine development took approximately six weeks, and afterwards we began improving, debugging, testing and marketing our new search. Even though the engine wasn't extremely popular, it was used by a fair number of people and we had a monitor set up prominently displaying a continuous list of the current searches that had been performed.
By the end of August, we were beginning to wind down our development phase and wind up our marketing phase. Our plan called for a large-scale crawl of the internet and officially launching our engine to the world. We had identified numerous revenue streams and we began the process of marketing our engine. We moved into new offices, placing our existing servers into one of WSi's data centers, and Allan went back to university.
Our next objective was to acquire the equipment needed to build up a full-scale version of our new search engine, and to show it off to the world. However, we were met with a surprising unwillingness from our parent company to provide us with the required equipment, and as a result we were forced to turn our efforts towards sales and raising money ourselves. We worked very hard for several months, but WSi was already showing the effects of it's mismanagement. By the end of 2000, WSi had laid off most of it's staff and was desperately looking for new sources of income to survive.
The months that followed were a roller coaster ride and took an enormous toll on everyone who remained with the company. Emotionally, it was a difficult ordeal - good news one day, bad news the next - the whole while dealing with a company in a volatile position; an experience I never want to repeat. Eventually, the company was rolled back into a shell while the few people who remained involved searched desperately for ways to survive. Shone and I both put forth valiant efforts towards preserving the future of WSi, and made amazing strides in reducing the company's debt. However, in the end, we negotiated our buyout by the company in exchange for stock, and we were eager to take a breather while the company's position stabilized.
The shell was eventually taken over by Data Fortress Group, a profitable and successful company primarily focused on managed hosting. I had originally hoped that the company would re-fund the development of the search engine - after all, it seemed like a perfect fit; a profitable company with a data center, lots of space, redundant power and tons of bandwidth. The engine was in a functional state as of late 2000 when we stopped development, and it wouldn't take that much to re-develop it as a modern search engine. But it seems that the interest in new, innovative products like the search engine has waned; and while I hope that the company will resurrect the search and build it into a sparkling new engine to compete with the other big names, the company seems to have little interest in doing so and as such, I have lost a lot of hope in this possibility.
For me, iaNett was both an adventure and an exercise in the business world. There are countless lessons that I learned during my time with iaNett; lessons that I could not have learnt otherwise. But in spite of the fact that the search engine has been basically shelved, all the time and effort that I put into it has not been for nothing; after all, I learned more from my experiences that I ever could from school, and that knowledge will be with me for the rest of my life.
I don't like to give up on projects; especially projects that I have put my heart and soul into for years. So in mid-2002, I quietly began work on a new search engine, completely fresh from the ground-up; 100% new code, 100% new concepts and ideas. There was no pressure, so I took my time designing it; incorporating every possible detail I could think of. Countless concessions were made to accommodate the most complex of algorithms, concepts that were built into the engine's kernel from the beginning. Ten months of careful programming, prototypes and two completely different programs later, I had completed the beta of this new engine which worked so well that most people I showed it to believed it to be a final product. But after all these years of learning and building search engines, I know what search engines are capable of. After all, this is the search engine of the future that it being built. There are fewer restrictions these days; RAM, CPU power and disk space are cheaper than ever before, and the trend will only continue. What I am running on my home test system blows what were mainstream search engines only five or six years ago completely out of the water. What is being built here is searching techniques and algorithms that must withstand the tests of time and of millions of queries being done each and every day; software that can scale far beyond that we think of as the size of the internet a few years from now and still return results besting the engines of the future. What I have now is merely a fully-functional platform upon which to test these new algorithms and to test new searching techniques. This is a lightweight package which allows me to easily add and change the algorithms it uses on the fly.
These are definitely exciting times!
|
