The Condensed History of iaNett
Page 2: The early days
At first I tackled the search software; it wasn't elegant but it worked. There was no web crawler at this point; the indexes were generated from a database of website data that I compiled by hand and extracted from browser bookmarks. The search engine worked well, but I soon realized that for the project to move forward I really needed more content.
It was around this time my cousin Shone took an interest in what I was building and we struck a deal to form a company around the project. Shone had a good business background and this proved to be the catalyst needed to move the project into a grander scope. With a partner taking care of the business side of things, I was now free to focus my energy entirely on the technical issues that were holding back the search engine. We agreed was that building a web crawler would be our highest priority and I immediately set to work on the task. However, as anyone who has built a crawler knows, writing such a tool is a formidable task, and this was a lesson I learned quickly. So for months I pored over the code of what was to become our first web crawler; at first testing against a local collection of documents, and then eventually expanding to a limited set of public websites. I worked on this crawler - nicknamed "Sven" - for months before we had a tool I felt comfortable unleashing on the public internet.
Shone's apartment had a spare bedroom that we used as an office. We purchased several servers, some network equipment, UPSs and a business internet DSL line, and constructed a makeshift data centre in the corner of the room. We were on a very limited budget, so we were determined to keep the costs low and get the most out of the limited capital we had.
Our most powerful machine was a used server that we bought from our uncle's business after they upgraded; it ran Novell and was our file server. Another machine, running Windows NT, served as our web server, and another PC, which Shone also used as a desktop, was typically tasked with running the web crawler. We also purchased three low cost but powerful (at the time) AMD 400mhz computers which formed our initial parallel search cluster. In 1999, RAM was still rather expensive - costing over $100 for 128MB - but since our indexes were stored in RAM for speed, we had no choice but to put a fairly substantial amount of memory in each of these machines.
The search nodes were nearly diskless; although they booted into DOS from a floppy disk, everything was stored on the Novell server, and the task of commissioning a new node was trivial. Indexes were loaded into RAM by the search node software, which then in turn waited for connections from the search server. When a query was submitted, the search server would broadcast a request to all nodes to initiate a search. When the search was complete each node returned their data and the search server merged the results, ranked them by score, and then returned the sorted results to the web server. The entire process for a typical search was substantially less than a second. The results were cached for a short period of time (several hours) which allowed a user to page through the results without performing a new search each time.
Although we were continually tweaking and tuning the software, the whole system worked extremely well. Many times we would work late into the night, occasionally taking breaks to sit back with a cup of coffee to watch the crawlers roving across the net or the screen that predominately showed search queries in real time. A few months later, Sven was rewitten and became "Origin", a vastly improved crawler that could run in parallel to harvest an exponentially greater number of websites. With the new crawler, our indexes grew quickly and soon we found ourselves pushing the limits of our hardware.
It was around this time that we caught the attention of a venture capital firm, then called WSi Corporation located in downtown Vancouver. We demonstrated the search engine and entered negotiations with them, which eventually culminated in an investment of salaries for myself and Shone, office and data centre space, equipment, bandwidth, salaries for a small team of employees, and a commitment to help us with developing the business.