d8888888b    888 .d8888b.888888888888888888888Y88b   d88P  .d8888b. 8888b
       d888888888b   888d88P  Y88b   888    888        Y88b d88P  d88P  Y88b88888b
      d88P88888888b  888Y88b.        888    888         Y88o88P   888    888888?88b
     d88P 888888Y88b 888 "Y888b.     888    8888888      Y888P    888       888 ?88b
    d88P  888888 Y88b888    "Y88b.   888    888           888     888       888  ?8bb
   d88P   888888  Y88888      "888   888    888           888     888    888888   ?8bb
  d8888888888888   Y8888Y88b  d88P   888    888           888  d8bY88b  d88P8888888888b
 d88P     888888    Y888 "Y8888P"    888    8888888888    888  Y8P "Y8888P" 888     ?88b
About Me

The Condensed History of iaNett

Page 2: The early days

At first I tackled the search software; it wasn't elegant but it worked. There was no web crawler at this point; the indexes were generated from a database of website data that I compiled by hand and extracted from browser bookmarks. The search engine worked well, but I soon realized that for the project to move forward I really needed more content.

It was around this time my cousin Shone took an interest in what I was building and we struck a deal to form a company around the project. Shone had a good business background and this proved to be the catalyst needed to move the project into a grander scope. With a partner taking care of the business side of things, I was now free to focus my energy entirely on the technical issues that were holding back the search engine. We agreed was that building a web crawler would be our highest priority and I immediately set to work on the task. However, as anyone who has built a crawler knows, writing such a tool is a formidable task, and this was a lesson I learned quickly. So for months I pored over the code of what was to become our first web crawler; at first testing against a local collection of documents, and then eventually expanding to a limited set of public websites. I worked on this crawler - nicknamed "Sven" - for months before we had a tool I felt comfortable unleashing on the public internet.

Shone's apartment had a spare bedroom that we used as an office. We purchased several servers, some network equipment, UPSs and a business internet DSL line, and constructed a makeshift data centre in the corner of the room. We were on a very limited budget, so we were determined to keep the costs low and get the most out of the limited capital we had.

Our most powerful machine was a used server that we bought from our uncle's business after they upgraded; it ran Novell and was our file server. Another machine, running Windows NT, served as our web server, and another PC, which Shone also used as a desktop, was typically tasked with running the web crawler. We also purchased three low cost but powerful (at the time) AMD 400mhz computers which formed our initial parallel search cluster. In 1999, RAM was still rather expensive - costing over $100 for 128MB - but since our indexes were stored in RAM for speed, we had no choice but to put a fairly substantial amount of memory in each of these machines.

The search nodes were nearly diskless; although they booted into DOS from a floppy disk, everything was stored on the Novell server, and the task of commissioning a new node was trivial. Indexes were loaded into RAM by the search node software, which then in turn waited for connections from the search server. When a query was submitted, the search server would broadcast a request to all nodes to initiate a search. When the search was complete each node returned their data and the search server merged the results, ranked them by score, and then returned the sorted results to the web server. The entire process for a typical search was substantially less than a second. The results were cached for a short period of time (several hours) which allowed a user to page through the results without performing a new search each time.

Although we were continually tweaking and tuning the software, the whole system worked extremely well. Many times we would work late into the night, occasionally taking breaks to sit back with a cup of coffee to watch the crawlers roving across the net or the screen that predominately showed search queries in real time. A few months later, Sven was rewitten and became "Origin", a vastly improved crawler that could run in parallel to harvest an exponentially greater number of websites. With the new crawler, our indexes grew quickly and soon we found ourselves pushing the limits of our hardware.

It was around this time that we caught the attention of a venture capital firm, then called WSi Corporation located in downtown Vancouver. We demonstrated the search engine and entered negotiations with them, which eventually culminated in an investment of salaries for myself and Shone, office and data centre space, equipment, bandwidth, salaries for a small team of employees, and a commitment to help us with developing the business.

Home - Geocaching - PHP Scripts - Articles - Music - Programming - Hot Sauce Review
About This Site - Site Search - About Me - Link Directory - Contact Me

This site was constructed entirely by hand { more info }
Modified Tuesday, November 4th 2014 UTC

(C) Copyright 2000-2017 Marty Anstey ~~ I didn't rip you off, so don't rip me off.