Search Engine

Hey there! Fancy meeting you this far into the "documentation". By now you should already be enlightened, though if you made it here without reading any of the prior documents leading up to this, let's walk through this together.

I have a working DNS server. I also have a working domain registrar that works together with the DNS server.
What else can we launch to make the experience even greater? Well, I know what domains are available via my DNS server, though I would also like a way to quickly find relevant information... I kinda miss Google here, but that won't do. No, I should create my own search engine!

This field required additional research, even though I was born in the early 80's and lived through the dial-up hellscape and everything that came with 90s and early 00s perusing of the world wide web, and had a great understanding of how searching worked.. more research was required.
What I came up with seems to have worked quite nicely... I could generate and add some random data to an index and try some different methods of searching.

Friggin' science, this.
Searching was working fairly well, but I could see how this would be a bit of an issue on a small and currently limited network. Larger sites would dominate the search results, I had to make sure that the data that was searched matched as closely as possible with what was being searched for, then group the best matches and limit the output per domain - This took a bit of effort, I must admit.

Searching, whole lot of search algorithm work, result grouping ... what's next. Ah yes, exact word or phrase matching, highlighting, "any of these words", "none of these words" and searching by site/domain with site:domain.tld
The latter were somewhat easy to implement, though it was slightly tricky at times I have to admit. At least I could see the light at the end of the tunnel now.

There was one final thing to add before I was completely happy with what I had up to this point, though it was a headscratcher at the time, so let's skip that for now and bring it up at the end instead. Hint, click tracking.
Anyways, a search engine is fine and all, now that I've emptied out the fake data I'm sitting here like ... well, wouldn't I have to crawl websites now to build a real index? Yes, I would have to do that. I must build a crawler / indexer.

The crawler needs to ...

build a zone index
check excluded subdomain.domain.tld or zonewide via domain.tld for misconfigured servers
check each zone if port 80 or 443 is open, remove any zone that does not respond on either of these ports
HTTP takes priority, and HTTPS crawling only takes effect if HTTP is unavailable
follows HTTP response 200, 301, 302, 308
check remaining zones if we get a response from a web server, remove any zone that does not give an appropriate response
check remaining zones for robots.txt, does the crawler have permission to crawl or not
build a list with sites that respond and give it permission to crawl
begin gently crawling at 1 request every 3 seconds, with up to 3 sites in parallell to speed up the process
automatically adjusts crawl speed depending on server response times (load checks)
parse and index the relevant data from every page, pick up links and follow these by a means of "best effort"
respect privacy, if a previously indexed site decides to not want to be indexed and disallows this in the robots.txt, then delete the index for that site instead

This took a lot of trial and error, and I'm not sorry. This nice and neat little package now contains a crawler/indexer, and a search engine to make use of the index. And it works!
Now I did mention something about click tracking as well, didn't I? It sounds a lot worse than it really is, honest. Here's how it works

clicking a link via the search engine registers the click
the clicked link function notes the date, the target url and the query that was made to find the link
the search engine forwards the user to the clicked link
the clicktracker data is aggregated and used in future searches and boosts the rank of links depending on content and query by a certain amount
better search results
retains privacy, since nothing about the client is used for click aggregation.

Accessing the Search Engine
The search engine is only available while on the Vesper Network via search.com

Next subsection: Archie