Understanding Google Search: A Basic Guide to Crawling, Indexing, and Serving

Understanding Google Search: A Basic Guide to Crawling, Indexing, and Serving
Image by Danni Liu, Google Bot sourced from Google

In my early years, I often found myself green with envy when I saw other kids receiving help from their parents with schoolwork. They were there to explain things, provide guidance, and teach. My parents, on the other hand, were perpetually tied up with work, labouring to pay off the debt they'd acquired to relocate our family to Australia. Even if they had spare time, their own education – cut short around the 6th or 7th grade – would have prevented them from helping much. This reality left me grappling with schoolwork on my own, struggling more often than not.

The handover of each graded paper from my teacher was a moment of dread. I could feel my face flushing a bright tomato red as soon as I saw my scores. Embarrassment overcame me whenever my classmates asked about my marks, and I couldn't bring myself to share. This cycle of struggle and shame gradually tricked me into thinking I was dumb.

Fortunately, in the latter years of secondary school, a chain of events began to nudge that self-demeaning belief. For one, I found someone who could help me with maths. What followed was a drastic transformation; I went from regularly failing maths to acing it, even outperforming those in the advanced levels. Around the same time, the internet was becoming a thing, and then Google came along. Together, they became my lifeline, my digital tutors.

Google became the teacher I yearned for growing up. If you've never stopped to marvel at this technology, now's the time. Think back to the world before its inception. With just a few keystrokes, this innovation offers a world of knowledge, making it possible to learn virtually anything you set your mind to. It's truly astounding. Have you ever wondered how a web search operates?

In this blog, I want to share how Google Search works, just the basics, nothing too overwhelmingly intricate. If you're a business owner or have a vested interest in digital marketing, learning how Google Search works will benefit you; it is inextricably tied to Search Engine Optimization (SEO), which is important for enhancing your visibility to prospective customers online.

Google's search engine works in essentially three steps:

  1. Crawling
  2. Indexing
  3. Serving Search Results

We will go through each step; let's start with the first one.

Crawling

Crawling is the first step in Google's search process. Google discovers new and updated content on the web, such as new websites, blogs, pages, images, videos, etc. Google uses automated bots known as "spiders" or "crawlers" (the most famous of which is called Googlebot) to crawl the web. These crawlers report back what they find to Google's servers. The data is then stored in a massive list for subsequent review. Google primarily finds new pages by following links from pages that are already known to it.

If you own a website, it's essential to ensure that Google can crawl your site effectively. If it can't, your website won't appear in search results because Google wouldn't know about its existence.

Indexing

Once a page is discovered during the crawling process, Google then examines the pages and tries to understand their content to index them. In other words, the pages are stored in Google's vast search index, which can be likened to an enormous digital library where information about all the web pages Google has discovered is stored. When indexing a page, Google's bots evaluate the content and the overall structure of the page, including text, images, videos, other media files, meta tags, and other technical aspects. The ultimate goal is to categorize the content to provide the most relevant results for each user's search query. Techniques like SEO are employed to optimize a webpage for this indexing process.

Serving Search Results

Having now learned about crawling and indexing, did it occur to you that when we enter a query into Google search, like "how rare are three rooted teeth", we aren't actually searching the web but Google's index of the web? If you use another search engine, such as Yahoo, which relies on Bing search, you're perusing another collection of pages.

So, what happens after you hit the 'Enter' button? Google searches its index to locate the most relevant pages. There could be hundreds of thousands of potential results from the search, so how does Google narrow it down to the most relevant pages to present to the user? It achieves this by asking a host of questions—over 200 of them! These include queries like, how often does this page contain your keyword? Do the words appear in the title or the URL (Uniform Resource Locator— also known as the web address)? Is this page from a reputable website, or is it of low quality? What is this page's PageRank? PageRank is a proprietary algorithm invented by Google's founders, Larry Page and Sergey Brin. It rates a web page's importance by examining how many external links point to it and how influential those links are.
Finally, Google combines all these factors to calculate each page's overall score and serves your search results. This entire process is executed in less than a second. Isn't it incredible?!

Now, to answer the query, "how rare are three rooted teeth". Can you guess how many results were found?
A hint—the answer can be found in the screenshot.

Google search results of how rare are three-rooted teeth
Google search results of how rare are three-rooted teeth

So, there you have it! A sneak peek into the fascinating inner workings of Google search. Just remember, next time you type in a query and hit 'Enter', what comes up on your screen is a result of complex, lightning-fast processes happening behind the scenes. Whether you're a student seeking knowledge, a business owner aiming to optimize your website's visibility, or a curious mind, understanding this intricate dance of crawling, indexing, and serving can give you a whole new appreciation for this incredible technology we often take for granted.

P.S. I'm one of the few who have a three-rooted molar. 😎