Can you remember a time before search engines?
Being in my late 20's, I certainly can't... Our lives are now so intrinsically linked to search engines to the point that if Google closed down tonight, the world might just fall apart tomorrow. But it wasn't always this way, of course. So, where did search engines come from? How did we get here? And what might the future hold for search engines and SEO?
The story of search engines and the history of SEO is a fascinating journey through the evolution of the internet, marking significant milestones in how we access information online. From humble beginnings to a multi-billion dollar industry, search engines have had as big an impact on the 21st century as anything or anyone else, but where can they go next? (Hint: we are going to be discussing AI)
Like all good stories, let's start at the beginning...
Search engines can be traced back to the early 1990s, when the internet was a growing network of interconnected documents, first pulled together as the 'World Wide Web' by Tim Berners-Lee.
The first search engines were pretty basic, as you'd imagine, and reliant on human-curated directories. Sites like Yahoo! started as a simple directory of websites, split into topic categories, allowing users to source the information they required from relevant websites. This directory method was fine while the internet was small and confined to a few scientists, but its limitations soon became apparent as the web expanded exponentially in no time at all.
To solve this issue, automated web crawlers were introduced, capable of scanning an entire webpage and indexing its content to be served to search engine users. WebCrawler, launched in 1994, was among the first web crawlers to provide full-text search - a significant leap forward in search technology that helped usher in the next generation of online search.
Yahoo! circa 1996, screengrab courtesy of The Web Design Museum
Google! circa 1998, screengrab courtesy of The Web Design Museum
Google's launch in 1998 was a step-change in the evolution of search engines, introducing the quite amazing PageRank algorithm on which Google has built an empire.
Developed by Larry Page and Sergey Brin while at Stanford University, PageRank was a new approach to web page evaluation. It not only assessed the content on the page but also the quality of links coming into that page to determine how reliable and relevant the content must be.
PageRank's premise was simple: a page must be important and useful to a user if other pages link to it, especially if those linking pages were of high quality themselves. This method of assessing page quality was influenced by academic citation, where a paper's importance is often judged on the number of times it is cited by others. A link to a webpage served as a recommendation, kind of like how word of mouth organically spreads.
With PageRank, Google was able to drastically improve the relevance and quality of search results. This was because PageRank leveraged the collective intelligence of the World Wide Web, making it much harder for webmasters to manipulate search rankings through simple on-page optimisations like keyword stuffing.
PageRank, combined with Google's simple and user-friendly interface, was a near-instant hit and Google was soon the number one search engine worldwide. They have held onto this market dominance by continually updating their algorithm and introducing new products into the mix. However, this core philosophy of using links as a signal of quality remains a fundamental part of Google.
The 2000s saw search engines become more sophisticated, with a focus on optimising search results for relevance and speed. Users could now receive millions of results for a query within a split second, leading to our reliance on search engines today. The classic '10 blue links' search engine results page began to evolve, providing more and more information to the user without them needing to click through to a website.
This period also marked the beginning of search engine optimisation, as webmasters and marketers began to understand how search algorithms work and how to influence their websites’ rankings.
The tactics they were able to employ were a little more basic than the SEO strategies, tools and techniques we use today. Keyword stuffing and other black hat SEO tactics have forced search engines to continually update their algorithms to ensure that their users are always served with the best content.
As search algorithms developed, so did SEO and the search engine marketing industry. The idea of SEO specialists was born and their services and expertise commanded a premium, with businesses everywhere desperate to rank at the top of Google.
Larger companies were able to employ in-house SEOs, but to service the needs of smaller companies with smaller budgets, expert SEO agencies (like us) began to spring up, levelling the search playing field against the corporate giants.
Another advancement in search engines during the noughties was increased personalisation. Huge developments in data analytics and tracking software allowed search engines to better personalise results to an individual's needs. Results could be tailored based on individual user preferences, search history and location, making search more relevant and user-friendly.
Search engines could even begin leveraging social signals within their results after the social media boom of the late 2000s. Trending topics within a user's network could be shown within search engine results pages as the mutually-beneficial harmony between organic search and social networking began to dominate the internet.
The BBC website from around June 2007, shown on a first generation iPhone, screengrab courtesy of The Internet Archive
On January 7th 2007, Apple founder Steve Jobs introduced the first iPhone to the world and things have never been the same since. Suddenly, a person's entire digital life was in their pocket and mobile network operators soon began offering mobile internet packages. These packages, along with the technologically-advanced iPhone, allowed users to access the internet freely wherever they were and mobile search demand exploded.
Search engines knew they had to adapt quickly. Users could now search for answers to their queries wherever they were, leading to massive growth in search engine use. Gone are the days of having to sit at a computer to Google a problem; now you simply unlock your smartphone and search away. Mobile internet speeds improved as users' patience and concentration spans reduced, meaning Google and the other major search engines had to serve results even faster than before.
Out of this arose the idea of mobile-first indexing. Large desktop websites would not fit onto a mobile screen and so created a poor experience for the user. Google is all about giving its users the best experience possible, so it started giving ranking priority to websites that were fully optimised for mobile devices. There was a brief era of websites providing separate desktop and mobile versions of their sites, but this was soon dropped in favour of responsive sites which could adapt to the needs of any device.
Soon after the mobile search revolution came the voice search revolution. Digital assistants like Siri, Alexa and Google Assistant were included in new smartphones and home devices, allowing users to ask questions out loud and receive answers promptly in text or audio form. Being able to simply speak rather than type was a hit with users worldwide, saving all-important milliseconds in our quest for information.
The voice technology boom meant that search engines again had to adapt and consider conversational search. The way people search by voice is entirely different to by written word, often using more complex and natural language.
This shift towards natural language queries requires search engines to understand and process spoken language - a complex challenge that continues to evolve.
Search engines have come a long way in three decades. Could the webmasters of the early 90s have predicted the search landscape we have today? Could they have foreseen mobile search, knowledge packs, voice search, featured snippets, image search, maps and everything else? Absolutely not. As such, predicting the future of SEO and search engines is pretty difficult.
One thing we know for sure is that AI is set to play a pivotal role in the evolution of search engines in the decades to come. AI can analyse vast amounts of data to provide more accurate, personalised search results. Machine learning algorithms can understand user intent more deeply, making interactions with search engines more intuitive.
AI is already changing the game in SEO, allowing SEO professionals to reduce the time it takes to perform mundane tasks. Software like Chat-GPT is great for SEO content ideation and review, but human-written content is likely to always outperform anything written entirely by AI due to Google's E-E-A-T guidelines.
Furthermore, the integration of AI in search engines with other emerging technologies like augmented reality (AR) and virtual reality (VR) could transform search into a more immersive experience. Imagine putting on VR goggles to explore a virtual representation of a search query or using AR tech like Apple Vision Pro to overlay information onto the real world.
With AI, the possibilities really are endless. In the real world, we're already seeing AI help scientists with life-saving research, solve decades-old conundrums with regard to nuclear fusion energy and even drive our cars for us.
How AI will fit into search engine land is still to be fully-determined, with Google rolling out countless algorithm updates to combat the threat of AI-generated content but also trialling its own AI software like Gemini. OpenAI - the AI research organisation behind ChatGPT - is even developing a search engine of its own, which could yet be a threat to Google's search dominance.
All in all, the likelihood is that what you see when you input a query into a search engine today will be entirely different to what you'll see for the same query in five years' time.
The short history of search engines has been quite a ride. The rise of search engines like Google personifies human ingenuity and our relentless pursuit of better, faster routes to information.
As we look to the future of search, it's clear that search engines will continue to evolve, driven by AI and other technologies yet to be imagined. The challenge for developers, users and regulators will be to ensure that this evolution benefits society by enhancing our access to information and solving human problems around the world.
Neil Bates
SEO Account Manager
Ey up! My name’s Neil – great to meet you. I was born and bred in the East Midlands and have six years’ industry experience in digital content creation, plus a further three years of many forms of writing for my BA English degree. I’ve recently joined The Digital Maze as SEO Account Manager and would be delighted to talk to you about your SEO needs.
Sign up to The Digital Maze Newsletter
