audits.com Open in urlscan Pro
172.67.186.131  Public Scan

Form analysis
1 forms found in the DOM

POST

<form id="brxe-iucmff" data-script-id="iucmff" class="brxe-form" method="post" data-element-id="iucmff">
  <div class="form-group" role="group" aria-labelledby="label-111987">
    <input id="form-field-ce874c" name="form-field-111987" aria-label="Email" type="email" value="" placeholder="email@website.com" required="">
  </div>
  <div class="form-group captcha">
    <div class="form-group recaptcha-error">
      <div class="brxe-alert danger">
        <p>Google reCaptcha: Invalid site key.</p>
      </div>
    </div>
    <div id="recaptcha-iucmff" data-key="6Le_BKooAAAAAJmtfFxfQVVxsSU59WAY3zUx-23q" class="recaptcha-hidden"></div>
  </div>
  <div class="form-group submit-button-wrapper">
    <button type="submit" class="bricks-button bricks-background-dark">
      <span class="text">Subscribe</span><span class="loading"><svg version="1.1" viewBox="0 0 24 24" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
          <g stroke-linecap="round" stroke-width="1" stroke="currentColor" fill="none" stroke-linejoin="round">
            <path d="M0.927,10.199l2.787,4.151l3.205,-3.838"></path>
            <path d="M23.5,14.5l-2.786,-4.15l-3.206,3.838"></path>
            <path d="M20.677,10.387c0.834,4.408 -2.273,8.729 -6.509,9.729c-2.954,0.699 -5.916,-0.238 -7.931,-2.224"></path>
            <path d="M3.719,14.325c-1.314,-4.883 1.969,-9.675 6.538,-10.753c3.156,-0.747 6.316,0.372 8.324,2.641"></path>
          </g>
          <path fill="none" d="M0,0h24v24h-24Z"></path>
        </svg>
      </span> </button>
  </div>
</form>

Text Content

Skip to main content Skip to footer
Open Nav
WE DO SEO
COMPANY
 * About
 * Contact

 * About
 * Contact
 * * 
   * 
   * 



Growth & Opportunity
 * SEO Discovery Audits
 * International SEO
 * SGE / AI SEO
 * Increase Organic Revenue
 * Share of Voice (SOV) Audits
 * Resolve Google Algorithm Issues

 * SEO Discovery Audits
 * International SEO
 * SGE / AI SEO
 * Increase Organic Revenue
 * Share of Voice (SOV) Audits
 * Resolve Google Algorithm Issues
 * * 
   * 
   * 



Technical Consulting
 * SEO Audits
 * Ecommerce SEO Audits
 * Financial Services SEO
 * Marketplace SEO Audits
 * SEO Product & User Experience (UX) Audits
 * Domain & Website SEO Migrations Audits
 * SEO Technology Selection

 * SEO Audits
 * Ecommerce SEO Audits
 * Financial Services SEO
 * Marketplace SEO Audits
 * SEO Product & User Experience (UX) Audits
 * Domain & Website SEO Migrations Audits
 * SEO Technology Selection
 * * 
   * 
   * 



Training & Development
 * Team Coaching
 * Bring SEO In-House
 * SEO Channel Alignment Audits

 * Team Coaching
 * Bring SEO In-House
 * SEO Channel Alignment Audits
 * * 
   * 
   * 


Expertise
 * SEO Consultancy
 * SEO Knowledge Base
 * Insights
 * Fractional SEO Consultancy

 * SEO Consultancy
 * SEO Knowledge Base
 * Insights
 * Fractional SEO Consultancy
 * * 
   * 
   * 



SEO TOOLS
 * PDF Analytics
 * Google’s Knowledge Graph Search Tool

 * PDF Analytics
 * Google’s Knowledge Graph Search Tool
 * * 
   * 
   * 




 * WE
   * About
   * Tools
     * Google’s Knowledge Graph Search Tool
     * PDF Analytics in Google Analytics
   * Contact
 * DO
   * Growth & Opportunity
     * SEO Discovery
     * International SEO
     * SGE / AI SEO
     * Increase Organic Revenue
     * Resolve Google Algorithm Issues
     * Share of Voice (SOV) Audits
   * Technical Consulting
     * SEO Audits
     * Ecommerce SEO Audits
     * Financial Services SEO Audits
     * Marketplace SEO Audits
     * Domain & Website SEO Migrations Audits
     * SEO Technology Selection
     * SEO Product & User Experience (UX) Audits
   * Training & Development
     * SEO Channel Alignment
     * Bring SEO In-House
     * SEO Training & Coaching
 * SEO
   * SEO Consultancy
   * Fractional SEO Consultancy
   * SEO Knowledge Base
   * Insights
 * * 
   * 
   * 



Home • SEO • Insights • The History of Search Engines


THE HISTORY OF SEARCH ENGINES

CARL HENDY

 * 
 * 

Updated on July 3, 2024
32 min read

ON THIS PAGE

On this page
 * Before search engines
   * Tim Berners-Lee
   * Open Directory Project
   * Archie wasn’t the first web search engine
   * Money Gets Involved
 * 1995 – The dawn of search engines
   * Archie wasn’t the first web search engine
 * Who is Robin Li? (Li Yanhong / 李彦宏)
   * He kept his head down
   * Why don’t you Ask Jeeves?
 * A monopolistic giant is born
   * Google
   * SEO made Google a giant
 * The stolen trillion dollar idea
   * Searching for profit
   * Google gets taken to court
 * What’s it called again?
   * Microsoft’s struggle for dominance
   * Any ideas?
 * Yandex (Яндекс in Russian)
   * Searching in Russia
 * Baidu – The Google of China
   * An unstoppable force
 * Baidu’s competition is from within China
 * The crazy world of DuckDuckGo
   * DuckDuckGo After Privacy
 * Mobile Search overtakes Desktop
   * Voice Search
 * Glasses & Wearable Tech
 * Machine Learning
 * The Future of Search: Generative AI
 * The AI Wars Wage On
 * Will anyone overtake Google?
 * Search Engines Launched
 * References

LET'S WORK TOGETHER?

Discuss Project



BEFORE SEARCH ENGINES

The internet existed for many years before the first web page came online. It
was initially a US military project at DARPA, then used as a way of sharing
academic and scientific knowledge. The internet as we know it, runs on TCP/IP, a
technology invented in 1974 by Bob Kahn and Vint Cerf (who now works for
Google).[1]


TIM BERNERS-LEE

It was the English scientist, Sir Tim Berners-Lee, who invented the World Wide
Web in 1989 while working at CERN in Switzerland. It used a technology called
Hypertext Transfer Protocol (HTTP) that transmitted data over TCP/IP, which is
why all URLs start with “HTTP” to this day. To make HTTP easier to interface
with, Berners-Lee built the world’s first web server and web browser to navigate
it in 1990. He also invented HTML (based on CERN’s SGML markup) for formatting
text-based content, distributing the technology outside of CERN in 1991.
Berners-Lee declared that the technology must remain freely available, with no
patents or royalty-fees, to be accessible to everyone.

Usenet newsgroups were how internet users communicated, a decade before the web
was invented. Even after the broad adoption of Sir Tim Berners-Lee’s technology,
web pages were often shared and linked to within special-interest newsgroups.
Some users began creating web pages that collated the URLs shared in newsgroups,
into catalogues or directories. Rather than having to download every message in
a newsgroup and searching for information or URLs within them, users could now
visit a directory/portal and navigate through the categories to find the
relevant web pages.

Sir Tim Berners Lee


OPEN DIRECTORY PROJECT

One of the most well-known directories was DMOZ (or the Open Directory Project),
created in 1998 by two engineers at Sun Microsystems. The directory became the
default way of finding information on the web for many, with a search function
for faster directory navigation.

DMOZ was acquired by Netscape later that same year, having indexed (listed)
around 100,000 URLs. A year later DMOZ had over a million URLs listed, and it
peaked at over five million URLs, before closing in 2017. The directory’s
objectives became blurred when AOL acquired Netscape, and the cost of running it
for free came into question.

Volunteer editors of the directory felt betrayed, becoming unpaid workers of
AOL, rather than a noble, free and open initiative. Commercial content creators
such as CNN were given editorial rights to add/edit/delete entries, putting the
impartiality of the project in question. Editors were found to be selling
inclusion in the directory, after a DMOZ listing was rumoured to increase a
website’s ranking in search engines.AOL failed to resuscitate DMOZ several times
and eventually took the website offline.

A directory based on the original DMOZ database and maintained by former DMOZ
editors, still exists today at Curlie.org.


ARCHIE WASN’T THE FIRST WEB SEARCH ENGINE

A pub quiz will tell you that Archie was the first search engine, which is
technically correct. But Archie, built in 1987 at McGill University, was
designed to search for files on the internet (FTP servers), not for content on
the World Wide Web.

W3Catalog (originally called “Jughead”) was later launched in September 1993 by
Oscar Nierstrasz at the University of Geneva. The service mostly took existing
lists/catalogues of web pages and made them searchable in a standardised format.

Aliweb (Archie Like Indexing for the Web) is widely considered to be the first
web search engine. Launched in November 1993, Aliweb allowed webmasters to
submit their web pages and enter the relevant keywords and descriptions for
these pages. The search engine was largely forgotten though, with internet users
at the time, still mostly preferring to navigate websites using directories,
lists and catalogues.


MONEY GETS INVOLVED

WebCrawler was the first search engine to be widely used, as well as the first
to fully index the content on web pages, making every word and phrase
searchable. It was developed at the University of Washington and launched in
1994, the same year as Lycos from Carnegie Mellon University. Both WebCrawler
and Lycos became commercial ventures, with WebCrawler supported by two primary
investors, one being Microsoft co-founder Paul Allen.

Lycos heavily invested in their brand, with TV ads featuring their iconic black
labrador dog, as well as hiring a vast team of volunteer and paid Editors for
their web directory.

Yahoo Search Engine


1995 – THE DAWN OF SEARCH ENGINES


ARCHIE WASN’T THE FIRST WEB SEARCH ENGINE

Two years after Aliweb, search engines became mainstream and big business.
Excite and AltaVista both launched in 1995, along with the less well-known
MetaCrawler, Magellan and Daum. But the most significant success was Yahoo,
founded by Jerry Yang and David Filo.

“Yahoo!” started as a traditional web directory in 1994 by two Stanford
University graduates, then launching a search engine in 1995. To the annoyance
of their lesser-known rivals, Yahoo didn’t build any significant new technology.
They bought and borrowed third-party tech, until the acquisition of Inktomi (a
search engine for hire) in 2002. The success of Yahoo was all packaging, with a
fun brand and a user-friendly interface.

Internet-connected computers started to become widely accessible in schools,
libraries and homes across the globe. A new generation began using websites more
than books, and search engines more than web directories. Yahoo, AltaVista and
Lycos dominated, with significant investment propping up the loss-making sites. 


WHO IS ROBIN LI? (LI YANHONG / 李彦宏)

Robin Li


HE KEPT HIS HEAD DOWN

“Robin” Li Yanhong is mostly an unknown in the western world, but one of the
richest men in China, with an estimated net worth of nineteen billion dollars.

His parents were factory workers, with four other children to care for. They
helped Li get into Peking University, where he studied Information Management.
Li then went on to study at the University of Buffalo in New York, where he
earned a doctorate in Computer Science. After leaving university, Li joined a
New Jersey based division of Dow Jones, where he built software to manage the
online edition of The Wall Street Journal.

In 1996, Li created and patented a system called RankDex, for ranking the
importance of web pages in a search result. For the first time, it used “link
analysis” to determine the importance of web pages by the number of other pages
linking to them.

RankDex is the basis of every major search engine’s ranking algorithm today,
predating Google’s “PageRank” by two years and being referenced in Larry Page’s
first patent. Without RankDex, search engines may still be using keywords, not
links, as their primary ranking factor today.

Shortly after welcoming in the new millennium, Robin Li and Eric Xu co-founded
and incorporated Baidu, now China’s largest search engine. Just as PageRank is
the soul of Google, RankDex is the soul of Baidu and the father of modern search
engine ranking algorithms.


WHY DON’T YOU ASK JEEVES?

Just as Boo.com was “before its time” in e-commerce (3D rotating product images
in 1998?!), Ask Jeeves was before its time in search. The search engine launched
in 1997, with a unique ability to answer questions.

Up until now, users had to carefully think about which “keywords” to search for
on AltaVista, Yahoo and Excite, to get a useful page of results back. A typical
search engine would give equal weighting and importance to each word in a
question, often returning irrelevant results. Ask Jeeves was able to extract the
important words and primary intent of a question, yielding much more relevant
results. It became hugely popular with the growing number of non-techies surfing
the web, who were not used to thinking like a computer.

The company was acquired by IAC (Match.com) in 2005 but struggled to compete
against larger rivals such as Google. It rebranded to Ask.com in 2006, to save
on the royalty fees to P.G Wodehouse’s estate (Jeeves was a butler character in
one of his books) and appear more modern[2]. In 2010, Ask made its search team
redundant and outsourced to another search engine provider.

The search engine technology Teoma that they acquired, struggled with ranking
and indexing relevant pages on an ever-expanding web. Revenue generation also
seemed to take priority over User Experience. Adverts consumed the search result
pages when Google at the time had a clean, fast and relatively ad-free
appearance. Any computer-related search query on Ask Jeeves would return a page
covered in Dell adverts, making the organic results challenging to find.

A year after throwing in the towel on search technology, Ask’s CEO, Doug Leeds,
said they still provided search to over 100 million people every month. Perhaps
if Ask kept developing their technology, they’d be a leader in voice search
today, where question-based searches are key.


A MONOPOLISTIC GIANT IS BORN

Google HQ Offices


GOOGLE

Originally named “Backrub” for its link-based ranking algorithm, Google was
founded by Larry Page and Sergey Brin at Stanford University. A play on the term
“googol” (10 to the power of 100), the company had big ambitions from the
outset. Page and Brin got a seed investment of $100,000 by Sun Microsystems
co-founder, Andy Bechtolsheim before the company was even incorporated. The seed
funding round raised a total of $1 million, the majority of which came from
three key investors. One of those investors was Jeff Bezos, the founder of an up
and coming online bookstore called Amazon.

Google came relatively late to the search party, building upon several existing
ideas in 1996 and launching at the end of 1997. Other search engines were
starting to suffer from spam and relevancy issues, which was to become Google’s
golden bullet and the secret to their success.

Photo Credit: Steve Jurvetson – Google’s first server rack


SEO MADE GOOGLE A GIANT

SEO started as a hobby by inquisitive webmasters, who wondered how search
engines ranked the order of web pages in their results. Older search engines
heavily relied on trusting the “keyword” meta tag on a web page, to tell it
which keyword searches to list the page for. This required a level of trust in
webmasters, that they would only enter relevant keywords. Newer search engines
used the content of a web page to determine its importance, often counting the
number of times a keyword appeared on the page. Both of these were exploited,
first for fun and then for profit. Websites would list every conceivable keyword
in their meta tags and content, repeating the same keywords multiple times.
These ugly lists would often be hidden at the bottom of the page or in the same
colour font as the background, rendering them invisible.

Baidu founder Robin Li was one of the first to tackle this problem with RankDex,
which used links from other websites as a vote of importance. Newer search
engines moved to this link based citation model but counted all links as equal.
Some webmasters exploited this by creating web pages with thousands of links on
them, all pointing to their other websites.

Larry Page addressed this issue with his PageRank formula, which not only looked
at how many backlinks a page had, but how important those linking pages were. A
link from the BBC was more important than a link from a spammy hub page, as
thousands of other sites would be linking to that BBC page and many of those
sites would themselves have thousands of relevant links. This ripple effect was
difficult to manipulate by webmasters, as every link required many levels of
other links pointing at them, to help give it the authority to push a web page
up the rankings.[3]

Spam-free results became increasingly important, as the internet was used for
more commercial purposes. Searches for academic papers with one or two results
were replaced by tens of thousands of searches a month for [car insurance],
[designer clothing] and [credit cards], each with hundreds of thousands of
competing pages. Google had the magic formula for weeding out the spammy SEO
keyword pages and ranking more trustworthy websites first.


THE STOLEN TRILLION DOLLAR IDEA


SEARCHING FOR PROFIT

If the PageRank ranking formula was Google’s golden bullet, “AdWords” was the
gunpowder that fired it. The computing power needed to crawl the growing web and
build Page’s elaborate link graph didn’t come cheap, and all of the investors
were expecting a healthy return on their investment.

Advertising was a complex problem in 1999. Search Engines needed to place banner
adverts on their search results to pay for their infrastructure, but the banner
ads detracted users and slowed down their result pages. Ask Jeeves and Yahoo
started to experiment with paid inclusion, where websites could pay for a higher
ranking in search results or a guarantee to be listed.

GoTo.com was a minor player in the market, having bought one of the oldest
search technology companies, World Wide Web Worm. Their skill was in monetising
searches, creating the first ad auction system in 1998, where businesses would
bid on keywords. Rather than paying for banner views, companies would pay
anything up to $1 per click on their premium search result listing. The company
rebranded to “Overture” in 2001 and started selling its advertising solution
into MSN and Yahoo, monetising hundreds of millions of searches a day. This
vastly outweighed the income from their own GoTo search engine but also gave
them the capital to acquire competitor search engines, AltaVista and AllTheWeb.
Other search engines were being bought for status and traffic; these purchases
were purely business, increasing Overture’s ad platform coverage and keeping
100% of the revenue.


GOOGLE GETS TAKEN TO COURT

Google AdWords launched in October 2000, initially on a CPM (cost per thousand
impressions) basis and transitioning to PPC (pay per click) in 2002. The PPC
model was remarkably similar to Overture’s patented advertising platform. Later
that year, Overture filed a lawsuit, claiming that Google had stolen their
proprietary technology.

In 2003, Yahoo bought Overture for $1.63 billion. It secured the ad platform
that drove most of Yahoo’s revenue, as well as increasing their market share
with the portfolio of search engines that Overture had previously acquired. Then
in 2004, Google settled the lawsuit with recently acquired Overture, offering
Yahoo 2.7 million GOOG shares as compensation. At today’s share valuation and
accounting for Google’s 2014 share split, these shares would be worth $6.9
billion.[4]

Yahoo struggled over the years to come, making poor investments, acquisitions
and business priorities. Blogging, Photo Sharing and Auction websites were
acquired and then left to whither while competing websites became billion-dollar
businesses. Search users left in droves, as did the partner search engines that
were previously powered by Yahoo. Finally, in 2017, the company was acquired by
Verizon for just $4.5 billion.


WHAT’S IT CALLED AGAIN?

Bing Search Engine


MICROSOFT’S STRUGGLE FOR DOMINANCE

Nothing symbolises panic and indecisiveness, better than Microsoft’s fall into
the search engine world. MSN Search launched in 1998 in the wake of Google when
Microsoft’s Windows operating system was used by over 90% of Americans. MSN
initially used Inktomi (a search engine for hire) search results, which also
powered Yahoo. They tried to stand out by blending Looksmart results and then
AltaVista results into their service, with limited success.

In 2004, Microsoft finally gave search the investment that it needed, building
in-house search engine technology and putting it live in 2005. MSN’s most
significant success at the time was in the B2B world that Microsoft was
comfortable in. They offered their technology to other search engines, internet
providers and portals, gaining market share and a cut of advertising revenues.

Just as MSN Search started to pick up momentum (helped by it being the default
homepage on millions of Internet Explorer browsers), Microsoft committed
harakiri by renaming the service “Microsoft Live” in 2006. The decision behind
this is vague and perplexing, other than reinforcing the company’s “Windows”
brand and making it seem modern. Just one year later, the company rebranded its
search engine again, removing the “Windows” reference and calling it “Live
Search”.


ANY IDEAS?

Former search industry leader, Danny Sullivan:

“Why not go back to MSN or Microsoft Search? Why not change it – or will it
change?”

Microsoft’s Kevin Johnson replied:

“There’s an opportunity for us to fix those brands. We acknowledge that we need
to get that fixed. If you have suggestions, we’ll take them.”

In 2009, Microsoft announced that Live Search would be rebranded one last time,
to Bing. The brand changes and search result quality of Microsoft became a point
of ridicule over the years, but Bing became a serious contender for Google in
the years that followed. It rarely gained more than a 10% market share in the
US, but did compete with, and often beat, Google on blind result comparisons and
user satisfaction studies. The same year that Bing launched, Microsoft signed a
deal with their closest search engine rival, to power Yahoo’s search results. To
this day, Yahoo is still “Powered by Bing™”.[5]


YANDEX (ЯНДЕКС IN RUSSIAN)

Yandex HQ


SEARCHING IN RUSSIA

Yandex went public in 1997 and became the market leader in Russia in 2001. Other
Russian search engines existed as well at the time, such as Aport.ru and
Rambler.ru, but they later pivoted into a marketplace and media portal
respectively.

Yandex’s dominance over Google in Russia, stemmed from it understanding the
morphology of Russian language better, resulting in more precise search results
for Russian websites. The company then released competing Maps and
Market/Shopping products, which were also better adapted to the needs of Russian
users. Google didn’t seem very interested in the Russian market at first,
allowing Yandex to become so dominant. As Yandex grew, search became part of a
much larger ecosystem of products for the company.

However, Russian internet users have been using two search engines for several
years. Thanks to the spread of Android and Chrome, Google has managed to win
back a large share of search, in particular, among young and more advanced
users. At one point, Google even surpassed Yandex for mobile search, but then
lost its lead again.

Taking into account the current global trend of tightening regulation over
nationally important segments of the internet, I would predict Yandex acquiring
an even greater market share over Google.

Some of the world’s best Machine Learning and speech recognition technologies
are being developed in Russia. Yandex has its own voice assistant, which is also
considered to be one of the best. Of course, in 5-10 years we will use search
differently, and I am sure that in terms of technology, Yandex will be one of
the leading players, in Russia and worldwide.

I think the secret to Yandex’s success is simple: they make a first-class
product, that is fully adapted to the habits and needs of the local user. Yandex
became part of Russian internet culture — not only the search engine, but also
its Map, Market/Shopping, Taxi, and Voice Search products.

Reported by
Konstantin Kanin

Reported by Konstantin Kanin


BAIDU – THE GOOGLE OF CHINA

Baidu Headquarters


AN UNSTOPPABLE FORCE

Baidu had an impressive pedigree from the day it launched in 2000, as a Chinese
focussed search engine. The company was co-founded by Robin Li, who inspired
Google’s Larry Page and his PageRank patent. Li’s RankDex idea of scoring pages
based on their link profile was published back in 1996, but now Li was playing
catch-up to an already popular Google.

The search engine’s growth was powered by ad revenue and Chinese government
compliance. Baidu launched its auction-based Pay Per Click ad platform before
Google but did not face the same patent infringement lawsuit from Overture that
Google did. It also complied with requests to censor keywords and news sources
by the Chinese government, which Google either refused or resisted. Highlighted
in 2009 by a leaked internal document, Baidu had a long list of keywords, topics
and websites that should return no results. It included news websites that were
critical of the government, civil rights, protests, ethnic conflicts, democracy
and the names of China’s leaders.[6] This was a necessary evil, for the search
engine to have any chance of succeeding within “The Great Firewall of China”.

Google began its Simplified and Traditional Chinese search engine in the same
year as Baidu launched. The Chinese government intermittently blocked the .com
website, so Google launched a censored version of its search engine at
Google.cn. The relationship continued to be rocky though, and Google stopped
censoring in 2010, after being attacked by hackers linked to the Chinese
government.[7] It struggled to maintain a sizeable market share, with the
website repeatedly blocked and Chinese users preferring native alternatives such
as Baidu.


BAIDU’S COMPETITION IS FROM WITHIN CHINA

Baidu has been the dominant player in China for a long time. But before Google
left the Chinese market, it had almost a 50% market share. Sogou and 360 Search
are also popular in China, getting most of their traffic by creating their own
web browsers.

Baidu is now facing a new challenge from Toutiao Search, which is rising in
popularity. It’s still far away, but could be a challenger to Baidu in the next
3-5 years.

China has a very different internet environment, from both a political and
cultural point of view. Thus, it’s very difficult for international players to
survive in the market, such as Google. When competing with domestic players,
Baidu did have many advantages over their US rivals.

Voice search is starting to become more prevalent in China, but with the Baidu
Smart Speaker, not Amazon Alexa or Google Assistant.

Reported by Allen Qu (渠成)


THE CRAZY WORLD OF DUCKDUCKGO


DUCKDUCKGO AFTER PRIVACY

Launched at the start of 2008, it was hard to take DuckDuckGo seriously. The
search engine was founded by Gabriel Weinberg, who had recently finished his
Masters at MIT and failed at launch a new social networking start-up. There were
no financial backers at first, and the search results were mostly tied together
from the APIs of other search engines. Why would people use it? The USP was
privacy – your searches weren’t tracked or recorded like on most other search
engines.

Two years prior, AOL publicly shared a data file with three months worth of
search history, for research purposes. It included 20 million search queries
from 650,000 users. While the searcher’s account details weren’t shown, they
were given a random ID that allowed researchers to group searches by individual
users. Some searches included PII (Personally Identifiable Information) and the
identity of some users was revealed. Concerns over search privacy stopped
becoming a paranoid techie issue and entered the mainstream conscience.

Over the next few years, DuckDuckGo started to attract a cult following of
privacy-concerned users. Google and Facebook were beginning to show their cards,
as collectors and sellers of personal data. In 2011, Union Square Ventures made
an angel investment in DDG, “Not because we thought it would beat Google. We
invested in it because there is a need for a private search engine. We did it
for the Internet anarchists, people that hang out on Reddit and Hacker News”. By
2012, the search engine announced it was serving 1.5 million searches a day and
made $115,000 from privacy-friendly advertising.

A year later, The Guardian and The Washington Post newspapers published an
expose on an NSA operation called PRISM. Powerpoint slides leaked by Edward
Snowden show how big tech companies were handing over user data and search
history to US Intelligence. It stated that “98% of PRISM data is sourced from
Yahoo, Google, and Microsoft”. DuckDuckGo hit 4 million searches a day that
year.[8]

There was enough demand that the Firefox and Safari browsers gave DuckDuckGo as
a default search engine option in 2014. Something that Google only followed suit
on in 2019.

DuckDuckGo now answers 1.8 billion searches a month and has a US market share of
1.24% (Yahoo has 3.65%).[9]


MOBILE SEARCH OVERTAKES DESKTOP

People were able to search the web on their mobile device before Google was even
invented, and the search giant themselves enabled their “WAP” based site for
mobile users in 2000. WAP phones transitioned into “Feature Phones” such as the
Blackberry, with pre-installed apps for searching and awkward navigation using a
physical phone keypad or a “trackball” if you were lucky. Each device maker and
telecoms company had their own deals for pre-installing a search engine on their
phones, in exchange for a revenue share or contract deal.

Smartphones were the catalyst to a truly unrestricted search and browsing
experience, putting a small computer in everyone’s pockets. But it also saw the
collapse of a diverse phone and software market, where dozens of hardware and
software manufacturers competed. The once-powerful Nokia and Motorola lost their
edge and eventually got acquired by Microsoft and Google respectively. Microsoft
couldn’t get a foothold in the phone market, losing out on hardware (Nokia),
software (Windows Mobile) and search (Bing) revenue.

This moved the power of which search engines people used, even further into
Google’s favour. Apple (iPhone) and Google (Android) were the kings of mobile.
Most Android smartphones had Google Chrome pre-installed, and the search engine
defaulted to Google. To make matters worse for Bing and Yahoo, Google signed a
deal with Apple to make their search engine the default on iPhone’s Safari
browser as well, reportedly paying $9 billion a year for the privilege.[10]

In 2015, Google announced that searches on mobile devices had outnumbered
desktops for the first time, in 10 countries, including the US and Japan.[11]
This trend continues today, with mobiles and tablets becoming the primary search
devices in people’s homes.

From 2016 onwards, Google then announced its “mobile-first index” which meant
that websites were crawled and then subsequently index content based on mobile
versions.


VOICE SEARCH

Voice Search, launched on Google search in 2012, was at the time framed as “the
next phase of search”. The method allows users to ask their questions to a
“Smart Device”, instead of typing the keywords into a web browser. This can
range from a variety of devices such as smartphones, home devices such as Amazon
Alex or via desktop devices directly via the means of AI assistants such as
Cortana or Siri. 

There was a lot of hype in the years following this, and voice search
technologies have become more advanced and accessible, with more languages being
made available to more people around the world. Due to this, usage has been on
the rise steadily over the years with the speech recognition market projected to
be valued at over $30 billion in 2030.

From a user behaviour standpoint however, voice search is unlikely to usurp
traditional search methods as in most cases, voice search assistants return
single answers to questions where users may want to explore a variety of
answers. This is suitable for queries such as “what is the capital of France”,
however for more open and subjective queries, users expect more information from
their search engines.

Despite growth in usage and availability, voice search does remain somewhat of a
novelty in answering closed questions such as the above, getting the weather
forecast, playing games or getting the device in question to play music or the
radio.

From a Microsoft standpoint, the Cortana voice assistant has struggled to make a
market outside of the Windows desktop app, with less than a 1% market share on
mobile devices in most countries.

Amazon is the surprise winner in the smart speaker battle, with 65% market share
in the US alone. This beats Google Home’s 24% market share. While Amazon does
own a search engine called. A9 (run by former WebCrawler and AltaVista execs),
the technology focuses on product and enterprise search. Instead, Amazon Alexa
web searches are powered by Bing, giving Microsoft an upper hand over Google for
the first time. Sometimes it pays to be 2nd best, especially when a
trillion-dollar company is an arch-rival of the top search engine.

The problem with voice search is that no matter how realistic the synthetic
voices sound, nobody wants to hear a computer read the web pages of the top ten
search results to them. Voice search will remain a mostly one-way conversation,
used to answer simple questions, carry out tasks and sometimes complete basic
transactions. They say that a picture is worth a thousand words – Alexa and
Google would take roughly 10 minutes to say those words compared to the seconds
needed to skim-read a web page and identify the information required.


GLASSES & WEARABLE TECH

Google Glass was a massive flop, largely down to marketing and PR. They forgot
that they were creating a fashion/lifestyle accessory that everyone will put on
their face, not just a clever piece of tech. Fashion is driven by influencers
that people aspire to be. Nobody aspires to be the tech blogger Robert Scoble or
that guy who still lives in his parent’s basement. By distributing the preview
devices to Silicon Valley keyboard warriors, which would ordinarily review
Google’s software products; the company had an image problem from the outset.
Overweight creepy Robocop impersonators aren’t going to sell the device, as well
as Beyonce or David Beckham would. Don’t worry though; you’ll know if they’re
photographing you, as they have to wink to take a picture. wink wink mmm… RAM.

But the wearable tech world is still booming, and smart glasses aren’t dead yet.
Google Glass is finding a new market in the B2B space, helping warehouse workers
locate products and doctors to look up patient data. New brands are popping up
such as North and Lance, which would know how to get an Instagram celeb
on-board. Amazon is building their own Glass prototypes, with Jeff Bezos’ black
book of Hollywood A-listers to support it.

Smart glasses or contact lenses are the sensible next step for search.


MACHINE LEARNING 

Search engines have been using some form for many years now, with Google
announcing its intention to become a “machine learning first company” back in
2016. Google AI was launched in 2017, and following this, the BERT algorithm was
launched in 2018.

BERT and other machine learning models intend to help understand the search
intent and meaning behind the queries of the search more accurately. This saw a
big move away from the traditional search results methods of matching keywords
in results in accordance with queries to a model based on anticipation of what
the user is searching for. This can be based on the search behaviour and
patterns the user has followed in the past.

This then developed further into Google’s Multitask Unified Model (MUM) which
was launched in 2021 and was framed as a major upgrade to BERT. MUM allowed
users to explore their queries further with Google providing follow-up query
prompts that saw search engines move into a more solely semantic direction. 

Queries such as “where to visit in Japan” resulted in answer box snippets being
pulled from web resources with the additional option of follow-up long-tail
questions to develop journeys further. “Where to visit in Japan” may have been
followed up with additional query prompts such as “Where to visit in Japan in
autumn” and “cultural nuances to remember when visiting Japan” or even “when is
the best time to visit Kyoto”, refining the user journey on a continual basis.

Overall, machine learning driven chatbots and answering services are likely to
play a more significant part in everyone’s lives in the future. Financial
institutions are already using technology to replace their human personnel, for
factual, transactional or numerical tasks (checking balances, opening accounts,
overdraft increases, credit checks and chargebacks).

When it comes to search engines, financial aspects such as intelligent flight
and credit card comparison services, served directly on the search results have
been a feature in recent years. It’s only a matter of time before search engines
take over the entire transaction process from the SERP (Search Engine Results
Page). The user will be saved the hassle of navigating through and ordering on a
third-party website. The products, images, content, shopping cart and checkout
can all be served on the SERP. Search engines would get a cut of the overall
transaction, instead of just a dollar for the click. Users feel safer with their
money stored and managed in a “Google Wallet”.

Overall, this sets the stage for the current and future stage of search:
generative AI.


THE FUTURE OF SEARCH: GENERATIVE AI

Large language models (LLMs) have been around for some time, with GPT1 launching
in 2017 by AI company OpenAI. The GPT3.5 version, which exploded into the global
public consciousness in the form of its chatbot ChatGPT in late 2022, saw a lot
of disruption and led to an immediate AI boom spurring the release of
competitors such as Microsoft Copilot (later deployed in Bing), Elon Musk’s Grok
(used on X, formerly Twitter) Baidu’s Ernie Bot and Google Gemini. 

ChatGPT, with its ability to clearly and coherently (although not always
accurately) respond to in-depth complicated questions, troubleshoot coding,
brainstorm ideas, produce content and write and debug computer programs among
other things, saw Google announce a “Code Red” in response to the commercial
threats posed by such an advanced chatbot.

Following this, Google then announced its response to this in the form of SGE
(Search Generative Experience, later known as AI Overviews) which used its own
AI chatbot, Gemini, in May 2023. SGE/AI Overviews saw the insertion of
AI-generated responses in search results alongside traditional links. These AI
responses typically occupy a heavy amount of real estate at the top of the
search results and intended to pick up where BERT and MUM left off by providing
AI-generated overviews to certain queries with the option for users to explore
more natively within the search results. 

Upon launch in both testing and to wider users (in the US) in spring 2024, AI
Overviews have brought on a lot of criticism in terms of accuracy of answers and
usefulness in general, with various examples of dangerous and misleading
content. The feature is clearly still in its infancy, and it is unclear what
form it eventually takes as Google grapples with accuracy concerns as well as
user engagement with AI Overviews in general.

Bing also followed suit in the generative AI department with the incorporation
of Microsoft Copilot in July 2023, and Chinese search giant Baidu’s Ernie Bot
was given the green light in October of that year to paying users.

Despite the aforementioned numerous concerns over accuracy and AI-generated
answers in search engines as well as the potential for them to take away traffic
from businesses and other online content creators, it is clear that AI is here
to stay and will play a key role in the evolution of search engines moving
forward. 


THE AI WARS WAGE ON

OpenAI’s GPT4 was released in March 2023, and search engines have had to play
continual catch-up in the AI wars to remain competitive. Baidu continues to make
strides with Ernie Bot, Bing has invested more in Microsoft Copilot, and Google
has doubled down on the use of its Gemini chatbot in search results following
its annual I/O event in May 2024, announcing its rollout to hundreds of millions
of users worldwide over the course of the year.

At its best, AI can analyse data to provide more accurate and helpful
information for the user in the form of personalised search results. This
certainly presents a massive paradigm shift for businesses operating in and
relying on search engines for their traffic and revenue, as the onus will be on
them to keep on producing the best content and web experience that seeks to
become part of the AI-led journey in the search results.

Much like ChatGPT wowed the world in its ability to perform often mundane tasks,
users will expect search engines to follow suit and do the heavy lifting for
them. This will likely involve every stage of the commercial journey, from early
informational discovery through to purchasing, and presents a potential massive
shift for online businesses. 


WILL ANYONE OVERTAKE GOOGLE?

The AI boom has seen a lot of development that could change what users expect
from search engines for a long time to come. Alongside Google Gemini, Microsoft
Copilot and others, OpenAI is also developing its own search engine and there
are new players on the block such as Perplexity AI, an AI-powered chatbot search
engine. While Bing and others have failed to make serious inroads into Google’s
monopoly over the years, these players could represent a threat to Google that
could disrupt the search engine market landscape in the years to come.


SEARCH ENGINES LAUNCHED

Search EngineLaunch DateDescriptionArchie1990The first internet search
engine.Gopher1991A file search protocol used before the web.Veronica1992A search
engine for the Gopher protocol.Jughead1993Another Gopher search
engine.WebCrawler1994The first full-text search engine.Lycos1994Early search
engine with a large index of documents.Yahoo! Directory1994A human-edited
directory.Infoseek1994Another early web search engine.AltaVista1995Notable for
advanced search capabilities.Excite1995Known for web portal
features.MetaCrawler1995A metasearch engine aggregating results.Daum1995South
Korean web portal with search capabilities.Inktomi (HotBot)1996Powered several
major search engines.Ask Jeeves1996Known for natural language
queries.Dogpile1996Another metasearch engine.Seznam.cz1996Czech search
engine.Rambler1996Russian search engine.Northern Light1997Unique categorisation
of results.Yandex1997Leading search engine in Russia.Google1998Introduced a link
analysis algorithm.Naver1999Major search engine in South
Korea.Teoma2000“Subject-Specific Popularity” algorithm.Baidu2000Major search
engine in China.AlltheWeb2000Known for its comprehensive
index.Wisenut2001Acquired by LookSmart.Gigablast2002An open-source search
engine.A9.com2004Amazon’s search engine.MSN Search2004Microsoft’s early search
engine, evolved into Bing.Sogou2004Popular Chinese search
engine.Yippy2004Clustered search results.Yacy2004Peer-to-peer search
engine.Ask.com2006Rebranded from Ask Jeeves.Startpage2006Privacy-focused search
engine.Mahalo2007Human-powered search engine.Cuil2008Large index and different
search methodology.DuckDuckGo2008Focuses on user privacy.Ecosia2009Eco-friendly
search engine.Bing2009Microsoft’s search engine.WolframAlpha2009Computational
knowledge engine.Blekko2010Spam-free search results.Qwant2013Privacy-oriented
European search engine.Brave Search2021Privacy-focused search engine by Brave
Software.


REFERENCES

[1] History of the Internet – Wikipedia

[2] That’ll be all for now, Jeeves – The Guardian

[3] What Is Google PageRank? – Search Engine Land

[4] Google and Yahoo Settle Dispute Over Search Patent – The New York Times

[5] Keynote with Kevin Johnson at Microsoft – Search Engine Roundtable

[6] Baidu’s Internal Monitoring and Censorship Document Leaked – China Digital
Times

[7] Why Google Quit China – The Atlantic

[8] DuckDuckGo search market share – Statista 

[9] Search Engine Market Share – US – StatCounter

[10] Google paying Apple $9 billion to remain default search engine in Safari –
9to5 Mac

[11] Building for the next moment – Google

[12] Smart speaker quarterly shipment share from 2016 to 2019, by vendor –
Statista

[13] How much data do we create every day? – Forbes

[14] Mobile-first indexing has landed – Google

[15] 69 Voice Search Statistics 2024 – Yaguara

[16] How Google Is Remaking Itself as a “Machine Learning First” Company – Wired

[17] A New Chatbot is a “Code Red” for Google’s Search Business – New York
Times[18] Google scrambles to manually remove weird AI answers in search – The
Verge





CARL HENDY

Carl has 20 years of experience in SEO and ecommerce advisory roles, covering
both strategy and technical execution.

 * 
 * 

LET'S WORK TOGETHER?

Discuss Project



MORE USEFUL INSIGHTS

September 13, 2024


SEO CHALLENGES IN 2024

CARL HENDY



June 12, 2024


SEO AWARDS: DO THEY REALLY MATTER?

CARL HENDY



June 12, 2024


HOW LONG DOES SEO TAKE?

CARL HENDY



SEO NEWSLETTER

Our newsletter is focused on the commercial impact that changes in the SEO
industry and updates to Google's algorithm have on your business.

Google reCaptcha: Invalid site key.


Subscribe
 * 
 * 
 * 




SEO TRUSTED BY BRANDS, INVESTORS AND IN-HOUSE TEAMS

Increase your website revenue, not your task lists.

Company

 * About
 * Contact
 * Insights
 * Tools

Growth & Opportunity

 * SEO Discovery Audits
 * International SEO
 * SGE / AI SEO
 * Increase Organic Revenue
 * Share of Voice (SOV) Audits
 * Resolve Google Algorithm Issues

Technical Consulting

 * SEO Audits
 * Ecommerce SEO Audits
 * Financial Services SEO
 * Marketplace SEO Audits
 * SEO Product & User Experience (UX) Audits
 * Domain & Website SEO Migrations Audits
 * SEO Technology Selection

Training & Development

 * Team Coaching
 * Bring SEO In-House
 * SEO Channel Alignment Audits

Expertise

 * SEO Consultancy
 * SEO Knowledge Base
 * Insights
 * Fractional SEO Consultancy

 * Privacy Policy
 * Terms & Conditions

Marketing Pty Ltd t/a Audits.com. All Rights Reserved © 2024