Search engine

A search engine is a special kind of software system that helps people find information on the Web. When someone wants to look up something, they type a question or word, called a query, into a web browser or a mobile app. The search engine then quickly looks through lots of web pages and other information to give back a list of search results. These results usually include links, short descriptions, and sometimes pictures to help the person find what they are looking for.

Behind the scenes, a search engine uses a big, spread-out system of computers in many places around the world, called a distributed computing system. It works very fast and accurately because of a process called indexing, which is updated all the time by special programs known as web crawlers. These crawlers go through many web servers to collect information, although not all content can be found this way.

Many search engines have existed since the Web began in the 1990s, but one called Google Search became the most popular in the 2000s and still is today. As of May 2025, Google is used by about 89–90% of people around the world for searching, with other search engines like Bing, Yandex, Yahoo!, DuckDuckGo, and Baidu making up the rest. Because of this, many websites try to make sure they show up well in Google’s search results through a process called marketing and optimization.

History

Further information: Timeline of web search engines

Pre-1990s

In 1945, Vannevar Bush described a system that would help people find information easily. He called this system a memex and wrote about it in an article titled "As We May Think" in The Atlantic Monthly. The memex was meant to make it easier to find information as more and more data grew. Vannevar Bush thought of libraries with connected notes, much like the hyperlinks we use today.

Link analysis later became important for search engines through methods like Hyper Search and PageRank.

1990s: Birth of search engines

The first search engines appeared before the web existed in December 1990. WHOIS allowed users to search in 1982, and the Knowbot Information Service began in 1989. The first search engine to look through files was Archie, which started on September 10, 1990.

Before September 1993, the World Wide Web was organized by hand. There was a list of webservers kept by Tim Berners-Lee at CERN. As more web servers appeared, this list could not keep up. On the NCSA site, new servers were listed under "What's New!".

The first tool to search the content of the Internet was Archie. It stood for "archive" without the "v". It was created by Alan Emtage, a student at McGill University in Montreal, Quebec, Canada. Archie downloaded lists of files from public FTP sites, creating a searchable database of file names.

The rise of Gopher in 1991 led to new search tools like Veronica and Jughead. Like Archie, they searched file names and titles in Gopher systems. Veronica allowed keyword searches of Gopher menu titles. Jughead got menu information from specific Gopher servers.

In the summer of 1993, no search engine existed for the web, but many catalogs were kept by hand. Oscar Nierstrasz at the University of Geneva created W3Catalog, the web's first simple search engine, released on September 2, 1993.

In June 1993, Matthew Gray at MIT created the World Wide Web Wanderer, an early web robot. It was used to measure the size of the web until late 1995. The web's second search engine, Aliweb, appeared in November 1993. It relied on website administrators to provide information about their sites.

JumpStation, created in December 1993 by Jonathon Fletcher, was the first tool to combine crawling, indexing, and searching. Because of limited resources, it only indexed titles and headings from web pages.

One of the first search engines to search all text was WebCrawler, released in 1994. It let users search for any word on any web page, which became the standard. Also in 1994, Lycos was launched and grew to be popular.

The first popular web search engine was Yahoo! Search. Started by Jerry Yang and David Filo in January 1994, it began as a Web directory called Yahoo! Directory. In 1995, a search function was added, making it a favorite way to find web pages.

Many search engines appeared after that, competing for popularity. These included Magellan, Excite, Infoseek, Inktomi, Northern Light, and AltaVista. People could also browse directories instead of searching by keywords.

In 1996, Robin Li developed the RankDex algorithm for ranking search results. It was the first to use hyperlinks to judge website quality, before Google's similar PageRank in 1998. Li later used this technology for the Baidu search engine, launched in China in 2000.

In 1996, Netscape planned to feature one search engine but ended up making deals with five: Yahoo!, Magellan, Lycos, Infoseek, and Excite.

Google started selling search terms in 1998, changing the search engine business.

Many search engine companies grew quickly in the late 1990s but were affected by the dot-com bubble that ended in March 2000.

2000s–present: Post dot-com bubble

Around 2000, Google's search engine became very popular. It used an algorithm called PageRank, created by Sergey Brin and Larry Page, the founders of Google. This method ranks web pages based on how many other popular sites link to them.

Yahoo! used Inktomi's search technology until 2002, when it bought Inktomi, and then Overture in 2003. Yahoo! used Google's search until 2004, when it launched its own search using its acquisitions.

Microsoft started MSN Search in 1998 using Inktomi's results. In 1999, it used Looksmart and AltaVista at times. In 2004, Microsoft began using its own technology with its web crawler called msnbot.

Microsoft launched its rebranded search engine, Bing, on June 1, 2009. On July 29, 2009, Yahoo! and Microsoft agreed that Yahoo! Search would use Microsoft's Bing technology.

As of 2019, active search engine crawlers include those of Baidu, Bing, Brave, Google, DuckDuckGo, Gigablast, Mojeek, Sogou and Yandex.

Approach

A search engine works by doing three main things all the time:

1. Web crawling
2. Indexing
3. Searching

Search engines gather information by moving from website to website, checking each one for allowed pages. They then store important words and details from these pages in a big list. When you type a question into a search engine, it looks at this list to quickly find the best matches.

When you search, you usually type in just a few words. The search engine already knows which websites have those words and shows them to you. It also lets you change your search to get better results, like looking for newer pages or using special words to narrow down what you want. The goal is to show the most helpful pages first, and many search engines also show ads that you can choose to see.

Market share

As of January 2022, Google is the most used search engine in the world, with a market share of 90%. Other popular search engines include Bing at 4%, Yandex at 2%, and Yahoo! at 1%. Many other search engines exist but have smaller shares of the market.

In Russia, Yandex is the leading search engine, with about 73% of the market, while Google holds around 25%. In China, Baidu is the main search engine, with a share of about 59.3%. Google does not operate in mainland China. In Japan, Google has the largest share, while Yahoo! Japan also has a notable presence. In South Korea, Naver leads the market, though Google's share has grown. In Taiwan, Google is the most used search engine.

Search engine bias

Further information: Algorithmic bias

Search engines try to show the best and most popular websites when you search, but sometimes they show information that is not fair or balanced. This can happen for many reasons. For example, companies that pay to advertise on a search engine might appear more often in the results. Also, some countries have laws that make certain information illegal, so search engines might not show those websites in those places.

Sometimes, the way search engines are set up can leave out less popular ideas or focus more on websites from certain countries, like the United States, rather than others. People have also tried to change search results for their own purposes, such as to influence what others think about important topics. Researchers have looked at how search engines affect our understanding of subjects like terrorism in Ireland, climate change denial, and conspiracy theories.

Google Bombing is one way people have tried to change what shows up in search results for political, social, or business reasons.

Customized results and filter bubbles

Some people worry that search engines like Google and Bing change what you see based on what you do online. This can make it feel like you only see things that match what you already think, which some call "echo chambers." In 2011, a person named Eli Pariser talked about this idea, saying that these changes might stop people from seeing different opinions.

Because of this concern, other search engines like DuckDuckGo were created. These try not to change what you see based on your past searches. However, some researchers say there isn’t strong proof that this big problem really exists. They found that most people still see many different ideas when they search online.

Religious search engines

Because the Internet has grown a lot in the Arab and Muslim world, some people wanted to create special search engines for them. These search engines help users find information that follows Islamic rules, called "halal", and avoid information that does not, called "haram". Examples include ImHalal, which started in September 2011, and Halalgoogling, which began in July 2013. These search engines use filters to keep out unwanted content.

Other religious search engines exist too, like Jewogle for Jewish users and SeekFind.org for Christian users. These also filter out websites that go against their beliefs.

Search engine submission

When someone who creates websites wants their site to be found easily, they can tell a search engine about it. This is called submitting a website. But usually, you don't need to do this because search engines have special programs called web crawlers that go around the internet finding websites on their own.

You can tell a search engine about just one page, like the main page, or you can tell them about your whole website using something called a sitemap. There are a couple of reasons to tell a search engine about your website: if it's brand new and not found yet, or if you've changed it a lot and want it to show up faster in search results. Some tools can tell many search engines at once and also add links to your site, but this might not always be the best idea because it can affect how well your site shows up in searches.

Comparison to social bookmarking

See also: Social media optimization

Unlike search engines, social bookmarking systems have some special benefits. In social bookmarking, people—not computers—choose tags to organize websites. This means a person can understand what a website is about, which helps make better choices. People can also save, or "bookmark," websites that search engines might not have found yet.

Social bookmarking can also show how popular a website is by counting how many people have bookmarked it. This can sometimes be more helpful than other ways of ranking websites. However, like search engines, social bookmarking systems can be tricked, so they need ways to protect against this.

Technology

Archie

The first web search engine was Archie, created in 1990 by Alan Emtage, a student at McGill University in Montreal.

Archie worked by collecting lists of files stored on File Transfer Protocol (FTP) sites. FTP is a way for computers to share files over the Internet. Users could connect to FTP sites to download files. Archie made it easier to find files by indexing them, so users did not need to know where to look.

Veronica

In 1993, Veronica was developed by the University of Nevada. It searched files stored on Gopher, another way to share information online, similar to how Archie searched FTP files. A similar tool called Jughead also appeared around the same time.

The Lone Wanderer

The World Wide Web Wanderer was created in 1993 by Matthew Gray. It was the first robot to travel the web and count how many websites existed. It also recorded the addresses of websites, creating the first web database called the Wandex.

Excite

Excite began as a project by six students at Stanford University in 1993. They wanted to make searching the Internet easier by studying how words were used together. Excite became a popular commercial search engine in 1995.

Yahoo!

In 1994, David Filo and Jerry Yang, two students at Stanford University, created Yahoo!. It started as a collection of web pages they found interesting. As more people used it, they organized the pages into categories, making it easy to search through. Yahoo! was not a typical search engine because it was first built by hand, but it later added search features.

Lycos

Lycos was developed in 1994 by Michael Mauldin at Carnegie Mellon University.

Types of web search engines

Web search engines help people find information on the Internet. They do this in three main ways:

1. Look for content that matches the words a person searches for.
2. Keep an index, or list, of where that content can be found.
3. Let users search through that index.

There are three main types of search engines. Some use robots, called crawlers, to go around the web collecting information. Others rely on people to submit information to be indexed. Some use a mix of both ways.

Crawler-based search engines send out robots to visit websites, read their content, and follow links to other sites. These robots bring back information to a central place where it is organized and stored. They check websites often to see what’s new.

Human-powered search engines depend on people to add information to be indexed.

When you search using a search engine, you are actually searching through an index, not the live web. This is why sometimes you might find links that no longer work — the index has not been updated.

Different search engines can give different results because they use different methods to decide which results are most relevant. They look at how often certain words appear on a webpage and how other pages link to it.

Modern search engines are very complex and use many computers to handle the huge amount of information on the web. Some search engines, like Google Scholar, focus on finding scientific research. Researchers are working to make search engines better at understanding the meaning behind words in articles.