Crawl and Index Websites
Search engines crawl and index billions of documents - not just websites. This may include files, videos, photos, news and any other media found on the web.
Search engines make use of spiders, a type of software, to find these web pages. Anytime a link appears on a web page, the spider will crawl that page and, hopefully, index it. Links, essentially, tie all web pages together.
This is how spiders are able to access the billions of pages on the web. Each time the crawler finds a new page, it looks for certain signals and saves the information for later recall. All of this information is stored on gigantic hard drives. To store this massive amount of information and ensure that results are generated in less than one second, search engines use multiple datacenters across the globe.
Inside of these datacenters are thousands of machines, which are capable of processing an extensive amount of data. When a user searches for a term, they expect results to appear instantaneously. A one or two second delay can cause a user to become frustrated, so search engines strive to make sure that they provide quick and reliable results.
A search query is essentially a question. Search engines seek to answer user questions by providing them with a list of websites that have information relevant to their query.
Each time a person performs a search, the search engine scours through its index of billions of web documents to find relevant results and then list them in order of usefulness. In order to rank high, a web page must be relevant to the query and the website itself must be seen as important to the search engine. The SEO process works to influence both of these factors.
While it may be easy to assume that relevance means simply making sure that the targeted keyword appears on the page a set amount of times, it is a bit more complicated than that. When search engines were in their infancy, this strategy may have been effective. Because this factor is so easily manipulated, it created poor results. Ultimately, search engines had to come up with more complex methods of determining relevancy. Now, they rely on more than one hundred signals to determine how relevant a web page is. Many of these signals will be discussed right here in this guide.
How Does A Search Engine Determine A Website's Importance?
The search results are kind of like a popularity contest. Search engines assume that the more popular a website is, the more important or valuable the information contained on that website must be. For the most part, this has been a fairly accurate assumption and has been rather successful in delivering relevant and useful web pages to users.
A website's importance and relevancy are not determined manually as this would be an impossible task. Search engines rely on algorithms, which are mathematical equations, to determine what web pages are and where to rank them in the search results. These complex algorithms rely on hundreds of different factors, or what we call "ranking factors" in the SEO world.