 |
| Website Indexing and Crawl Information |
Website Indexing and Crawl Information
Today is the fifth episode of the series 'Learn SEO', Learn how search engine bots can find your website, how to know the number of indexed pages? What are the reasons search engines can't find a website? About Robot.txt file and its usage.
Crawling: Will your pages be found by search engines?
You already know that a prerequisite for your site to appear in the SERP is that the site must be crawled and indexed by search engines. If you have a website of your own, check to see how many pages have been indexed. You can find out if Google is able to crawl and index your website.
One way to check a site's index is to search for "site: yourdomain.com" by typing "site: yourdomain.com" into Google's search bar.
See the picture below.
The image above is a method of checking the number of indexed pages.
Although the number that Google shows is not real or accurate, it lets you know how many pages are being indexed and how it looks in the SERP.
If your site is not found anywhere in search engines, there may be some possible reasons. E.g .:
- If your site is brand new and has not yet crawled.
- If your site has no backlinks.
- The menu/navigation on your site is so messy that search engines can't find the bot to crawl.
- Your site may contain code that prevents search engines from crawling.
- Your site may have been penalized by Google as a spam site.
Robots.txt file
The robots.txt file is located in the root directory of the web site (ex. Yourdomain.com/robots.txt) and indicates to search engines which parts of the site will be indexed and which will not. In addition, instructions are given on the speed at which the site will be indexed.
How does Google bot follow the robots.txt file?
If the Google bot doesn't find the robots.txt file for a site, it starts crawling the site without any instructions. If the Google bot finds the Robots.txt file for a site, it will follow the instructions given there and start crawling the site. If a site encounters an error while accessing the robots.txt file and cannot determine it, it will not crawl the site.
Read More
- Lesson - I (Introduction to SEO)
- Lesson-II (About SEO)
- Lesson-III (Search Engine Guidelines)
- Lessin-IV (About Search Engine)
- Next and All Post
.jpg)