Sitemap is a file contain list of URLs on our website where the URLs are allows to be crawled and indexed by search engine i.e. Google Search. On the most cases it’s intended to only consumed by search engine crawled so it isn’t need to be human friendly; but just naked link list. Usually a sitemap file has an extension of .xml but it’s also allowed to have other extension like .html or .txt by Google.
For optimization purposes sitemap file usually saved on the root directory for example on hxxp://www.example.com/sitemap.xml. It’s seems an un-important thing but we could think what’s the faster way within crawl sitemap file on the root directory (i.e. hxxp://www.example.com/sitemap.xml) and on a specific folder (i.e. hxxp://www.example.com/folder/../sitemap.xml). It looks likely the faster way to craw is when we save our sitemap file on the root directory of our website.
Robots.txt is a text file otherwise noted is exists on the root directory of a website. On the most case it’s the first destination to be crawled by search engine robot i.e. googlebot. Robots.txt contains information about what folder and/or what file allowed and don’t allowed to be crawled by search engine crawler (i.e. googlebot).
Benefits of adding sitemap file link to the robots.text file
On the common practice, submitting sitemap to search engine is a simple task. But on uncommon condition when we aren’t able to submit our sitemap to a search engine due to a special reason for examples because of the search engine native language is differ from ours and they don’t offers a translator or different language version this will help the search engine crawler to find the sitemap file on our website by read it on the robots.txt file. Another reason why we need to add our sitemap file link on the robots.txt file is for giving a direction to un-known crawler (if any) for the existence of our sitemap file so it could indexing our sitemap file by itself without the need of manual sitemap submission.
So, this is the end of this article, I hope it’s useful for you to increase up your website visibility on the eyes of the prospective visitor, and I don’t forget to ask you a forgiveness if there are many mistakes exists on this article made by me. Good bye and wish you success with your website!
This article is proudly presented by Home Ideas Made Simple.