Why do we need to add sitemap link on our robots.txt file?

On this article I’d like to try to explain the need of adding sitemap link on the robots.txt file. I do it on a hope it could be useful for this article’s readers so they aren’t loose too much time to search on for a useful reading.
What’s sitemap?
Sitemap is a file contain list of URLs on our website where the URLs are allows to be crawled and indexed by search engine i.e. Google Search. On the most cases it’s intended to only consumed by search engine crawled so it isn’t need to be human friendly; but just naked link list. Usually a sitemap file has an extension of .xml but it’s also allowed to have other extension like .html or .txt by Google.
For optimization purposes sitemap file usually saved on the root directory for example on hxxp://www.example.com/sitemap.xml. It’s seems an un-important thing but we could think what’s the faster way within crawl sitemap file on the root directory (i.e. hxxp://www.example.com/sitemap.xml) and on a specific folder (i.e. hxxp://www.example.com/folder/../sitemap.xml). It looks likely the faster way to craw is when we save our sitemap file on the root directory of our website.
What’s robots.txt?
Robots.txt is a text file otherwise noted is exists on the root directory of a website. On the most case it’s the first destination to be crawled by search engine robot i.e. googlebot. Robots.txt contains information about what folder and/or what file allowed and don’t allowed to be crawled by search engine crawler (i.e. googlebot).
Benefits of adding sitemap file link to the robots.text file
On the common practice, submitting sitemap to search engine is a simple task. But on uncommon condition when we aren’t able to submit our sitemap to a search engine due to a special reason for examples because of the search engine native language is differ from ours and they don’t offers a translator or different language version this will help the search engine crawler to find the sitemap file on our website by read it on the robots.txt file. Another reason why we need to add our sitemap file link on the robots.txt file is for giving a direction to un-known crawler (if any) for the existence of our sitemap file so it could indexing our sitemap file by itself without the need of manual sitemap submission.
So, this is the end of this article, I hope it’s useful for you to increase up your website visibility on the eyes of the prospective visitor, and I don’t forget to ask you a forgiveness if there are many mistakes exists on this article made by me. Good bye and wish you success with your website!

This article is proudly presented by Home Ideas Made Simple.

Posted Sitemap on Google, They said it wasn’t correct?

I'm using the the Fluid store and Fluid WP themes. When I finished my store and submitted to google, I submitted the url to the sitemap (that was automatically generated in the fluid theme) without putting xml or php on the end.

I tried to resubmit, but google wouldn't allow it. weird. I'm hitting the end of my knowledge rope here.

Anybody know how to fix this?

Thanks, -FN

Why sitemap ranks for some keywords?

Hi,

In India, for a popular keyword ‘matrimonial’, this sitemap page (https://www.shaadi.com/sitemap) is ranking in 2nd position for long time. Good number of competition is there but how these sitemap is ranking? because these page links to lot of another pages with anchor text éxample matrimonial’ ?

If a page has a good number of anchor links which points to internal pages, does it increases the source page’s chances of ranking for that specific keyword?

Having anchor links texts like ‘bengali matrimonial’, ‘gujarati matrimonial’, etc increased the ranking of ‘matrimonial’ for the sitemap page?

Otherwise, how the ranking could have been happened?

Submitting sitemap of blogger blog to google search console

In many of the articles, I found that I can submit sitemap of my blogger blog by inputting

“atom.xml?redirect=false&start-index=1&max-results=500”

as my sitemap url. But when I did this, I got a URL error saying that it couldn’t fetch. I’ve also tried “myblogaddress/sitemap.xml”. None of it works. And on the old search console, it shows pending. Now google automatically index some of my page randomly. So I need to manually input all the pages on URL inspection and check if its indexed and request for index. It takes so much time. It will be very helpful if I can do it by a sitemap.

How can I upload a sitemap to my Google Sites site with a custom URL?

I created a site for an e-portfolio using the new Google Sites. Google sites automatically generates a sitemap for you at sites.google.com/yoursite/system/feeds/sitemap, which can then be submitted via Google’s “Search Console”. However, I migrated to a custom URL, and so I no longer have a sitemap at that location. Google Sites does not have FTP support, and applications such as FileZilla can’t access it. I have heard that SML Cloud can make up for this, but I would prefer not to pay $ 5 a month just for a sitemap.

One idea I had is to put my sitemap on a different domain from my site, but I am not sure if this would even work, or where I could go about doing that. Also, if I need to finish making my website and then export it to a different host somehow to be able to use FTP, then I don’t mind doing that either.

Is there any way to put a sitemap on my Google Sites site that has a custom domain?

Thanks,
Calan Smith

How will my blogs get crawled without sitemap entry?

I am completely rebuilding a site for a friend of mine, and am in the process of completing the
blog section. I’m not using wordpress, however I have been studying the source code on pages
that utilize wordpress and yoast.

I have a question: Is there any way to ensure, or help to make more likely, that Google crawls
the blog posts, even though all the blogs obviously cannot be included in the sitemap file?