Search Engine Friendly Robots.txt and Sitemap.xml
Making it Easy for the Internet to Find Your ecommerce Products
What is Robots.txt and Why Does My Website Need It?
A Robots.txt file is the standardized way in which your ecommerce website tells search engines what they may and may not crawl on your site. This is very useful if you have pages which, for various reasons, you do not want search engines to crawl. Common applications include blocking crawlers from non-public portions of your site, print versions of HTML pages (and other duplicate content), and auto discovery of XML Sitemaps. Having a robots.txt makes indexing a website and its content much more efficient for the search robot. Clarity's ecommerce solution automatically generates this file and keeps it updated as you add new content to your website. Additionally, editing the robots.txt file is very easy for individual pages through Clarity's user friendly platform. It is important to note two common mistakes people make in regards to the use of robots.txt. First, malicious robots will ignore the file, so DO NOT try and use it as a security measure. Second, sometimes it is better to tell a search crawler not to index the page with a robots meta tag (noindex) on that page rather than the robots.txt; Google does not support noindex rules in robots.txt, and a page blocked by robots.txt cannot be crawled, so its noindex tag will never be seen. This way, other areas of your site still receive 'link juice,' or links from the noindex page, that help boost the SEO value of other areas of your site. For more information on Robots.txt best practices for SEO, visit Moz's article here.
Learn What Clarity Can Do for You
We're happy to talk with you and answer any questions. Click the button below and fill out our short form or use our live chat feature (button on the bottom right corner) to talk to an expert right away!
What is Sitemap.xml and Why is it Important for SEO?
Whereas the Robots text's purpose is to efficiently tell search crawlers what to steer clear from within your site, the Sitemap.xml is used to tell search crawlers which URLs within your site you want crawled and indexed. This bit of Extensible Markup Language (XML) lists each URL, optionally with its last-modified date, so search engines can discover new and updated pages; under the sitemaps.org protocol, each sitemap file can hold up to 50,000 URLs or 50MB uncompressed. Again, Clarity's ecommerce platform makes this easy by automatically generating new content within their appropriate place within the XML Sitemap. The Sitemap Clarity's solution generates is clean, from both a user's and a search crawler's standpoint, and follows best practices. These two documents are essential to any website, but are a pain to generate manually. The automated generation of these pages is just one more example of how Clarity's ecommerce solution makes the jobs of your team both easier and more efficient.