Robots.txt & Sitemaps for Small Sites

What robots.txt and sitemap.xml do, why your small business site needs them, and how to set them up correctly (with common mistakes to avoid).

Two small files quietly control whether Google can crawl and index your site properly: robots.txt and sitemap.xml. Get them wrong and your pages may simply never show up in search, no matter how good they are. Get them right — a ten-minute job — and you've cleared one of the most common silent blockers in all of SEO. Here's exactly what each file does and how to set it up.

robots.txt: the crawl instructions

This plain-text file sits at yourdomain.com/robots.txt and tells search engine crawlers which parts of your site they may request. A simple, safe version that works for almost every small business site looks like this:

User-agent: *
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

That says "any crawler may access everything, and here's my sitemap." For most small sites, that's all you need.

The #1 mistake: a broken or 500-erroring robots.txt. If Googlebot requests the file and gets a server error (rather than a clean file or a clean 404), it may cautiously refuse to crawl your entire site until the file responds normally. Always confirm your robots.txt returns a clean 200.

robots.txt does NOT hide a page from Google

A crucial misconception: Disallow stops crawling, not indexing. If other sites link to a page you've disallowed, Google can still list that URL in results (usually without a description). To actually keep a page out of the index, let it be crawled and add a <meta name="robots" content="noindex"> tag, or protect it behind a login. Don't rely on robots.txt for privacy.

sitemap.xml: the map of your pages

A sitemap is an XML file listing the URLs you want indexed, which helps search engines discover every page — especially newer pages or ones that aren't well linked internally. A minimal entry looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://yourdomain.com/</loc>
    <lastmod>2026-07-14</lastmod>
  </url>
</urlset>

List only your real, canonical URLs — no redirects, no www/non-www duplicates, no pages you've set to noindex. Then submit the sitemap in Google Search Console so Google re-crawls it when you update. Most CMS platforms and SEO plugins generate and update the sitemap for you automatically; you just have to submit the URL once.

Common mistakes to avoid

• Listing URLs that redirect instead of the final, canonical URL.
• A leftover Disallow: / from a staging build that blocks the whole live site — this one has tanked many launches.
• Forgetting to submit the sitemap to Search Console.
• Mixing protocols/hosts (http vs https, www vs non-www) between your links, canonicals, and sitemap.
• A robots.txt that errors instead of returning a clean 200.

FAQ

Do I even need a robots.txt? Technically no — a missing robots.txt (clean 404) means "crawl everything," which is fine. But having one lets you point crawlers to your sitemap and is the standard.

Will a sitemap make me rank? No. It helps Google discover pages faster; it doesn't boost rankings. It's most valuable on large or poorly-linked sites.

How do I test them? Load both files in your browser and confirm they return correctly. Search Console has a sitemaps report showing what Google fetched and any errors.

Check yours in seconds

Run your homepage through the free Audit SEO tool — it checks that your robots.txt and sitemap are present and valid, along with 16 other on-page factors. These two files are the entry point to the wider technical SEO checklist, and keeping your sitemap free of redirects works hand in hand with staying on top of find and fix broken links.