Christchurch SEO Blog

XML Sitemaps and robots.txt: What They Do and How to Check Yours

Learn what robots.txt and an XML sitemap each do, how to open and check yours in a few minutes, and the leftover staging mistake that can hide a whole site.

Quick Answer

Robots.txt tells search engine crawlers which parts of your site they may visit, while an XML sitemap lists the pages you want found. You can check both in your browser by adding /robots.txt and /sitemap.xml to your address. Look for blocked pages that should be open, and sitemap entries that are redirected, broken or noindexed.

Robots.txt tells search engine crawlers which parts of your site they may visit, and an XML sitemap lists the pages you want them to find. Both are plain files you can open in a browser in under a minute. The real risk is small but serious: one wrong line in robots.txt can quietly hide your whole site.

1. Open your robots.txt file

Type your web address followed by /robots.txt, for example yourbusiness.co.nz/robots.txt. You should see a short block of plain text. If you get a 404, the file is missing; that is not fatal, because crawlers then assume everything is open, but it is better to have one.

The lines you will see are simple. "User-agent" says which crawler a rule applies to (an asterisk means all). "Disallow" lists paths they should not crawl. "Allow" makes exceptions.

One thing to understand: robots.txt controls crawling, not indexing. A blocked page can still appear in Google if other pages link to it, and blocking it stops Google reading a noindex tag on it. Keeping a page out of results is a separate job.

2. Check nothing important is blocked

Read each Disallow line and ask what it covers. Blocking admin areas, cart pages or internal search results is normal. Blocking your services, blog or images is not.

The classic mistake is a single line reading "Disallow: /", which blocks the entire site. It is often switched on deliberately while a site is being built on a staging copy, then carried across at launch. Say a Rolleston builder launches a new website and six weeks later wonders why nothing shows up. The reason can be one leftover line. If that sounds like your situation, why a business is not showing up walks through the other likely causes.

If a line looks wrong, ask your web developer to remove it. Do not edit the file blindly on a live site. If you are unsure what else a wrong line has affected, a full site audit will show it.

3. Open your sitemap and check what it lists

Visit yourbusiness.co.nz/sitemap.xml. If it is missing, check your robots.txt, because the address sometimes differs, or ask your developer. Larger sites often use an index file pointing to several smaller ones.

The sitemap should list the canonical, indexable pages you want found: your homepage, services, location pages and blog posts. It should not list:

  • pages that redirect somewhere else
  • pages that return a 404 error
  • pages marked noindex
  • duplicate versions of the same page

Every entry should be the address you want in Google, including the secure version and your chosen www or non-www form. If you are unsure which addresses count as preferred, our explainer on canonical tags covers it. Click a dozen entries at random. Anything that lands on a different address or an error page needs removing or fixing.

Be realistic about what inclusion means. Being listed in the sitemap tells Google a page exists and that you consider it important. It does not guarantee the page will be indexed or ranked. If a handful of pages sit in your sitemap but never appear in Google, the cause is usually quality, duplication or a block elsewhere, and the Pages report in Search Console will usually say which.

4. Make sure lastmod dates are honest

Many sitemaps include a last-modified date per page. Google has said it can use this when it proves reliable. That only works if the date means something. Some tools stamp today's date on every page whenever the sitemap regenerates, so everything looks freshly changed every day. Over time that teaches Google to ignore the dates.

Compare a few dates with when you actually edited those pages. A page last touched in 2023 should not claim it changed this morning. If they all show the same recent date, ask your developer or plugin settings to use the real edit date instead.

5. Reference the sitemap in robots.txt

Add one line to the file giving the full sitemap address, written as "Sitemap:" followed by the complete address, including https. It helps any crawler find it without being told, not only Google. Your developer or SEO plugin can usually add this in a couple of minutes.

6. Submit it and read the status

In Search Console, open Sitemaps, paste your sitemap address and submit. Then check the status. "Success" is what you want, and the report shows how many URLs Google discovered. If the count is far lower than your page count, or the status reports errors, go back to step 3. Our beginner's guide to Search Console covers the rest of the tool.

Revisit it after any redesign, domain change or plugin update. Catching an accidental block at that point is usually a quick fix compared with finding it months later. It sits within the wider technical SEO checklist, and it is easy to forget in the list of common mistakes on Christchurch websites.

A tidy sitemap does not make pages rank. It makes sure the pages you have are findable.

If you would like us to check both files and everything around them, our SEO audits cover crawling and indexing in detail, and a free SEO audit will flag any obvious block.

Stu Clark
Written by Stu Clark

Former plumber, 15 years in the trade including 10 running his own business in Cheltenham, UK, before founding EightySix Digital. Every SEO engagement is run by Stu directly - no offshore writers, no junior account managers.

Common Questions

Quick questions.

No. Robots.txt controls crawling, not indexing. A blocked page can still appear in results if other sites link to it, usually without a description. To keep a page out of Google, let it be crawled and add a noindex tag, or remove it.

Not strictly. Google can find pages through links. But a sitemap costs almost nothing, helps new pages get discovered sooner, and gives you a report in Search Console. For most business sites it is worth having.

Look at both after any redesign, migration or plugin change, and glance at them a few times a year otherwise. Most serious problems come from a site change, so those are the moments to check.

You should include only the pages you want people to find in search, meaning live, canonical and indexable. Leave out redirected pages, thank-you pages, internal search results and anything marked noindex. A cleaner sitemap is more useful to Google.

Keep Reading
Free SEO Review

Want this done for your business?

Get a free, no-obligation review of your Christchurch business's SEO - what's working, what isn't, and what to fix first.