Crawl/Index Readiness Checklist
If you’re trying to make content “AI readable,” you don’t start with prompts. You start with infrastructure: can bots access the page, and should they keep it?
This checklist is written to be used in tickets, QA runs, and release gates. Print it. Put it in your definition of done. Make it boring. Boring is reliable.
Crawl vs index: the 60-second mental model
- Crawling = a bot fetching a URL and reading its content.
- Indexing = storing that content so it can be retrieved later (e.g., via search).
- robots.txt mostly controls crawling.
- meta robots and X-Robots-Tag mostly control indexing and serving behavior, but they must be discovered by crawling.
Google’s documentation is explicit: robots meta tags and X-Robots-Tag headers are discovered when a URL is crawled. If the URL is blocked by robots.txt, those directives might be ignored.
Part A – Crawl readiness (can machines fetch it?)
A1) HTTP status and redirects
- Important pages return 200 (not 404/410/500).
- Redirect chains are short (ideally 1 hop).
- Canonical pages do not redirect (your canonical URL should be the final destination).
- HTTPS is enforced sitewide.
A2) robots.txt sanity check
- Do not block service pages, location pages, or critical category pages you want indexed.
- Do not block essential resources (CSS/JS) that affect rendering of primary content.
- Avoid using robots.txt as a canonicalization strategy (it’s not).
- Verify any bot-specific rules (e.g., AI crawlers) align with your goals.
Reminder: if you block crawling, you also block discovery of indexing directives. If you need noindex to be respected, the page must be crawlable.
A3) Renderability (what the bot actually sees)
- Critical content is in the initial HTML or reliably rendered for the smartphone crawler.
- No infinite scroll walls that hide key content without URL states.
- No cookie/consent overlays that block content for crawlers.
- Structured data is present in the rendered HTML.
Part B – Index readiness (should machines keep it?)
B1) meta robots and X-Robots-Tag
- Important pages are not noindex.
- Thin/duplicate pages are intentionally noindex (if needed).
- PDFs or non-HTML assets use X-Robots-Tag when appropriate.
- Index directives are consistent across environments (no staging rules leaking into prod).
Google documents how to block indexing with noindex and also documents robots meta tags and X-Robots-Tag as mechanisms to control indexing.
B2) Canonicalization and duplicate control
- Each page has exactly one rel=canonical pointing to the canonical URL.
- Canonical is self-referential on canonical pages (points to itself).
- Internal links point to canonical URLs (not parameter versions).
- Sitemaps list canonical URLs (not duplicates).
- If duplicates exist, use redirects where possible and consistent canonical tags everywhere.
Google documents multiple methods for canonicalization and warns that conflicting signals can cause Google to pick a different canonical than you intended.
B3) XML sitemap hygiene
Sitemaps are not a dump of every URL you’ve ever generated. They’re a clean list of what you want discovered and treated as canonical.
- Sitemaps include only canonical, indexable URLs.
- Sitemaps are kept within limits (50,000 URLs or 50MB uncompressed per sitemap).
- Sitemap index files are used for large sites.
- Sitemaps are referenced in robots.txt where appropriate.
Part C – Structured signals (can machines interpret it?)
C1) Structured data presence and correctness
- Organization schema exists on the homepage.
- LocalBusiness schema exists on each location page (if applicable).
- Service schema exists on each service page.
- FAQPage schema is only used where Q&A is visible on-page.
- Schema validates in Rich Results Test and Schema Markup Validator.
Google’s structured data guidelines recommend JSON-LD where possible and emphasize that markup should represent visible content.
Part D – Practical QA workflow (the ‘don’t ship broken signals’ loop)
D1) Pre-release (staging) checks
- Crawl the staging site with a crawler tool (or scripted fetches) to confirm response codes, canonicals, and index directives.
- Validate a representative set of templates (homepage, service page, location page, blog article).
- Check mobile rendering and structured data parity.
D2) Release checks (production smoke test)
- Run spot checks on the top revenue pages (services/locations).
- Confirm sitemap submission and fetchability.
- Validate one page per template in Rich Results Test and Schema Markup Validator.
- Confirm no accidental noindex headers on key pages.
D3) Post-release monitoring
- Use Search Console URL Inspection for key pages to see how Google sees the URL.
- Watch indexing reports for spikes in excluded or duplicate statuses.
- Monitor server logs for crawler errors (5xx) and blocked resources.
- Schedule a recurring audit (monthly or per sprint) to catch regressions.
Next Module: Schema Playbook
Troubleshooting flows (when things go sideways)
1st Flow : ‘Page not indexed’ but you swear it should be
- Check robots.txt (is crawling blocked?).
- Check meta robots / X-Robots-Tag (is noindex present?).
- Check canonical (is it pointing somewhere else?).
- Check sitemap (is the canonical URL included?).
- Check rendering (is key content missing on mobile?).
2nd Flow: ‘Duplicate, Google chose a different canonical’ warnings
- Compare canonical tags, redirects, and internal links for duplicates.
- Ensure sitemap only contains canonical URLs.
- Remove mixed signals (e.g., different canonicals by template).
- Eliminate parameter duplicates (where possible) or standardize canonical handling.
3rd Flow: ‘AI keeps attributing our service to the wrong brand’
- Verify Organization schema (name/url/logo/sameAs) is consistent and stable.
- Verify service pages reference the Organization @id as provider.
- Verify canonical URL consistency (AI systems often follow canonical truth).
- Verify location data (if applicable) matches on-page and markup.
Next module: Performance + Mobile fundamentals
Sources
[1] Google Search Central. Robots Meta Tags Specifications. Accessed January 11, 2026.
[2] Google Search Central. Block search indexing with noindex. Accessed January 11, 2026.
[3] Google Search Central. How to specify a canonical URL (consolidate duplicate URLs). Accessed January 11, 2026.
[4] Google Search Central. Build and submit a sitemap. Accessed January 11, 2026.
[5] Google Search Central. Introduction to structured data markup. Accessed January 11, 2026.
[6] Google Search Central. General structured data guidelines. Accessed January 11, 2026.
[7] Google Search Central. Mobile-first indexing best practices. Accessed January 11, 2026.
[8] Google. Rich Results Test. Accessed January 11, 2026.
[9] Schema.org. Schema Markup Validator. Accessed January 11, 2026.
About The Author
