Check whether Google can actually crawl and index a URL: status, robots.txt, meta robots, X-Robots-Tag and canonical, in one go.
Indexation issues silently cap your traffic. JW Digital fixes crawl, indexation and technical SEO at the root.
Most indexing problems get misdiagnosed because the two words get used interchangeably. They describe different stages of a pipeline, and a page can pass one and fail the other.
The practical consequence catches people out constantly: blocking a URL in robots.txt does not remove it from the index. It stops Google fetching the page, which means Google can never read the noindex tag you put on it. The URL can stay indexed on the strength of external links alone, listed with no description. To remove a page properly you have to allow crawling and serve a noindex, leave it that way until it drops out, and only then consider blocking it.
Rules are matched per user-agent group, and if a group names Googlebot explicitly then Googlebot follows that group and ignores the wildcard group entirely. So a permissive User-agent star block tells you nothing if there is a stricter Googlebot block further down the file. Within the winning group Google applies the most specific path rule, measured by rule length, with Allow winning a tie against Disallow. Reading the file by eye is unreliable, so confirm the verdict for a specific URL with URL Inspection in Search Console, and use the robots.txt report there to see the exact file Google last fetched. Remember robots.txt is scoped per protocol, per host and per port: the file on your www host does not govern a bare-domain or staging host.
The meta robots tag is the obvious one, and the one most tools check. The X-Robots-Tag response header does the same job invisibly, which makes it the more common silent killer: it can be set at CDN, reverse proxy or application level by somebody who is not you, and it is the only way to noindex non-HTML files such as PDFs and images. Check headers directly with curl -I on the live URL. Two more traps: a noindex injected by JavaScript is still honoured once the page is rendered, and noindex inside robots.txt has done nothing since Google stopped supporting that directive in 2019.
A self-referencing canonical is what you normally want. A canonical pointing at a different URL is a request to consolidate this page into that one, and if Google honours it your page will be reported as a duplicate with a different canonical chosen. The classic causes are a template hardcoding the homepage, a staging domain leaking into production, or pagination pointing every page back to page one. Note that rel=canonical is a strong hint rather than a directive: Google can and does pick a different canonical when your other signals, such as internal links and the sitemap, disagree with your tag. If Google is ignoring your canonical, the fix is to make the signals agree, not to shout louder in the tag.
A 404 or 410 will drop the URL, which is correct if the page is genuinely gone. Persistent 5xx errors are more damaging than they look, because Google responds by slowing its crawl rate across the site, not just for the failing URL. The subtle version is the soft 404: a page returning 200 that looks empty or like an error, such as an out-of-stock product, a category with no results or a client-rendered shell that fails to populate. Google classifies these itself and declines to index them. Fix them by returning a real status code or by making the page carry genuine content.
Every extra hop is another fetch and another chance to lose the signal. The most common source is stacked historical rules: http to https, then non-www to www, then old path to new path, then a trailing-slash rule, all firing in sequence. Audit with curl -IL and look at the whole chain, then collapse it so every legacy URL redirects to the final destination in a single hop. Loops, where two rules disagree, will stop the URL being indexed at all.
Google does render JavaScript, but rendering is queued separately from crawling and is not guaranteed to happen promptly. If your main content, internal links or canonical tag only appear after a client-side fetch, you have added latency and fragility to indexing for no benefit. Compare the raw HTML, via view-source or curl, against the rendered DOM in DevTools: anything present only in the second is at risk. Blocking your own JS or CSS in robots.txt makes this worse, because Google then renders a broken page. The durable fix is server-side rendering or static generation for anything that matters to search.
A page with no internal links pointing at it is technically indexable and practically invisible. A sitemap entry is a suggestion, not a substitute for internal linking, and pages that sit many clicks from the homepage with no links tend to get crawled rarely. Crawl your site with Screaming Frog or Sitebulb, then compare the crawled URL set against your sitemap: anything in the sitemap that the crawler never reached is an orphan. Also check how the page is linked. Links added by JavaScript on click, links inside a search box, and links carrying rel=nofollow all make discovery harder, and Google has treated nofollow as a hint rather than an instruction since 2020.
This is the category the checker above cannot detect, and on most sites it is the largest one. If Search Console says Crawled currently not indexed, nothing is blocking you: Google fetched the page and decided it was not worth storing. Usual causes are near-identical templated pages differing only by a city or product name, paper-thin tag and filter archives, and parameter URLs multiplying the same content. Discovered currently not indexed is different again and usually points at crawl capacity or perceived site quality rather than at the individual URL. Neither is fixed by resubmitting. Consolidate the duplicates, remove or noindex the low-value archives, and make the pages you keep genuinely distinct.
The site: operator is the wrong instrument. Google has said publicly that it is not built for debugging and is not guaranteed to be complete, so a URL missing from site: results is weak evidence of anything. Use it for a rough look, never for a verdict.
Remember the reporting lag. Search Console data is not live, so give a genuine fix time to be recrawled and reprocessed before concluding it did not work.
Worth being explicit, because a green result is a permission check and not a promise. The tool reads the raw HTML response, so it does not execute your JavaScript. It evaluates only the wildcard User-agent group in robots.txt, by simple path prefix, so a Googlebot-specific group or a wildcard pattern inside a rule will not be reflected. It reports the URL it finally landed on, not the individual redirect hops taken to get there. It reports whether a canonical exists, not whether it points somewhere sensible. And it cannot judge whether the content is distinctive enough to be worth indexing, or whether anything on your site links to the page at all. Use it to rule out the mechanical blockers in seconds, then take the questions of quality, duplication and internal linking to a crawler and to Search Console.
If a whole section of the site is stuck rather than a single page, it is almost always systemic: a template, a header rule or a canonical pattern. For help untangling that, see our technical SEO service, or fix the crawl rules yourself with the robots.txt generator and XML sitemap generator.
Indexability is usually a symptom, not the cause. We find what's actually blocking the pages that matter and fix it. Tell us the domain.
Call 0161 399 4659