Technical SEO Fundamentals: Crawling, Indexing and a Checklist

Technical SEO is the infrastructure work that lets search engines crawl your site without problems and add the right pages to their index. Content and links decide what you say; technical SEO makes sure Google can see it. A page that cannot be crawled or indexed will not rank, however well it is written.

What does technical SEO cover?

Google Search processes a page in three stages: crawling (crawlers download the text, images and video on the page), indexing (the content is analyzed and stored in Google's index) and serving search results. Technical SEO focuses on removing obstacles in the first two stages. In practice it covers:

• Crawlability: robots.txt, internal link structure, server responses
• Indexability: noindex, canonical tags, duplicate pages, HTTP status codes
• Discovery: the XML sitemap
• Rendering: whether Google sees content built with JavaScript
• Mobile version, speed and structured data

Google's technical requirements are actually short: Googlebot must not be blocked, the page must work (return an HTTP 200 code), and the page must have indexable content. Google also states clearly that meeting these requirements only makes a page eligible for indexing; it does not guarantee crawling or indexing. The job of technical SEO is to make sure these three conditions are really met on every important page and that the site does not send Google conflicting signals.

What is the difference between robots.txt and noindex?

Confusing the two is the most common technical mistake we see. robots.txt manages crawler traffic; in Google's own words, it is not a way of keeping a page out of Google. A URL blocked by robots.txt can still appear in the index without its content being read if other sites link to it.

If you do not want a page to show in search results, use noindex or protect the page with a password. There is a subtle trap here: for noindex to work, the page must not be blocked by robots.txt. If Googlebot cannot enter the page, it cannot see the noindex tag either.

A practical split:

• robots.txt: filter and sort parameter URLs that do not need crawling, cart and account pages
• noindex: pages that should stay crawlable but not appear in the index, such as thank-you pages and expired campaign pages
• Password: content that should never be visible, such as staging environments and client documents

What should an XML sitemap look like?

A sitemap tells Google which URLs you care about. Google's limit is 50,000 URLs or 50 MB uncompressed for a single sitemap; larger sites split their sitemaps and combine them with a sitemap index file. You can submit the sitemap through the Sitemaps report in Search Console or add it to robots.txt with a "Sitemap:" line.

Two details are set up wrongly on many sites. Google ignores the `priority` and `changefreq` values, and uses `lastmod` only when it is consistent and verifiable. A plugin that stamps today's date on every page every day makes that signal worthless. Put only URLs that return 200, are canonical and that you want indexed into the sitemap; redirected or noindexed URLs create conflicting signals.

How do you handle duplicate pages and canonicals?

If the same content can be opened from several URLs, Google chooses one of them as the canonical page. Typical causes are http and https versions, trailing slash differences, filter parameters and a product appearing under more than one category path.

Google's documentation ranks the signals from strong to weak: redirects and rel="canonical" are strong signals, while listing a URL in the sitemap is a weak one. A canonical is a stated preference, not a command. If the tag, internal links and sitemap contradict each other, Google may pick a different URL. The same document also explicitly asks you not to use robots.txt for canonicalization.

The correct setup is: 301 redirects for permanently moved URLs, canonicals for duplicate variants, and always the canonical URL in menus and internal links.

Why do HTTP status codes matter?

Google acts on the code your server returns. Three points stand out in its documentation:

• 301 and 302: both are followed, but a 301 is a strong signal that the target should be canonical and a 302 is a weak one. Use 301 for permanent moves.
• 5xx and 429: these server errors cause Google's crawlers to slow down crawling temporarily. If your server errors often, new content is crawled late too.
• Soft 404: if a page returns 200 but is empty or shows an error message, Search Console flags it as a soft 404. Out-of-stock products and empty category pages are typical examples.

This is also the issue that loses the most traffic during site redesigns: old URLs not being mapped one by one to their new equivalents with 301s. We cover what to plan for when building a new site in our article on SEO-friendly web design.

How are JavaScript sites crawled?

Google processes JavaScript applications in three phases: crawling, rendering and indexing. In the rendering step it runs the page in a headless Chromium browser. Two rules matter. Google can discover links only from `<a>` elements with an `href` attribute; menu items that open only through a click event are not crawled. Google also still considers server-side rendering or pre-rendering a good idea, because the site becomes faster for both users and crawlers, and not every bot can run JavaScript. For more detail, see our article on Google's rendering window.

Are mobile and speed part of technical SEO?

Yes. Google uses the mobile version of a site, crawled with its smartphone crawler, for indexing and ranking. Main content that is hidden or shortened on mobile is missing for Google as well. Google also does not load content that requires user interaction such as scrolling, clicking or typing, so main text behind a "show more" button is risky. For speed, the measure is Core Web Vitals; we cover the thresholds and how to improve them in our Core Web Vitals guide.

Is crawl budget your problem?

For most small and mid-sized business sites, no. Google's crawl budget guide is aimed roughly at three kinds of site: large sites with more than 1 million unique pages whose content changes weekly, sites with more than 10,000 pages whose content changes daily, and sites where a large share of URLs sits in Search Console under "Discovered - currently not indexed". If pages on a 300-page corporate site are not being indexed, the problem is usually content quality or missing internal links, not budget.

How do you run a technical SEO audit?

Search Console is enough for a free start:

1. Page indexing report: shows which pages are not indexed and why. Google stresses that you should not expect every URL to be indexed and that duplicates staying out is normal. Focus only on why your important pages are excluded.
2. URL Inspection tool: shows the index status of a single URL, runs a live test and gives a screenshot of the page as Google sees it. After a fix, you also request crawling from here.
3. Sitemaps report: shows whether your submitted sitemap was read and how many URLs were found.
4. Crawler software: use an SEO crawler that walks the whole site to see status codes, redirect chains, canonical and noindex tags in bulk. Search Console reports only what Google sees; a crawler lists the whole site from your point of view.

Technical SEO checklist

• Important pages return 200 and are not blocked by robots.txt
• Pages that should not be indexed have noindex and are not blocked by robots.txt
• The sitemap contains only canonical URLs that return 200, and lastmod shows the real update date
• Every page has a single canonical URL; duplicates are linked with canonical or 301
• Moved URLs go to the new address in one step, without chains
• Menus and internal links are `<a>` elements with an `href` attribute
• The mobile version carries the same main content as desktop
• The reasons important pages are excluded from the index have been reviewed in Search Console

If you would like to map out a technical crawl and a fix plan for your site together, see our SEO service, or contact us about your situation.