⚠️ It Looked Simple. Until It Wasn't

You know that moment in the planning workshop. Someone says,

Famous last words in almost every Sitecore Search solutioning meeting I've sat in.

This article dives deep into Incremental Updates and Delta Crawling in Sitecore Search - two powerful but complex features that look simple on paper but reveal important surprises during real implementations.

Before diving in, I recommend reading my previous article on Content Indexing with Sitecore Search for the foundational concepts. This article picks up where that one left off, focusing on the two mechanisms that decide how your index stays in sync: Delta Crawling and Incremental Updates.

By default, a Sitecore Search web crawler does exactly what its name implies - it crawls. All of your configured URLs. Every single one. On every scheduled run.

For a small site, this is fine. But for a Newsroom, product catalog, or large enterprise content hub, a full recrawl can take hours - and run daily doesn’t match the reality of how fast content teams publish and update.

Sitecore Search gives you two levers to tackle this problem:

1. Delta Crawling (Smarter scheduled crawls)

2. Incremental Updates (API-driven, near-real-time pushes)

Historically you had to pick one (because enabling one could disable the other). Community updates suggest this is improving in newer releases for eligible sources - but it’s still worth validating the behavior in your own tenant.

This diagram shows the two common paths to keep the index fresh - sitemap-driven vs event-driven.

Delta crawling is an optimisation of the standard web crawler. Instead of re-crawling every URL on every run, it only re-crawls URLs that have changed since the last run. It does this by reading the lastmod field from your sitemap.

• You use a sitemap or sitemap_index trigger
• Your sitemap includes a reliable lastmod
• Your crawler depth is set to 0

This flow makes it clear why lastmod quality is so important: the crawler decides to crawl or skip purely on change detection.

If your sitemap uses date-only lastmod (for example 2026-05-26), multiple edits on the same day can look identical to the crawler. The end result is a **quiet mismatch **between what editors expect and what the crawler can detect.

Incremental Updates take a different approach: instead of the crawler discovering changes later, you push the updated document to the index through the Sitecore Search Ingestion API .

• In Sitecore Search, open your source under Sources
• Edit Incremental Updates
• Turn on ENABLE INCREMENTAL UPDATES and publish the source

This sequence diagram shows the end-to-end publish-to-index path.

Teams often assume they can just turn both on. Historically, enabling Incremental Updates could disable delta crawling so always validate this behavior in your environment.

A scheduled (or manually triggered) full crawl can overwrite changes made via the Ingestion API if those changes are not represented in the original content source. This is why many teams keep a scheduled full crawl as a safety net - and treat the CMS as the source of truth.

A reliable pattern for SitecoreAI (XM Cloud) + Sitecore Search looks like this:

In practice, the web crawler is powerful because it indexes rendered page output (including components inserted into placeholders) without needing you to model every component field. A middleware-driven incremental update often needs explicit knowledge of which fields to fetch - and that can become fragile as the component library grows.

One alternative (worth prototyping) is using an ingestion approach that creates/updates a document based on a URL using the crawler’s extractor, so you keep the rendered output advantage while still doing targeted updates.

If a full-site publish triggers thousands of events, your middleware can flood the Ingestion API (and hit request caps).

Two common mitigations are:

• Add a Reindex boolean and only trigger incremental updates when true
• Use stateful middleware that pauses incremental updates during bulk publish

This state diagram shows a simple mental model for pausing incremental updates during a bulk publish event:

A lightweight way to align expectations before you estimate custom work:

💬 How many times per day is the same item typically updated?

💬 Do your pages rely on many components / datasources / render-time composition?

💬 Does your sitemap expose lastmod and is it date-only or datetime?

💬 What does a bulk publish look like (volume and frequency)?

💬 Do you need a Reindex flag, stateful middleware, or both?

💬 What's the safety net schedule for a full crawl?

💬 How will you monitor failures in webhook => middleware => ingestion?

💬 Have you budgeted custom implementation work separately from "crawler setup"?

Sitecore Search indexing isn’t hard - but it’s not a one-click configuration either. Delta crawling and incremental updates solve the same problem from different angles. The teams that get this right are the ones that ask the hard questions during solutioning, not during UAT.