Website Migration Tools in 2026: Crawlers, Redirect Mappers and Diff Checkers We Use

The Screaming Frog SEO Spider crawls 500 URLs for free, and a licence that removes the cap costs $279 per user per year, according to its pricing page on 26 October 2026. For most marketing sites, that one licence covers the three jobs a migration needs a tool for: saving the old site, comparing it with the new one and testing every redirect.

This page is a tool roundup, not a process guide. Our earlier article, Website Migration Without Losing Rankings, covers the redirect map, the rehearsal and the 30-day watch. Here we compare the tools behind each step, using each vendor's own documentation and pricing pages, all read in October 2026.

The migration toolkit at a glance

A migration needs tools for five jobs: finding every old URL, mapping each one to a new URL, loading the redirects into the new platform, checking that they work, and spotting what changed on the pages themselves. No single tool does all five well.

Crawlers: the tool that does most of the work

A crawler is the centre of any migration toolkit, because it is used before, during and after launch. Screaming Frog's own migration tutorial lists the steps: crawl the live site and save it as a baseline, export the URLs to redirect, crawl staging, compare live with staging, detect changes, then crawl the new site on launch and audit the redirects.

Screaming Frog SEO Spider. The free version includes most features but stops at 500 URLs. The paid licence removes the cap, and the maximum crawl then depends on your computer's memory and storage. Crawl comparison and change detection are both in the paid version.

Sitebulb. Sitebulb is the main desktop alternative. Lite audits up to 10,000 URLs and suits occasional users. Pro audits up to 500,000 URLs by default, which can be raised to 2 million, and adds audit comparisons, scheduled audits and over 300 hints. Sitebulb Cloud starts at £95 a month for teams that want to share one crawl.

Ahrefs and Semrush. Both include a site audit inside a wider SEO suite. Ahrefs' free tier, formerly Ahrefs Webmaster Tools, checks over 170 issues and gives 5,000 crawl credits per verified project per month, on an unlimited number of verified sites. Semrush lists over 140 checks and a Compare Crawls report.

Our bias first: we build marketing sites in Webflow, and on a Webflow migration we would use Screaming Frog as the main crawler. It is the tool Google's own site move documentation names for checking redirects, and its list mode is the quickest way we know to test a full old URL list.

Finding every old URL

The redirect map is only as complete as the list of old URLs, and no single source has them all. Google's site move guide suggests starting with the important ones: URLs in sitemaps, pages with high traffic and pages with many links. Screaming Frog adds analytics, Search Console, the CMS database, log files and backlink tools.

The live crawl. A crawl finds every page linked from somewhere on the site. In Screaming Frog, you can also connect Google Analytics and Search Console and turn on crawling of new URLs found there, which picks up orphan pages that nothing links to.

The Wayback Machine. The Internet Archive's CDX server returns every URL it has captured under a domain. Its documentation shows a query that combines a prefix match with collapse=urlkey to return one row per URL, and the output can be JSON. It finds pages deleted years ago that may still have links. It is not complete: when we ran the query on our own domain on 26 October 2026, it returned no captures at all. Small and young sites may simply not be in the archive.

The CMS export. Webflow, like most CMSs, exports each Collection to CSV. That list includes draft and archived items that a crawler never sees, which matters if any of them were once live.

Redirect mappers: from a spreadsheet to near-duplicate matching

Mapping is still mostly a spreadsheet job; tools only draft it.

Near-duplicate matching. Screaming Frog's tutorial describes a method "not built for this purpose": crawl old and new URLs together in list mode with near duplicates enabled, run crawl analysis, and export the pairs. If content is unchanged, each old URL should have one near duplicate with close to 100% similarity. The vendor says the export "won't always be perfect". We would use it for a blog or resource library of hundreds of items and map service and product pages by hand.

Pattern rules. Webflow supports capture groups. An old path such as /oldfolder/(.*) can redirect to /new-structure/%1, and several groups can be combined. Special characters in the old path, including ? and =, must be escaped with a % sign. Webflow also strips a trailing slash before a query string before matching, so an old path written with that slash silently fails.

Loading redirects into Webflow

Webflow's redirect limits decide how the map should be written.

Rule count. Webflow sets no hard limit, but recommends 1,000 redirects at most, because every rule is added to the site's manifest file that browsers download. Wildcards keep the count down.

Rule order. Webflow runs redirects in the order they were created, oldest first. A broad wildcard added early will catch URLs before a later, more specific rule can fire.

CSV import. Redirects can be imported as a two-column CSV of up to 16 MB. A new import removes every existing redirect on the site, and one invalid row fails the whole import. Export the current redirects first, merge, and then import. Redirects need a paid Site plan or a paid Workspace plan.

Locales. Redirects are relative to the root domain and do not apply to localised paths, so /old-url and /es/old-url need separate rules.

Content. The CMS CSV import maps columns to fields, but images imported by URL get no responsive variants, and file fields cannot be imported at all. On our largest Webflow CMS, which holds over 1,200 items, that is the kind of gap we would plan for before the import, not after.

Our WordPress to Webflow migration checklist covers the content side in more detail.

Redirect and status checkers

Once redirects are live on staging or production, every old URL needs testing: one hop, a permanent redirect, a final 200 and an indexable page.

Screaming Frog list mode. Upload the old URL list, turn on Always Follow Redirects and export the All Redirects report. It shows the final address, the number of hops, loops, any temporary redirect in the chain and whether the destination is indexable. Screaming Frog still says each final address should be checked by hand, because Google will likely ignore a redirect to an unrelated page.

httpstatus.io. A browser-based checker for up to 100 URLs at a time, showing up to ten redirects per chain with headers and response times. It suits a launch-day check of the top pages.

Google Search Console. Google's documentation names the URL Inspection tool for single URLs and command-line tools or scripts for large lists. For a domain change, the Change of Address tool tells Google about the move. It works only for domain-level moves, not for http to https or www changes, and it forwards signals for 180 days.

Diff checkers: what changed on the page

A redirect can be perfect and the page behind it can still have lost its title, its H1 or half its internal links. Two kinds of diff catch that.

Crawl comparison. Screaming Frog's URL mapping compares a live crawl with a staging crawl even across hostnames, using a regex. Change detection then lists URLs whose titles, meta descriptions, word count, internal links or crawl depth changed. Sitebulb Pro and Semrush offer similar crawl comparisons. This is the diff we would rely on most, because SEO losses after a migration usually come from changed content and links, not from design.

Visual diff. Playwright's toHaveScreenshot saves a reference screenshot on the first run and compares later runs against it, with thresholds for colour difference and changed pixels. Playwright warns that rendering varies by operating system, browser version and hardware, so baselines should be made where the tests run. BackstopJS wraps the same idea in a report with a diff scrubber and Docker rendering, but its GitHub page says the project needs a new maintainer, which is worth weighing before you adopt it.

Visual diffs suit a platform move without a redesign; with a redesign, every page fails and the crawl comparison is the useful diff.

What this means for your migration tool budget

For a B2B marketing site of under a few thousand URLs, the toolkit we would use is short: a Screaming Frog licence, a free Ahrefs project on the verified site, the Wayback Machine for old URLs, Search Console, and a spreadsheet. The only spend is $279. Sitebulb suits teams that want visual reports or shared crawls.

The tools do not decide where each old URL should go; a person does. Budget the time for that, and do the top pages by traffic and links first. Over twelve months we built more than 20 Webflow sites as a white-label partner to Web Tonic, at two to three weeks per site, and the redirect map is the step we would protect most on any move. Our marketing websites service page explains how we scope a Webflow migration.