Website Cloner

Duplicates a site locally with its assets and link structure intact.

Language
Python
Version
1.3
Status
Released
View on GitHub

Why it exists

Two recurring needs, both boring, both real. Archiving a site before it is decommissioned, so there is a record of what was actually there rather than what the documentation claims. And having a local copy to work against when the live environment is not somewhere you should be experimenting.

Browser save produces one page with broken paths. Command line mirroring tools are capable but their flag surface is large enough that most people get it wrong and end up with either half a site or a recursive crawl of the whole internet.

What it handles

The interesting work is not fetching pages, it is rewriting them so the copy actually opens. That means walking assets referenced from CSS as well as HTML, resolving relative and absolute paths to the local tree, keeping directory structure so that links between pages still resolve, and knowing when to stop.

Knowing when to stop is most of it. Without a boundary, following links from a site takes you off it within two hops and the clone never finishes.

What it does not do

It copies what a browser is served. A site that assembles itself at runtime from an API produces a shell, because the content was never in the response. That is a fundamental limit rather than a missing feature, and pretending otherwise would make the tool worse by making its output untrustworthy.

It also has nothing to say about whether you should be cloning a given site. That decision belongs to the person running it, along with the copyright and terms of service that come with it.

Status

Released and stable at version 1.3. It does the job it was written for, and I have deliberately not extended it into a general purpose crawler, which is the direction every tool like this drifts and the reason most of them become hard to use.

Read next

Tell me what is breaking.

Migration work, an estate that has outgrown its last design, or a question about the toolkit. All of it is welcome.

Follow the writing

Prefer RSS? Subscribe to the feed.