Collect
Crawl within a defined boundary
The crawler records public site evidence with robots rules, request limits, and separate handling for pages and linked resources. Stored crawl history supports comparisons and reports.
Build 13 / Evidence-led SEO operations
Better SEO decisions
start with better evidence.
An own-site SEO application built around crawling, stored evidence, prioritised issues, reports, and a private operator workflow. The most useful change was teaching the crawler what should count as a page.
Private application
TypeScript
SQLite
Drizzle
Cloudflare Access
Published 26 October 2026 · Repository evidence through 6 September 2026
Greg's take
A convincing dashboard can still be working from the wrong facts.
An SEO tool can generate plenty of recommendations. The harder question is whether the evidence underneath them deserves attention.
This build made that very concrete: images and other non-page URLs could contaminate page analysis. Fixing the evidence layer became more valuable than adding another recommendation. The app needs to explain what it knows, where it came from, and what still needs a human decision.
The system
How the pieces fit together.
Interpret
Keep the source visible
Crawled, imported, inferred, and AI-generated data have distinct source labels. Search Console imports and public competitor content evidence do not justify invented keyword volumes, rankings, or traffic estimates.
Decide
Keep the operator in the loop
The product direction connects evidence to recommendations, review, and measurement. A recommendation is not permission to rewrite or publish the website.
Build journey
From the first useful slice to the latest change.
01 / Working foundation
Close the audit loop
The TypeScript application uses Express, SQLite, and Drizzle to crawl, store, analyse, prioritise, display, compare, and report. Markdown and HTML exports reuse the stored evidence.
02 / Private operation
Protect the working interface
Later build records document the Express application behind Cloudflare Tunnel and Access, with authenticated provider connection panels. The public URL still reaches an Access sign-in screen; that alone does not verify the application behind it or a successful provider sync.
03 / Evidence correction
Classify before analysing
The 6 September change separates pages, resources, documents, redirects, errors, offsite references, and disallowed URLs. Redirect sources remain distinct from their final HTML destinations. Page analysis and health reporting use trusted HTML pages.
Evidence and limits
Repair the inputs before trusting the recommendations.
The latest crawler change includes regression coverage for document types, redirects, robots-disallowed records, and external references. Its build log records 333 passing tests and a passing TypeScript check. Those are recorded results from that change, not a fresh execution during this website update.
Current limits
The repository README and early roadmap predate the later hosting and integration work, so this account uses the subsequent build log and committed source. A configured OAuth panel does not prove a completed connection or current data sync. No ranking improvement or measured SEO uplift is claimed. Older crawl statistics may need a fresh crawl under the corrected classification rules.
Source review: committed implementation and project records at da1147c, reviewed 26 October 2026. Private source, operational data, and credentials are not published with this story.
What the build taught us
The next step follows the evidence.
The strongest lesson is about data quality: a small number of well-supported decisions is more useful than a large queue built on misclassified evidence.
Build something useful
Have a workflow that needs this kind of care?
Start with the problem, the constraints, and what a useful result would look like.