ILCrawler
Crawler/audit system for turning raw crawl, rendered, Lighthouse, and issue evidence into reviewable SEO handoff data.
Stack
FastAPI, Next.js, React, PostgreSQL/Neon, S3-compatible storage, Docker workers, Playwright, Lighthouse
Proof angle
Shows crawler architecture, worker orchestration, evidence modeling, private operator UI, and portable audit output.
Overview
ILCrawler is a private crawl workbench backed by FastAPI, PostgreSQL/Neon, S3-compatible storage, Docker worker profiles, and a Next.js frontend. It keeps raw crawl data, rendered checks, Lighthouse artifacts, issue workflow state, reports, and exports visible enough to operate.
Where this is relevant
Relevant for technical SEO audits, crawl/indexation work, rendered-page checks, Lighthouse evidence, crawler/audit-tool proof, FastAPI plus Next.js systems, and issue handoff workflows.
What it shows
- authorized raw crawls with robots.txt, sitemap discovery, URL normalization, frontier leasing, and crawl limits
- metadata, canonical, heading, robots, hreflang, word-count, TTFB, and link-graph capture
- Playwright rendered audits, screenshots, local Lighthouse evidence, and optional Google PageSpeed enrichment
- issue rows for duplicate metadata/content, broken links, orphan/dead-end pages, canonical problems, redirects, and hreflang issues
- workflow state, ignore rules, schedules, webhooks, API tokens, admin diagnostics, worker drain controls, CSV exports, and branded PDF reports