AutomationSTATUS: Open Source 1 0

Competitor Tracking Engine

Headless web scraping and change detection engine

A FastAPI-based web scraping and change detection engine. It tracks target competitor websites, detects structural DOM shifts, and generates change summaries.

Inspect GitHub Source Code

01 // THE BUSINESS PROBLEM & SITUATION

Operational Context

Product teams lacked automated visibility into competitor pricing adjustments and unannounced feature releases.

CLIENT / ENVIRONMENT: Built for marketing and product managers needing automated website change tracking.

02 // SYSTEM ARCHITECTURE & SOLUTION

What Was Built

An automated tracking pipeline using Playwright headless browsers to capture page snapshots, compute DOM diffs, and generate alert summaries.

COMPETITOR.MONITOR // SCRAPERSystem Architecture Diagram

CRON

Scheduler

SCRAPER

Playwright

COMPARE

DOM Diff

ALERT

Email Digest

SYSTEMS_ARCHITECTURE_EXPLODER // LAYERED_STACKFULL-STACK SYSTEM DECOUPLING

Frontend Layer

Next.js 16 App Router, React 19, Tailwind CSS, TypeScript

API & Middleware

FastAPI, Pydantic Schema Validation, Rate Limiters

Orchestration

LangGraph Multi-Agent Supervisor & State Machines

Model Providers

OpenAI GPT-4o, Anthropic Claude 3.5, Vision AI

Data Stores

PostgreSQL (Prisma), Qdrant Vector Store, Redis Caching

Observability

LangSmith traces, Pydantic audit logs, Error retries

03 // ENGINEERING CONTRIBUTION & FEATURES

Exact Contribution

Developed Playwright scraping scripts for target web pages
Implemented DOM diffing algorithms to isolate relevant text shifts
Integrated automated email alerts for detected changes

Key Capabilities Built

✓ Headless Browser Scraping

✓ DOM Change Detection Engine

✓ Automated Alert Triggers

✓ Periodic Digest Summaries

DATA_PROVENANCE_TIMELINE // AUDIT_TRACETRACE_ID: #TR-992041

14:22:01.002Source Ingestion

Received webhook payloadOK

14:22:01.045Schema Validation

Pydantic structured field checkOK

14:22:01.210LangGraph Reasoning

Evaluated criteria rulesOK

14:22:01.350Human Audit Check

Confidence threshold evaluationPASSED

14:22:01.480Database Commit

Created audit record in PostgreSQLCOMMITTED

*Illustrative system trace — demonstration data, not a client result.

04 // EXECUTION PIPELINE

Step-by-Step Workflow

STEP 01Scheduled task triggers Playwright scraper.

STEP 02Scraper captures target page content.

STEP 03Engine compares new text against baselines.

STEP 04Alert dispatches if significant shifts are detected.

05 // ENGINEERING LESSONS & OUTCOMES

Technical Challenges Overcome

• Filtering out noise such as dynamic timestamps and banner ads.
• Handling single-page application rendering cycles.

Measurable Outcomes

→ Provided automated visibility into competitor pricing adjustments.

→ Reduced manual overhead for weekly site checking.

Lessons & Engineering Rules

"Combining semantic HTML analysis with visual diffing yields reliable change detection."

NEXT RELATED CASE STUDY

WhatsApp Automation Hub

View Study

PROJECT METADATA

ARSLAN'S ROLEFull-Stack Automation Developer

YEAR & STATUS2024 // Open Source

CATEGORYAutomation

TECHNOLOGY STACK

PythonFastAPIPlaywrightBeautifulSoupCron

GITHUB METRICS

Stars: 1

Forks: 0

Verified GitHub Update: 8/2/2026

Need a similar solution?

If your business faces similar data, automation, or software bottlenecks, let's discuss your requirements.

Start a Conversation