Real-Time Monitoring Systems for Prices, News and Competitors

An e-commerce client told me: 'Every morning at 8 I open my main competitor's site and check their prices. If they've dropped, we drop too. Can we automate this?'. Yes, I said. Two weeks later the system was live: 350 products checked every 10 minutes, Telegram alerts on changes. Sharing what I learned.

Architecture

• Scheduler — fires scraping jobs every X minutes (cron + Redis queue)
• Scraper workers — run in parallel, one per product/URL
• Storage — Postgres holds current and previous values
• Diff engine — compares new vs old, alerts when threshold exceeded
• Notification router — Telegram, email, Slack, webhook
• Dashboard — Next.js admin panel with history charts

Price monitoring — the actual code

# scraper.py
from selectolax.parser import HTMLParser
from curl_cffi import requests

def check_product(url: str) -> dict:
    r = requests.get(url, impersonate="chrome120", timeout=15)
    tree = HTMLParser(r.text)

    price_el = tree.css_first("[data-testid='price']")
    stock_el = tree.css_first("[data-testid='stock']")

    return {
        "url": url,
        "price": parse_price(price_el.text()),
        "in_stock": "stock" in stock_el.text().lower(),
        "scraped_at": datetime.utcnow(),
    }

# diff_engine.py
def detect_changes(old, new):
    changes = []
    if abs(new["price"] - old["price"]) > 0.01:
        pct = (new["price"] - old["price"]) / old["price"] * 100
        changes.append({
            "type": "price",
            "from": old["price"],
            "to": new["price"],
            "pct": pct,
            "severity": "high" if abs(pct) > 5 else "low",
        })
    if old["in_stock"] != new["in_stock"]:
        changes.append({
            "type": "stock",
            "from": old["in_stock"],
            "to": new["in_stock"],
            "severity": "high",
        })
    return changes

Avoiding alert fatigue

First version sent alerts on every change. After one day the client said 'turn this off'. New rules: price change under 5% — dashboard only, no alert. Stock status change — alert. Same product already alerted within 24 hours — skip. These rules took us from 100 alerts to 6 per day.

News monitoring — different approach

For prices, diffing is simple. For news, content shifts and diffing doesn't help. AI handles this: I send the headline + first paragraph to GPT-4o-mini and ask 'is this news relevant to {X company}? If yes, why, briefly.' Positive answer → alert. A PR team uses this for competitor and industry news tracking.

Notification channels

Telegram bot is the fastest. Email via Resend (developer-friendly), Slack via incoming webhook. Each client has their own rules: P0 severity → all channels, P1 → Slack only, P2 → dashboard only. I added these to the admin panel so the client can adjust without calling me.

Scaling — when you hit 1000+ products

First version checked 100 products sequentially in 25 minutes. At 350 it ballooned to 90 minutes — no longer real-time. Fix: Redis queue with 10 parallel workers. Now 350 products finish in 4 minutes. For 2,000 products I use 20 workers, each on its own rotating proxy IP. The bottleneck is network latency, not CPU.

Client outcomes — real numbers

• E-commerce: matched a competitor discount within 12 minutes of them posting it, same-day sales jumped 18%
• Construction firm: monitors tender sites, the team hears about new tenders within 2 minutes of publication
• Real estate: 'new listing' tracked across 3 sites, agents call clients 30 minutes ahead of competing agents
• PR agency: client's brand name tracked across 5 local news sites, negative coverage triggers an immediate response prep

If you want a monitoring system for your business — prices, competitors, news, any public data — get in touch with the use case.