Build a competitor monitoring agent

You're supposed to know what your competitors are doing.

In practice, you check their site once a quarter, miss the pricing change they shipped in March, and find out from a customer call in August. The customer mentions a competitor's new feature. You nod knowingly and quietly panic when the call ends because you didn't know.

Every product and marketing team has this gap. Most don't fill it because the dedicated tools (Crayon, Klue, Kompyte) are built for big enterprise marketing teams with five-figure annual budgets.

You need a lightweight version that catches the changes that matter without the enterprise price tag. A local-first competitor monitoring agent gets you 80 percent of the value at 5 percent of the cost.

This post is the build.

What manual competitive intel actually looks like

Most teams' "competitive intel process" is some version of:

A Notion page with competitor names and links. Last updated six months ago.

A Slack channel where someone occasionally drops a screenshot. No analysis, just "look at this."

A quarterly meeting where someone presents what they noticed in passing. Limited rigor, limited depth.

A panic-driven research project when a deal is lost to a competitor. Pull together everything you can find about why they won. Conclusions get filed in someone's folder, never referenced again.

The result: you're constantly behind. The competitors that matter shipped changes you missed. The ones you're tracking maybe don't even matter. The customer who tells you about the new feature has more current intel than your competitive analyst.

What an agent does differently

The agent does what a junior analyst would do, but every week, automatically, with no procrastination.

Site monitoring. The agent visits each competitor's key pages on a regular schedule (typically weekly). Pricing pages, product pages, homepage, blog. Saves snapshots.

Change detection. For each visit, the agent compares against the previous snapshot. Detects new pages, modified content, removed content. Flags meaningful changes vs cosmetic ones.

Classification. Each change gets classified. Cosmetic (button color, image refresh). Content (new blog post, updated copy). Pricing (new tier, changed amounts). Product (new feature, removed feature). Positioning (changed tagline, new use case).

Impact assessment. The agent rates each change by likely strategic impact on your business. High impact: pricing change, new competing feature, new geographic market entry. Medium impact: positioning shift, new customer case study. Low impact: blog posts, team page updates.

Weekly digest. Once a week, the agent emails the team a summary of high and medium impact changes. With context, with strategic implications, with suggested actions.

That's the system. Five Avery NXR CRs, two days.

The Avery NXR build

CR 1: Competitor configuration.

"Build a Competitor model with id, name, primary_url, urls_to_monitor (array of {url, type (homepage, pricing, product, blog, other), monitoring_frequency_days}), our_classification (direct_competitor, adjacent, partner_that_could_compete), notes. Build an admin UI at /admin/competitors for managing the list. Start with 5-10 named competitors."

CR 2: Scheduled crawler.

"Build a crawler that runs daily. For each Competitor URL where the last crawl is older than the monitoring_frequency_days, fetch the URL. Respect robots.txt. Use a reasonable user-agent. Add small randomized delays. Store the HTML as a CrawlSnapshot record with timestamp, URL, content_hash, full_html, and extracted_text (cleaned of nav/footer)."

CR 3: Change detection.

"For each new CrawlSnapshot, compare against the most recent prior snapshot for the same URL. Compute: content_diff (text-level diff), structural_diff (new sections, removed sections), and meaningful_change (boolean, true if not just whitespace or trivial changes). Store as Change record linked to both snapshots."

CR 4: Classification and impact.

"For each meaningful Change, use the local model to classify. Prompt provides the diff and the URL type. Output: change_category (cosmetic, content, pricing, product, positioning, other), summary (2-3 sentences), impact_level (high, medium, low), strategic_implication (specific reasoning about why this matters or doesn't), suggested_action (what your team should consider doing)."

CR 5: Weekly digest.

"Each Monday morning, generate a digest of all medium and high impact Changes from the past 7 days. Group by competitor. For each change: the summary, the diff snippet, the strategic implication, the suggested action. Include a one-paragraph executive summary at the top with the top 3 changes of the week. Email to a configured list."

That's the core. The agent runs continuously and the digest arrives every Monday.

The classification step is where the magic happens

Anyone can crawl websites. Anyone can diff HTML. The valuable layer is "is this change strategically meaningful?"

A button color change is cosmetic. A new pricing tier is pricing. A new product line is positioning. A new use case page is content. Each requires different attention.

The local model handles this classification competently if you give it examples of what matters to your specific business. The few-shot examples in the prompt should include:

3 examples of high-impact pricing changes and why they mattered.

3 examples of high-impact product changes and why they mattered.

3 examples of low-impact cosmetic changes that should be ignored.

The model extracts the pattern. New changes get classified consistently.

For your business, tune the impact assessment to your strategic priorities. A pricing change matters more if you sell into the same segment. A new use case page matters more if you've been positioning into that use case yourself.

Real outcomes from a deployment

A marketing team at a SaaS company tracking 8 competitors:

Before the agent: spotty competitive intel. They missed two major pricing changes from competitors in 2025 that directly affected their own deal dynamics.

After the agent: weekly Monday digest. Caught a competitor's pricing reduction within 3 days. Adjusted positioning before losing two deals that were in flight. Direct revenue impact attributable to the agent in the first quarter: real money.

The team's competitive intelligence went from "we should pay more attention to competitors" to "we know what they're doing." The product team started treating the competitive landscape as a real input to roadmap decisions.

The build took 16 hours of engineering work. The ongoing maintenance is essentially nothing (occasional adjustments to the competitor list and the classification prompt).

The legal considerations

Web scraping has nuances. The relevant rules:

Stay on public pages. Don't try to access content behind login walls. Don't try to bypass paywalls.

Respect robots.txt. If a site says don't crawl certain paths, don't crawl them.

Don't hammer the server. Small randomized delays between requests. Respect rate limits if specified.

Use a clear user-agent identifying your crawler. Some teams use "research-bot/1.0 from company.com" so competitors can identify and contact you if there's an issue.

Don't reproduce competitor content. Snapshots are for internal analysis. Don't republish competitor content publicly.

This is legitimate competitive intelligence. It's the same thing any analyst would do manually, just automated. Stay on the right side of the legal line and the practice is well-established.

What this doesn't replace

Worth being explicit about the limits.

Direct customer intel. Talking to your own customers about why they chose you over a competitor (or why they're considering switching) is higher signal than web scraping. Don't replace this with automation.

Trade shows and analyst calls. The qualitative intel from talking to industry analysts, attending competitor sessions, and walking the booth at conferences gives you context the agent can't.

Employee referrals from competitor companies. People who've worked at competitors have ground truth. Engage with them ethically.

Strategic interpretation. The agent surfaces changes. Humans interpret what they mean for strategy. Don't let the digest replace the strategy conversation.

The agent fills the gap of "weekly mechanical surveillance that nobody's currently doing." It doesn't replace the higher-quality intel channels.

When to use Crayon, Klue, or Kompyte instead

The dedicated competitive intel platforms add value beyond web scraping. They include:

News monitoring across press releases, news outlets, social media, podcasts.

Sales enablement features (battle cards, competitive plays, win/loss analysis).

Integration with sales tools so the intel surfaces in deal context.

Analyst-curated insights and competitor profile pages.

If your team has the budget and the headcount to use these platforms well, they're more comprehensive. The custom agent is the answer for teams that can't justify the enterprise spend.

The hybrid pattern also works: use the custom agent for the always-on web monitoring, use the enterprise tool for the deeper analysis on key deals.

The cost comparison

Crayon and similar tools: real annual budget for an enterprise marketing team.

Custom local agent build: a couple of days of engineering work, hardware you already have (the agent runs on a workstation), enrichment API costs are minimal because you're just crawling websites.

Recurring cost: essentially zero.

For a team that needs the basic surveillance but can't justify the enterprise tool, the custom agent is the obvious choice.

The privacy angle

This one's different from other agents in the local-first story. Competitor monitoring doesn't typically involve sensitive customer data.

What it does involve: your competitive analysis, your strategic interpretations, your team's thinking about how to respond. That intel is valuable to your competitors if it leaked.

A local-first agent keeps it on your infrastructure. The cloud-based competitive intel tools have their own data, and your usage patterns are visible to them in some way.

For teams that consider their competitive analysis a competitive advantage in itself, local-first is the right architecture.

The v1.1 additions

Once the core is running, the natural extensions:

Press release monitoring. Watch your competitors' press release feeds. Different content type but same change detection pattern.

Job posting monitoring. New roles at a competitor often signal strategic direction. Watch their careers page.

Social media monitoring. Track competitor LinkedIn pages, Twitter accounts, executive posts. Different APIs but same pattern.

Product update monitoring. If competitors have public changelogs or release notes, monitor those for feature releases.

Customer review monitoring. Watch G2, Capterra, Trustpilot for competitor reviews. Sentiment trends are valuable intel.

Conference talk monitoring. Watch for your competitors' speakers at industry events. Their talks often reveal positioning.

Each adds another competitive intelligence vector. Each is a focused CR set on top of the core.

The strategic value

Competitive intelligence is one of those activities that nearly every company knows they should do better but doesn't because the cost-effort calculus has been wrong.

In 2026, the math works for custom builds. A 16-hour engineering investment gives you weekly competitive intelligence indefinitely. The ROI compounds the longer you run it.

The teams that have this running ahead of their competitors get a real edge. They see pricing changes earlier. They notice positioning shifts. They catch the competitive moves that affect deals before deals are lost.

If your company's competitive intel is "spotty," this is one of the highest-leverage automations you can build.

avery.software