Open-Source AI Visibility Tracker: What to Release and What to Keep Private
An open-source AI visibility tracker is worth releasing when it makes AI-answer evidence inspectable, while keeping fragile collection and client-operation infrastructure appropriately controlled.
· 16 min read
Five runs of one prompt can produce five different brand lists, and a tracker that reduces those answers to one visibility number hides the fact that its evidence moved. For founders considering an open-source AI visibility tracker, the useful payoff is a clear release boundary: publish the measurement logic and proof behind results, while treating live collection, reliability operations, and client workflows as product infrastructure.
A recent TechSEO discussion made the case in practical terms. The original builder said their internal tracker collects answers from live AI interfaces rather than model APIs, runs prompts five times per engine, matches brands through an alias list, and treats cited URLs as the most actionable output. Several readers agreed that recurring citations matter more than a noisy mention score; one commenter also identified a crucial distinction: a response that triggered web search is not equivalent to a response generated from model memory.
That is the right starting point for an honest postmortem. An AI-answer tracker should not be positioned as a conventional rank checker. It is a sampling system for probabilistic answers, changing sources, and interface-dependent results.
The candid verdict: open source the evidence layer, not every operational layer
An open-source AI visibility tracker can earn trust and community contributions, but only if the project exposes how it arrived at each metric. Publishing a dashboard with an unexplained 62/100 score will not solve the transparency problem that makes many GEO and AEO tools hard to evaluate.
The better proposition is: anyone can inspect the prompt, run context, captured answer, citations, extraction decisions, and aggregation method. That lets an agency explain why a client was mentioned in ChatGPT on three of five runs but cited in none, or why Google AI Mode surfaced a competitor in one city but not another.
The Reddit builder’s workflow is a sensible minimum:
- Submit the same prompt to live answer-engine surfaces.
- Capture the response users would actually see.
- Extract brand mentions and cited URLs.
- Repeat samples rather than trusting a one-off answer.
- Review ambiguous brand matches.
- Turn recurring cited domains into research or outreach opportunities.
Audra’s view is that the measurement and evidence model should be inspectable. The operational machinery does not necessarily need to be. This distinction follows the same practical principle behind AI search visibility measurement: a visibility finding is only useful when a team can trace it back to what was observed.
Why an AI answer is evidence, not a ranking position
Google’s traditional results have always varied by query, location, device, and time, but answer engines add more variables: retrieval may or may not occur, citations may be present or absent, an interface may carry conversational context, and the answer itself can be rephrased between runs.
That does not make tracking pointless. It changes what a tracker can legitimately claim. Instead of saying “Brand A ranks number two in ChatGPT,” a defensible report can say:
- Prompt: “best project management software for a 20-person design agency”
- Engine and surface: ChatGPT web answer, fresh session
- Locale: United States, English
- Sampling: 5 independent runs on October 26, 2026
- Brand mention rate: 3 of 5 responses, or 60%
- Citation rate: 0 of 5 responses
- Search-trigger rate: 2 of 5 responses
- Most frequent cited publishers: three domains, with counts
Those are observations with a scope. They are not a promise about every user’s next answer.
The distinction is especially important when comparing ChatGPT, Gemini, Perplexity, Microsoft Copilot, Google AI Mode, and Google AI Overviews. Each product has different interfaces, retrieval behaviors, citation treatments, conversation patterns, and access paths. A cross-engine share-of-voice metric can still be useful, but only when it is decomposable into underlying outcomes.
A practical report should preserve separate measures for:
- Brand mention: whether the answer names the brand.
- Brand position: where the first clear mention appears in the response.
- Citation: whether a cited source points to the brand’s own domain.
- Source opportunity: whether a third-party cited URL is relevant to earning coverage.
- Search or retrieval state: whether the interface visibly searched or returned sources.
- Answer sentiment or framing: whether the brand was recommended, merely listed, or described negatively.
- Coverage: the percentage of the defined prompt set that produced an observable answer.
That approach avoids the main failure of composite AI visibility scores: they imply precision while hiding whether a gain came from mentions, owned citations, or just a different sample of prompts. For a broader measurement model, see how to track AI citations and brand mentions.
The collection stack: capture the live answer, retain the context
The source post makes a legitimate product observation: a model API response may not match what a user sees in a consumer AI interface, particularly where live web search, citations, or interface-specific retrieval are involved. A tracker intended to assess answer-engine visibility therefore needs a collection method that can observe the target surface, not merely generate a similar answer through an API.
There are generally three collection paths.
Browser automation with Playwright
Playwright can launch Chromium, Firefox, or WebKit instances and interact with pages. That makes it capable of reproducing parts of a browser journey: opening a fresh context, entering a prompt, waiting for a response, and saving page state or extracted content.
Its strength is observability. A local-first desktop application can preserve screenshots, HTML fragments, text, response timing, browser configuration, and errors alongside each run. Its weakness is fragility: selectors break, consent banners change, logins expire, anti-bot systems intervene, and a “successful” automation can still capture an incomplete answer.
Specialist SERP or scraping providers
Commercial data providers now offer endpoints for Google AI Mode and AI Overviews that return answer text and reference links in structured formats. For example, SearchApi documents fields for AI Mode text and reference links, along with country, language, and canonical location controls.
This path can reduce maintenance for a small team, but it does not remove methodological responsibility. The tracker still needs to log provider, request parameters, collection time, parsing version, and failures. It also needs to make clear that the observation came through an intermediary rather than a locally operated browser session.
Official APIs, where they fit
Official APIs are usually more stable and contractually clearer than browser automation, but may not expose the exact consumer answer surface or live-search behavior a marketer wants to measure. That makes APIs valuable for research and controlled generation tests, not automatically a substitute for real-interface observation.
A robust system can support adapters for all three methods. It should label them rather than mixing their results together. “ChatGPT API response” and “ChatGPT consumer web response” are different observation types.
The metadata that makes scraped answers defensible
Raw answer text without run metadata is difficult to audit. If a client asks why their brand disappeared from Gemini last month, “the score changed” is not an answer. A data model needs enough context to distinguish a genuine visibility movement from an engine change, collection error, or different locale.
At minimum, every run should store:
| Field | Why it matters |
|---|---|
| Prompt ID and exact prompt text | Identifies the question being sampled and prevents silent wording drift. |
| Prompt-set version | Shows whether results changed because the research set changed. |
| Engine, product surface, and collector adapter | Separates Perplexity from Google AI Mode and browser collection from a provider endpoint. |
| Timestamp and timezone | Establishes when the observation occurred. |
| Country, city or canonical location, language | Answer and source selection can vary by market and language. |
| Fresh-session or continuation state | A follow-up prompt is not comparable with a clean first-turn prompt. |
| Search-trigger or citations-present state | Separates retrieved answers from non-retrieved responses. |
| Raw text, source URLs, and snapshot reference | Supports later review when parsing rules change. |
| Parser and brand-dictionary version | Explains extraction changes over time. |
| Status, error, and completion timing | Stops failed or partial captures from becoming silent zeroes. |
The need for location controls is not theoretical. Google AI Mode data providers explicitly expose country, language, and location parameters, while browser sessions can carry their own geographic and personalization signals. An AI visibility report that omits locale should state that it is measuring a generic or provider-defined context, not universal visibility.
Local storage is especially useful here. A desktop tool such as Audra can keep client prompts, captured evidence, audit data, and report inputs on the operator’s machine rather than requiring every raw answer to live in a third-party SaaS database. That is a deployment choice, not a guarantee of compliance; teams still need to manage credentials and personal data responsibly.
Brand matching is a review problem, not just a string-search problem
The original poster found that domain matching misses multi-word brands and that common-word brand names create false positives. That is a familiar extraction problem with high reporting consequences.
Consider a fictional brand named “North Star.” A simple case-insensitive text match may count directional advice, a school, a nonprofit, or a competitor’s product. Domain matching can miss a mention of the brand where the engine cites an independent review site instead of northstar.example.
A more reliable matching workflow includes:
- Canonical brand entity: the name that appears in reports.
- Aliases: abbreviations, old names, product names, punctuation variants, and common misspellings.
- Disambiguation rules: surrounding words, expected category, allowed domains, or exclusion phrases.
- Match confidence: exact, alias, contextual, or manually confirmed.
- Review queue: every low-confidence match and all newly discovered variants.
The same model should identify competitors and publishers. If “Acme” appears in a citation title but refers to a different company, an unreviewed parser can inflate a competitor’s share of voice and produce bad recommendations.
This is why an open-source tracker should publish its entity schema and test fixtures. A public repository can include anonymized example answers with known expected outputs: mention true/false, first position, citation ownership, and confidence. Contributors can then fix parsers without relying on a founder’s hidden dataset.
Repeat sampling: five runs is a baseline, not a law
Running each prompt five times per engine is a practical starting point from the source discussion, not a statistically universal answer. It is enough to expose obvious volatility; it is not enough to establish a stable market-wide estimate for every prompt or engine.
The correct rerun cadence depends on the decision being made. For a monthly client report covering 30 prompts across six engines, five fresh runs per prompt creates 900 observations before retries. That may be appropriate for a high-value competitive study but excessive for weekly operational monitoring.
A useful operating model is:
- Baseline study: 3–5 fresh samples per prompt and engine to establish volatility.
- Routine monitoring: 1 sample per scheduled interval, with raw evidence retained.
- Investigation mode: rerun 3–5 times when a major gain, loss, or new citation pattern appears.
- Prompt-set refresh: review prompts quarterly, and version the set rather than overwriting it.
Aggregate outputs should show both the numerator and denominator. “60% mention rate (3/5)” is more honest than “visibility score: 60.” A rolling window can reduce noise, but it must not erase the daily record. If the last ten runs contain three failures, the report should show that fact instead of averaging only the successful captures.
The commenter’s advice to log search activation deserves to be a first-class field. Averaging a no-search answer with a search-and-citation answer can obscure the exact behavior an SEO team is trying to influence.
Citations create an action queue, but they are not a link-building order
Cited URLs are often the most actionable tracker output because they reveal the publishers and pages being surfaced for a particular question. If the same “best accounting software” listicle is cited in four of five AI Overviews, it becomes a sensible page for editorial analysis, relationship research, or a pitch.
That does not mean every cited site is an outreach target. Teams should classify citations before acting:
- Owned citation: the answer cites the client’s site directly.
- Relevant earned-media opportunity: a third-party review, listicle, trade publication, or resource page where inclusion may be appropriate.
- Reference-only source: an academic, government, documentation, or news page not suited to a placement request.
- Competitor-controlled source: a competitor’s own site or affiliate property.
- Unclear extraction: redirect, broken URL, or incomplete source data requiring review.
This workflow connects AI visibility with traditional SEO fundamentals rather than treating it as a separate magic channel. Better pages, clear entity information, evidence, digital PR, and useful third-party coverage can all affect what sources are available to answer systems. AEO versus SEO is therefore best understood as overlapping work with different evidence and reporting methods.
Avoid promising that obtaining a placement on a cited domain will cause an answer engine to mention a brand. The relationship is directional and contextual, not guaranteed. A tracker should record the hypothesis, the outreach action, and post-action observations without asserting causation it cannot establish.
Where browser scraping breaks—and why open source should say so plainly
Browser scraping can capture the real app experience, but it carries technical, commercial, and policy risk. Playwright itself is legitimate browser automation software; the risk arises from how a collector uses it against specific services and whether that use complies with their rules, technical controls, and applicable law.
The operational failure modes are concrete:
- DOM structures and selectors can change overnight.
- Authentication, CAPTCHA challenges, consent flows, and rate limits can block collection.
- Session history, cookies, account state, and IP geography can change responses.
- An answer may render progressively, leaving a collector with incomplete text or missing citations.
- Provider costs can rise as retry rates, proxy requirements, and anti-bot friction increase.
- Service terms may restrict automated extraction; OpenAI’s Terms of Use, for example, prohibit using automated or programmatic methods to extract data or output except as permitted through its APIs.
No repository license overrides third-party terms. The project documentation should require users to review each provider’s current rules, avoid instructions designed to evade safeguards, respect rate limits, and accept that an adapter may be disabled when the legal or operational risk changes.
This is also why “self-hosted” does not automatically mean “free.” A local browser runner may avoid per-request vendor fees, as one discussion participant noted, but it still has costs: machine time, maintenance, account administration, failed-run investigation, and the opportunity cost of fixing collectors. A hosted commercial tool bundles some of that work; a self-hosted system shifts it to the operator.
An open-source boundary for an SEO auditing product
The right boundary lets the community verify outcomes and improve non-sensitive components without making a product responsible for distributing brittle circumvention infrastructure. The table below provides a practical default.
| Component | Open source? | Reasoning |
|---|---|---|
| Normalized result schema | Yes | Enables portable, auditable datasets across collectors and engines. |
| Prompt-set format and versioning tools | Yes | Helps teams reproduce studies and compare methodology. |
| Brand alias matcher and test fixtures | Yes | Community review improves a high-error extraction task. |
| Citation parser and URL normalization | Yes | Useful shared infrastructure, provided fixtures avoid sensitive data. |
| Score calculations and report templates | Yes | Makes share of voice and mention-rate claims inspectable. |
| Desktop local-storage layer | Usually yes | Supports local-first adoption and independent exports. |
| Collector adapter interface | Yes | Lets users build permitted adapters for their environments. |
| Live collector implementations | Selectively | Release only where terms, stability, and maintenance obligations are clear. |
| Proxy pools, anti-bot workarounds, account automation | No | High fragility and risk; not a responsible community default. |
| Hosted reliability orchestration | Usually no | Includes operations, vendor contracts, monitoring, and support burden. |
| Client workspaces, permissions, and agency workflow | Usually no | Product differentiation lies in reliable delivery, not hidden scoring. |
For Audra, the logical open-source contribution would be an evidence specification: portable run records, transparent metric definitions, fixtures, parsers, and report templates that can accompany local audit results. The desktop application can then focus on dependable auditing across technical SEO, performance, accessibility, links, and AI-answer visibility without requiring a subscription-only black box.
A release checklist before publishing the repository
Before announcing an open-source AI visibility tracker, a founder should be able to answer these questions in documentation, not only in issue comments:
- What does the project observe: APIs, browser interfaces, provider endpoints, or all three?
- Which engines are supported today: ChatGPT, Gemini, Perplexity, Copilot, Google AI Mode, Google AI Overviews, or a subset?
- What exact fields are retained for each run?
- How are fresh sessions, location, language, and prompt versions handled?
- How does the tool distinguish brand mentions from brand-owned citations?
- Which metrics are counts, rates, rankings, or derived scores?
- What happens to failed runs and partial answers?
- How can a user export raw snapshots and reproduce a report?
- Which collector paths depend on third-party services or paid credits?
- Which provider terms or technical restrictions must users evaluate themselves?
A repository that answers these questions may be less flashy than another AI dashboard, but it will be more credible. It gives practitioners a way to build their own AI visibility tracker, compare it with commercial GEO tools, and decide whether the data quality justifies the operating cost.
FAQ
Should I open source an AI visibility tracker?
Yes, if the release exposes the evidence and methodology behind the metrics. The highest-value open-source components are normalized run data, prompt versioning, brand and citation extraction, scoring logic, fixtures, and report templates. Keep infrastructure that depends on fragile browser collection, proxy operations, client management, or commercial vendor agreements private or selectively released.
Can you scrape ChatGPT, Gemini, Perplexity, and Google AI results reliably?
A collector can produce useful observations, but “reliably” should mean monitored, versioned, and failure-aware rather than permanently stable. Interfaces change, sessions vary, and providers may restrict automated extraction. Use collector adapters, retain raw evidence, log errors, and separate browser-sourced results from API or third-party provider results rather than treating them as identical.
What should an AI visibility tracker measure besides a visibility score?
It should report mention rate, first-mention position, citation rate, owned-domain citations, recurring third-party cited URLs, competitor mentions, answer framing, search-trigger state, prompt coverage, failures, locale, and sample count. Every aggregate should link back to raw answers and prompt versions. A score may summarize results, but it should never replace the underlying evidence.
How often should you rerun prompts when AI answers change?
Start with three to five independent samples per prompt and engine to understand volatility. For regular monitoring, one scheduled sample can be enough if the tracker keeps historical evidence. Repeat three to five times when a significant change appears, when an engine changes its interface, or when a team needs confidence before reporting a gain or loss to stakeholders.
Is a self-hosted AI visibility tracker cheaper than commercial GEO tools?
It can be cheaper in direct software fees, particularly for a small prompt set or a local browser-based workflow. It is not cost-free: teams still pay for setup, maintenance, retries, data providers, hardware, and troubleshooting. Compare total operating time and data quality—not only monthly subscription price—before deciding between self-hosted tracking and a commercial platform.