In this post, we focus on measuring the second question. I built a ground-truth benchmark from Amazon’s Shopping Queries Dataset (ESCI), where human raters labelled which brands are relevant to each query, and compare share of voice against agreement with those judgements. The output is a dashboard that ranks brands both ways and surfaces where the two disagree.

The dashboard is live at https://princy2310.github.io/esci-brand-study/ and the full source is in the esci-brand-study GitHub repository.

Solution overview

The project is split into two parts: a Python step that samples ESCI and derives aggregates, and a Next.js dashboard that renders them.

1. precompute.py pulls rows from the ESCI dataset through the Hugging Face datasets-server API, filtered to the US locale.

2. It cleans the brand field, rolls product-level relevance judgements up to brands, and derives the competitive sets, share-of-voice figures, and substitute pairs.

3. It writes the aggregates to a single JSON file, which I commit to the repository.

4. The dashboard reads that file at build time, validates it against a Zod schema, and renders four views.

We commit the aggregates rather than fetch them at runtime because the datasets-server API rate-limits anonymous access to 500 requests per 5-minute window. Committing the output also keeps the numbers stable between page loads.

Why ESCI

This exercise needs a dataset where brands are labelled and relevance is judged by people, and ESCI (Reddy et al., arXiv:2206.06588, Apache-2.0) is the one that fit. It contains around 130,000 shopper queries and 2.6 million relevance judgements. Each judgement is a human rating of how well a product matches a query, using four labels:

Every product carries a brand, so the judgements roll up from products to brands. For the query revent 80 cfm, raters judged Aero Pure, Broan-NuTone, Delta Electronics, Homewerks and Panasonic. That gives me a competitive set with a human verdict on each brand.

Sampling the US slice: 10,746 rows across 2,770 queries. Of those, 998 are competitive, meaning more than one brand was judged against the same query. These competitive sets are my unit of analysis.

The two metrics

Computing two numbers per brand.

• Share of voice is the share of competitive queries a brand appears in. This is the presence metric the AEO tools report.

• Agreement is the share of a brand’s judgements that raters labelled Exact. This is the correctness metric the ground truth makes possible.

Ranking the brands by each produces two orderings. In this sample they barely relate: the Spearman rank correlation between them stays within 0.12 of zero at every threshold I tested. Share of voice does not predict agreement.

The individual cases are clearer than the correlation. These brands rank high on visibility and low on agreement:

A visibility report places these brands near the top. A correctness report places them near the bottom.

Why use query floor

Agreement is not comparable across brands of different breadth. A brand judged against a single query cannot be wrong on average, so it scores 100% agreement mechanically. In this sample, brands appearing on one query average 79.4% agreement, with half at a perfect 100%, against 70.2% for brands on nine or more queries.

When left unfiltered, single-query brands filled the top of the agreement ranking and pushed the correlation slightly negative on exposure alone. So we require a brand to appear against at least five distinct queries before it enters the ranking, which removes the effect. The full threshold sweep ships in the data.

Cleaning the brand field

The product_brand field needed cleaning before the aggregates held up. precompute.py handles three cases:

• Case variants. 1MORE and 1More are one company. Left split, they divide the judgements and understate agreement. I collapse them on a case-folded key.

• Product titles. Some sellers fill the field with keyword-stuffed descriptions rather than brand names. I flag these and exclude them by default.

• Unresolved aliases. A non-Latin name with a bracketed romanisation folds cleanly, so コールマン(Coleman) becomes Coleman. A bare one does not, so I exclude キヤノン for Canon from the substitute graph, where it would otherwise appear as a brand paired with itself.

Of 3,928 brands in the sample, 602 appear against three or more queries. The rest are single-appearance marketplace sellers.

Scoring a model against the benchmark

With the ground truth in place, I scored a model against it. I put 120 competitive queries to Claude Sonnet 4.5 with no tools and no web search, so the answers reflect what the model itself associates with each query, then matched the brands it named to the human judgements.

The headline is the last row. When the model names a brand that ESCI also judged, it is Exact 83% of the time. It is accurate on the overlap. But it recovers only a quarter of the full Exact set, and it names the single leading brand 40% of the time, so it is incomplete against the whole picture.

One number needs care. The model names about 11 brands per query, and ESCI had judged only about 2 of them, because ESCI only labelled products that appeared in Amazon search results. So 82% of the model’s named brands are not in ESCI’s judged set for that query. That is mostly real brands outside ESCI’s coverage, not inventions, and it is why raw precision across all named brands is 14%. We report precision over the brands ESCI actually judged instead, which removes the coverage gap. Reading the off-corpus rate as a hallucination rate would be wrong.

The full 120-query run cost about 12 cents.

Prerequisites

• Node 20 or later

• Python 3.10 or later, only if you want to regenerate the aggregates or run the model scoring

• An Anthropic API key, only for the model scoring

Walkthrough

Clone the repository and run the dashboard:

git clone https://github.com/princy2310/esci-brand-study.git
cd esci-brand-study/dashboard
npm install
npm run dev

Open http://localhost:3000. The dashboard loads the committed aggregates, so it runs without any dataset access or API keys.

To regenerate the aggregates from ESCI:

cd ../scripts
python3 precompute.py --rows 20000 --locale us --out ../dashboard/public/data

The sampling backs off through the rate limit, so a full run takes several minutes. I also wrote verify.py, which checks the output against the source and confirms the figures quoted here.

To run the model scoring, put an Anthropic key in .env and run the scorer:

echo "ANTHROPIC_API_KEY=sk-ant-..." > .env
python3 score_model.py --limit 120

It checkpoints after every query, so an interrupted run resumes without repeating calls.

Limitations

• Temporal. ESCI reflects Amazon search behaviour up to its 2022 publication. A model naming a product released after 2022 is hitting a gap in the dataset, not making an error. This is the caveat most likely to produce a wrong conclusion.

• Relevance is not market share. Exact means a product matched the query intent, not that it sells well. A brand can be relevant and obscure.

• Amazon-shaped intent. These are product searches, not open-ended advice questions, so the findings apply to purchase-adjacent prompts.

• No category column. ESCI labels relevance per query and product, so the unit of analysis is the query.

Conclusion

We built a correctness baseline for AI brand recommendations from human relevance judgements, showed that share of voice and agreement rank brands differently on the same data, and scored a model against it. Visibility tools answer whether a brand appears; this answers whether it belonged. Claude Sonnet 4.5 is accurate when it names a judged brand but recovers only a quarter of the human-relevant set, and finds the leading brand 40% of the time.

In this post, I will show you how to deploy Market Intel Agent, an autonomous research agent that runs this workflow on a schedule. The agent searches across community, developer, and web sources for competitor and market signal, uses Anthropic’s Claude Sonnet 4 on Amazon Bedrock to synthesize findings, and delivers a formatted digest by email through Amazon SES. It is built with the Strands Agents SDK, runs on Amazon Bedrock AgentCore, scales to zero when idle, and supports both on-demand deep dives and daily incremental reports.

The full source is available in the market-intel-agent GitHub repository.

Solution overview

The solution has three entry points that converge on the same research agent: an on-demand request from the web UI, a scheduled daily run, and a direct invocation for testing.

The following describes the request flow:

1. A user opens the web UI and submits a product name, focus areas such as adoption or security, and optionally a list of competitors to compare.

2. Amazon API Gateway routes the request to an AWS Lambda function (lambda_api), which writes the subscription to the MarketIntelSubscriptions Amazon DynamoDB table and checks Amazon SES verification status for the supplied email address. If the address is unverified, the function triggers a verification email.

3. When the user chooses Send First Report Now, the API function invokes the MarketIntelWorker Lambda function asynchronously and returns immediately. The agent takes one to two minutes to run, which exceeds a reasonable synchronous request timeout, so the API does not wait for it.

4. For scheduled delivery, an Amazon EventBridge rule fires daily at 08:00 UTC and invokes lambda_trigger. That function reads active subscriptions from DynamoDB and invokes the agent once per subscription in daily mode.

5. The agent runs on Amazon Bedrock AgentCore, invoked through the bedrock-agentcore API (invoke_agent_runtime). It executes a four-step research pipeline, calling web research tools to gather signal and Claude Sonnet 4 on Amazon Bedrock to plan and synthesize.

6. The agent composes an HTML digest and sends it through Amazon SES to the subscriber.

Because every compute component is either Lambda or an AgentCore runtime, there is no always-on infrastructure. The solution costs nothing while idle.

Note that the repository provisions the DynamoDB table and EventBridge rule through a setup script, but the Lambda functions, API Gateway endpoint, and the EventBridge target wiring are deployed separately. The walkthrough covers this ordering.

How the research pipeline works

The agent (agent.py) does not issue a single “research my competitors” prompt. It runs four explicit stages, which keeps gathering separate from synthesis:

Step 1, Discover. Calls search_competitors_and_market and search_news_and_announcements to find competitor launches, market reports, and analyst coverage.

Step 2, Listen. Calls search_community_sentiment, search_pain_points, and search_security_and_conferences to surface what users are actually saying, in their own language.

Step 3, Go deeper. Uses fetch_page_content to read two or three promising pages in full. Search snippets indicate that a page is relevant without revealing what it says, so this step is what produces direct quotes and specific figures rather than paraphrase.

Step 4, Compose and send. Builds the HTML digest and delivers it through send_digest_email.

The agent supports two modes. Full analysis performs a comprehensive pass across all sources, and is appropriate when you first start tracking a product. Daily mode reports only what changed in the previous 24 hours, which is what makes a recurring digest worth opening.

Multi-source research with normalized recency

The research layer (tools/web_research.py) queries community, developer, and web sources. Each exposes different date semantics, so the tool applies per-source filtering rather than a single global window:

Sorting Stack Overflow by activity rather than creation date is deliberate: a two-year-old question receiving new answers this week is a live signal, not a stale one.

All results carry dates and are deduplicated before reaching the model. Without deduplication, the same launch announcement arrives from five sources and repetition is misread as importance.

Digest composition

The digest (tools/email_digest.py) renders a responsive HTML email:

• A top signal block containing the single most important finding

• Fixed cards for New Launches and Industry Pulse

• Dynamic cards generated from the subscriber’s chosen focus areas

• An optional head-to-head comparison table when competitors are specified

Requiring a single top signal acts as a forcing function. Asking a model to summarize produces several paragraphs of equal weight; asking for the most important finding produces a judgment.

Prerequisites

Before you begin, make sure you have:

• An AWS account with permissions for Amazon Bedrock, AWS Lambda, Amazon DynamoDB, Amazon EventBridge, and Amazon SES

• Model access enabled for Anthropic Claude Sonnet 4 in Amazon Bedrock in your Region

• A sender email address verified in Amazon SES

• Python 3.9 or later. The codebase uses builtin generic type annotations such as list[dict], and the repository does not pin a version.

• The AgentCore starter toolkit, installed with the project dependencies (bedrock-agentcore and bedrock-agentcore-starter-toolkit). The agent itself is built on strands-agents.

• Optionally, a SerpAPI key for Google-quality search results. The free tier provides 100 searches per month; without a key the agent falls back to DuckDuckGo.

If your SES account is in the sandbox, both sender and recipient addresses must be verified.

Walkthrough

Step 1: Clone the repository and install dependencies

git clone https://github.com/princy2310/market-intel-agent.git

cd market-intel-agent

python -m venv .venv && source .venv/bin/activate

pip install -r requirements.txt

Step 2: Configure the deployment

Update the following values before deploying:

The agent runtime ARN does not exist until after the first deployment, which creates a chicken-and-egg ordering. Deploy the agent first, capture the ARN, then configure the Lambda functions. Only lambda_api hardcodes the ARN; the worker and trigger functions read it from the environment, so those two can be configured without editing code.

Set the optional search key as an environment variable:

export SERPAPI_KEY=your_key_here

export AWS_REGION=us-east-1

Step 3: Deploy the agent

agentcore configure --entrypoint agent.py --name market_intel_agent --non-interactive --create

agentcore launch

Note the runtime ARN from the output and use it to configure the three Lambda functions as described in the previous step.

Step 4: Provision the supporting infrastructure

The setup script creates two resources and is idempotent, so it is safe to re-run:

./setup_infra.sh

Specifically, it creates:

• The MarketIntelSubscriptions DynamoDB table, keyed on subscription_id with on-demand (PAY_PER_REQUEST) billing

• The market-intel-daily EventBridge rule with the schedule expression cron(0 8 * * ? *)

The script does not deploy the Lambda functions, create the API Gateway endpoint, or attach the EventBridge rule to a target. Deploy the three functions from the lambda_api, lambda_worker, and lambda_trigger directories, expose lambda_api through API Gateway, and add lambda_trigger as the target of the market-intel-daily rule.

Step 5: Test the agent

Invoke the agent directly to confirm end-to-end delivery before wiring up the UI:

agentcore invoke ‘{”product”: “Your Product”, “dimensions”: [”adoption”, “security”], “email”: “you@company.com”, “mode”: “full”}’

The agent runs for one to two minutes. A digest should arrive at the specified address. If nothing arrives, check that the recipient is verified in SES and review the AgentCore logs in Amazon CloudWatch.

Step 6: Enable scheduled delivery

The EventBridge rule invokes lambda_trigger daily at 08:00 UTC. To change the schedule, update the rule’s cron expression. Each subscription in DynamoDB produces one digest per run in daily mode.

Cost considerations

The solution scales to zero, so cost tracks usage rather than uptime. At approximately 50 active subscriptions, expect roughly $15 to $25 per month, with Amazon Bedrock model invocation accounting for most of it. Lambda, DynamoDB, EventBridge, and SES contribute marginally at this volume.

To control spend, reduce the number of pages fetched in step 3, run daily mode rather than full analysis on a schedule, or lower the frequency of the EventBridge rule.

Cleanup

To avoid ongoing charges, remove the resources you created:

# Delete the DynamoDB table

aws dynamodb delete-table --table-name MarketIntelSubscriptions --region us-east-1

# Remove the EventBridge target, then the rule

aws events remove-targets --rule market-intel-daily --ids 1 --region us-east-1

aws events delete-rule --name market-intel-daily --region us-east-1

# Delete the worker Lambda function (the name lambda_api invokes)

aws lambda delete-function --function-name MarketIntelWorker --region us-east-1

Also delete the API Lambda function and trigger Lambda function under whatever names you deployed them with, remove the API Gateway API, and remove any SES verified identities you no longer need. Delete the AgentCore runtime through the AgentCore CLI or the Amazon Bedrock console.

Because the repository does not script Lambda or API Gateway deployment, the names of those resources depend on how you deployed them. MarketIntelWorker is the exception: lambda_api invokes it by that exact name, so it must match.

Limitations

Be aware of the following when evaluating results:

• Search quality bounds output quality. Without a SerpAPI key, the agent falls back to DuckDuckGo, which offers weaker recency filtering.

• Recency is best effort. Sources expose inconsistent date metadata, so a 24-hour window is approximate rather than guaranteed.

• Synthesis compresses. A model reducing dozens of results to a handful of cards will omit detail. The digest links sources so findings can be verified.

• Coverage is public signal only, weighted toward developer and early-adopter communities. That is a real population but not necessarily a representative one.

• Deployment is partially manual. The setup script covers the table and schedule; Lambda, API Gateway, and the EventBridge target are left to you. Packaging the whole stack as infrastructure as code would make this reproducible.

Conclusion

In this post, I deployed an autonomous research agent that replaces a recurring manual workflow with a scheduled, reproducible one. The design separates gathering from synthesis across four explicit pipeline stages, normalizes recency across 11 sources with differing date semantics, and uses asynchronous invocation so that a two-minute job does not block an HTTP request.

The pattern generalizes beyond competitive research. Any workflow that involves collecting scattered signal on a cadence and summarizing it for a human reader maps onto the same architecture: scheduled trigger, multi-source retrieval, model synthesis, and a delivery channel the recipient already checks.

Leave a comment

Share

Subscribe now

In this post, I show you how to deploy AccessDesk, an IT self-service portal that automates routine access decisions and escalates only the requests that genuinely need review. The solution combines a deterministic rules engine for authorization decisions with a conversational assistant powered by Anthropic’s Claude on Amazon Bedrock for tool discovery and employee lookups. It runs on Amazon ECS with AWS Fargate, stores state in Amazon DynamoDB and sends tokenized approval emails through Amazon SES.

A key design decision, covered in detail below, is that the language model never makes the access decision. Understanding why is the most transferable part of this architecture.

The full source is available in the access-desk GitHub repository.

Solution overview

AccessDesk exposes two portals, an employee self-service catalog and a manager approval dashboard, both served by a single containerized Express application.

The following describes the request flow:

1. An employee browses the software catalog in the employee portal and requests a tool.

2. The application invokes the rules engine, which evaluates the request against tool policy and the employee’s job level. This step is deterministic code with no model involvement.

3. If the employee’s job level appears in the tool’s autoGrantJobLevels list, the access agent grants the request immediately and records the decision. No human is involved.

If the employee is not eligible, or the tool carries a requiresApprovalAlways flag, the agent generates a cryptographically random approval token, records the escalation reason, and identifies the designated approver.

1. Amazon SES sends the approver an email containing tokenized approve and reject links.

2. The approver acts, either from the email link or from the manager portal queue. The application applies the decision to Amazon DynamoDB using a conditional update.

3. An AWS Lambda function, invoked every 15 minutes by Amazon EventBridge, re-sends the approval email for requests still pending beyond the configured follow-up window, up to three times.

Separately, both portals include a chat assistant backed by Claude on Amazon Bedrock. The assistant answers questions about the catalog, eligibility, and pending requests, and performs employee lookups for managers. It has no authority to grant access.

Architecture components

The CDK stack provisions:

• Amazon DynamoDB: a single TicketsTable with composite PK/SK string keys, plus a global secondary index keyed on status and createdAt so the manager queue can query pending requests efficiently without a table scan

• Amazon ECS on AWS Fargate: a task definition with 512 MiB memory and 256 CPU units, exposing container port 3000, running behind an Application Load Balancer with a desired count of 1

• AWS Systems Manager Parameter Store: configuration held outside the container image: the access catalog, default approver, follow-up window, provider type, SES sender address, and portal base URL

• AWS Lambda and Amazon EventBridge: a reminder function on a 15-minute schedule that re-notifies approvers about stale pending requests

• AWS IAM: ssm:GetParameter* scoped to the access-desk/* parameter path, plus bedrock:InvokeModel and SES send permissions. DynamoDB access is granted through table.grantReadWriteData, so it is scoped to the table.

Holding configuration in Parameter Store rather than environment variables means the access catalog and approver assignments can change without rebuilding or redeploying the container.

Why the model does not decide

It is tempting to hand the whole problem to a language model: accept a natural language request, return a decision. AccessDesk deliberately does not do this, for three reasons.

Determinism. The same request must produce the same outcome every time. An authorization decision that varies with sampling is not a policy. Model output shifts measurably with phrasing and configuration, which is acceptable for a recommendation and unacceptable for an access grant.

Auditability. When someone asks why a request was denied, the answer must be a rule rather than a rationalization. The rules engine emits a human-readable reason with every decision, for example Job level ‘intern’ not eligible for auto-grant of Camtasia. That string is generated by policy evaluation, so it is precisely and permanently accurate about why the decision occurred.

Prompt injection. As soon as model output authorizes an action, user-supplied request text becomes an attack surface. A rules engine reading a job level from DynamoDB cannot be argued with. A model reading a free-text justification field can.

The resulting principle: use the model for ambiguity, keep it away from authority. Tool discovery, phrasing, and explanation are ambiguous problems where a model excels. Authorization is not, and should not be solved probabilistically.

How the components work

Rules engine

rules-engine.js performs ordered policy evaluation:

1. Check the tool’s requiresApprovalAlways flag. Tools such as Adobe Creative Cloud always require human review regardless of requester.

2. Compare the employee’s job level against the tool’s autoGrantJobLevels list.

3. Return a structured decision containing the outcome, reason, approver ID, and approver name.

Access agent

access-agent.js acts on the decision. It either provisions access immediately, or generates an approval token, persists the escalation with its reason, and triggers notification to the approver.

Conversational assistant

The /api/chat endpoint and llm.js implement the assistant using Claude 3 Haiku on Amazon Bedrock, configurable through the BEDROCK_MODEL_ID environment variable. The endpoint supplies the model with the software catalog, the employee profile, pending requests, and lookup results as context.

For employees, the assistant handles tool discovery. A request such as “I need to record my screen” surfaces Camtasia along with eligibility context. It also answers availability and approval-status questions. For managers, it performs employee lookups by name, returning job level, department, email, and software cost, and will reason about policy when asked whether a specific request should be approved.

Haiku is chosen for latency and cost. Catalog questions do not require a frontier model.

Security considerations

Approval-by-email is convenient and has real sharp edges. The implementation addresses them as follows:

• Token strength and lifetime. Approval tokens are 32-byte cryptographically random hex strings with a 72-hour expiry, long enough to resist guessing and short-lived enough that a forwarded email stops working.

• Concurrency. Two approvers acting on the same request simultaneously must not both succeed. For transitions out of pending_approval, the data layer applies a DynamoDB ConditionExpression asserting the status, so the write fails if another approver already acted. Exactly one action is applied.

• State machine validation. Status transitions are validated, so a request cannot skip stages.

Two limitations to plan for before production use. First, possession of the token is sufficient to act, so a forwarded email is a transferable capability until expiry; a hardened deployment should require the approver to be authenticated rather than merely to hold a link. Second, job levels come from the application’s data layer rather than from an identity provider. A production deployment should read identity and role from your existing directory, such as through SAML or SCIM integration, rather than maintaining a parallel copy.

Current gaps

One thing is deliberately left incomplete in the reference implementation.

Auto-grant records a decision rather than provisioning a license. The agent writes the approval and updates the employee’s granted-access list but connecting that to real entitlement assignment is per-tool integration work against each vendor’s API. That is the substantial remaining work for a production deployment, alongside reading identity from a real directory rather than the local data layer.

Prerequisites

Before you begin, make sure you have:

• An AWS account with permissions for Amazon Bedrock, Amazon ECS, Amazon DynamoDB, Amazon SES, AWS Lambda, Amazon EventBridge, and AWS Systems Manager

• Model access enabled for Anthropic Claude 3 Haiku in Amazon Bedrock in your Region

• A sender email address verified in Amazon SES

• Node.js 20 or later and the AWS CDK installed. The container image builds from node:20-alpine.

• Docker installed and running. The stack uses ecs.ContainerImage.fromAsset, so CDK builds the image locally at deploy time.

Walkthrough

Step 1: Run locally

Local development uses an in-memory store with JSON file persistence, so state survives restarts and you can evaluate the full workflow without provisioning AWS resources. Email events are logged to the console instead of being sent.

git clone https://github.com/princy2310/access-desk.git

cd access-desk

npm install

npm run dev

Open the employee portal and submit a request for a tool your test user is not eligible for, to observe the escalation path. The chat assistant requires AWS credentials with Bedrock access even in local mode:

export AWS_REGION=us-east-1

export BEDROCK_MODEL_ID=anthropic.claude-3-haiku-20240307-v1:0

Step 2: Run the test suite

The project uses Vitest, including property-based tests through fast-check. Property tests are particularly valuable for a rules engine, because defects tend to appear in combinations of job level and tool configuration that individual test cases miss.

npm test

Step 3: Deploy the infrastructure

cd infra

npm install

npx cdk bootstrap # first time in this account and Region only

npx cdk deploy

The stack outputs TableName, TableArn, SsmParameterPrefix, and AlbDnsName. Use AlbDnsName to reach the deployed portals.

Step 4: Populate configuration

Set the Parameter Store values under the prefix from the stack output, including your verified SES sender address, the default approver, the follow-up reminder window, and the portal base URL. The portal base URL is used to construct approval links in outbound email, so it must match the address approvers can actually reach.

Step 5: Validate end to end

Submit a request that should auto-grant and confirm it provisions without human involvement. Then submit one that should escalate, confirm the approver receives an email, and approve it from the tokenized link. Check that a second click on the same link is rejected, which confirms the conditional update is working.

Cleanup

To avoid ongoing charges, destroy the stack:

cd infra

npx cdk destroy

This stack is not serverless and bills while idle, so cleanup matters more than it would for a Lambda-only deployment. The VPC is created with natGateways: 1, and a NAT Gateway carries an hourly charge plus data processing regardless of traffic. The Application Load Balancer and the Fargate task running at desiredCount: 1 also bill continuously.

Also remove any SES verified identities and Parameter Store entries you no longer need.

Conclusion

In this post, we deployed an IT self-service portal that automates routine access provisioning while preserving human review where it matters. The architecture separates a deterministic rules engine, which owns every authorization decision, from a conversational assistant that handles discovery and explanation.

That separation is the reusable idea. When you add a language model to a workflow, the instinct is to point it at the hardest-looking step. Where a wrong answer carries consequences, the better pattern is to keep the decision boring, deterministic, and auditable, and to spend the model on the surrounding work: helping people find what they need and explaining what happened.

Leave a comment

Share

Subscribe now

The two interfaces can produce different results. A harness is the application layer around a model that supplies instructions, tools, retrieval, memory, routing, and execution logic. For example, when someone asks ChatGPT for a product recommendation, the ChatGPT harness can add account memory, decide whether to search the web, route the request to a model, and format the response with citations. An agentic development environment such as Kiro uses a harness to add workspace context, invoke tools, and manage multi-step tasks around the model.

The following diagram shows how a request can move through an application harness.

By contrast, a direct API request might use a fixed OpenAI model with web search enabled and no conversation history. A request in ChatGPT might use account memory, search only when needed, and route the prompt between model variants. Both requests use OpenAI models, but they do not test the same system.

This post explains five harness factors that affect brand mentions and citations. It also describes what an AI visibility audit should record so that its results are useful and repeatable.

1. Web search changes the source of the answer

A model API does not search the web unless the request uses a search-enabled model or makes a search tool available. Without retrieval, the response depends mainly on information learned during model training and the content supplied in the request.

A chat application can decide to search before it answers. When it does, the retrieved pages can affect which brands appear, how they are described, and which sources receive citations. The result measures whether current web content about a brand can be found and used, in addition to what the model already knows.

This distinction creates two valid but different tests:

• A request without search measures whether the model already associates a brand with the topic.

• A request with search measures whether the system can find and use current information about the brand.

OpenAI documents this difference directly. API developers enable web search by adding the web_search tool to a Responses API request. The model can then decide whether to use it unless the developer requires the tool. ChatGPT can automatically search when a question might benefit from web information. Both interfaces can return source citations when search is used (OpenAI API web search, ChatGPT Search).

An audit should state whether search was available, whether it was required or optional, and which search tool was used. Reporting only the model name is not enough.

2. Application instructions shape the response

Chat applications can add instructions that users do not see. These instructions can define the assistant’s role, response format, safety behavior, available tools, and preference for concise or detailed answers. OpenAI’s Model Spec, for example, describes a hierarchy of platform, developer, and user instructions. It also notes that users might not see system or developer messages (OpenAI Model Spec).

The instructions can affect brand visibility. Consider these two directions:

• “Recommend the best product and explain why.”

• “Compare at least five products and include advantages and disadvantages.”

The first is likely to concentrate attention on one brand. The second creates more opportunities for brands to appear. A direct API test that uses different instructions will not reproduce the same behavior.

Complete application instructions are not always available to an auditor. The audit can still identify the interface used and publish any system or developer instructions that the auditor controls.

3. Memory and location personalize the result

A direct API request is usually stateless unless the application supplies prior messages or stored user data. Consumer assistants can use conversation history, saved memory, language, and location.

These signals can change a recommendation. A user who previously discussed enterprise procurement might receive different products from a user who asked about tools for a five-person startup. A local-services query can change when the user’s location changes.

OpenAI states that ChatGPT can use memories, past chats, files, and connected applications to personalize responses. ChatGPT Search can also use general IP-based location, optional device location, and relevant memories when it rewrites a search query (Memory FAQ, ChatGPT Search).

Research supports the broader effect. One 2026 benchmark found that relevant and irrelevant memories can change how assistants interpret user intent (How Does Personalized Memory Shape LLM Behavior?). A separate study of 2,050 memory entries from 80 ChatGPT users found that many entries were created automatically from prior conversations rather than only through explicit requests to remember information (The Algorithmic Self-Portrait).

A single visibility score cannot represent every personalized experience. For a stable baseline, an audit can use a new session with memory disabled and record the language and region. A separate personalized test can then show how results vary for defined customer profiles.

4. The application can route prompts to different models

An API request normally names a specific model. A consumer product can select a model at runtime.

OpenAI, for example, describes GPT-5 as a system that routes requests between faster and deeper-reasoning models based on the prompt, required tools, and user intent (GPT-5 System Card). The documented GPT-5 system can also use a smaller model after an account reaches its usage limit (Introducing GPT-5).

This creates a tradeoff:

• A fixed API model improves repeatability.

• Application-level routing better represents the product experience.

Neither test is universally better. The choice depends on the question. A model comparison should use fixed model versions. A customer-experience audit should test the consumer application and note that the exact routed model might not be visible.

Models and routing rules also change. Every result should include the collection date. Comparisons over time should use the same interface and configuration where possible.

5. Prompt wording and execution settings affect which brands appear

Small wording changes can produce different recommendations. “Best project management software for a small team” and “easy project planning tool for a ten-person agency” express a similar need, but they provide different signals about price, industry, and ease of use.

A 2026 arXiv preprint tested approximately 6,000 paraphrase runs and 6,000 same-prompt reruns across OpenAI and Anthropic models. In that experiment, natural paraphrases of the same buyer intent produced less consistent recommendation sets than repeated use of the same prompt (Paraphrase Brittleness in Production Retrieval-Augmented Commercial Recommendation). Because this is a preprint and one experimental setup, its effect sizes should not be treated as universal. The result still illustrates why an audit should test more than one phrasing for each buyer intent.

Generation settings add more variation. Temperature, output length, and repeated sampling can change the result where the selected model and API support those controls. An agentic development environment such as Kiro can also plan a task, use tools, inspect files, and revise its response. A basic API request might make one model call. Access to a model through an agentic product is therefore not equivalent to access through the model provider’s API.

To reduce this variance, group several prompt variations under the same intent and repeat each prompt. Report the average result and the variation between runs.

What changed in a controlled test

To show these factors are not just theoretical, I ran a small experiment in one category: project management software. I tracked seven brands (Asana, Trello, monday.com, ClickUp, Notion, Basecamp, and Jira) across three paraphrases of a single buyer intent, with three repetitions each.

The API conditions used claude-sonnet-5 and changed one factor at a time: web search on or off, a neutral instruction versus “recommend one product” versus “compare five products”, and a startup versus enterprise buyer profile supplied as context. I then ran the same three prompts in the Claude application (Sonnet, standard settings) and the ChatGPT application (default automatic model), each in fresh sessions. This is a small, category-specific test, not a general benchmark

What the results showed, factor by factor:

• Web search. Turning search off removed citations entirely and reordered the brands. With search on, monday.com led first mentions; with search off, Trello led. Same model, different prominence.

• Instructions. “Recommend one product” produced 1.8 brands per answer; “compare five products” produced 3.7. The instruction, not the model, set how many competitors appeared.

• Buyer profile. The enterprise profile was the sharpest shift: in 44% of its answers, none of the seven tracked tools appeared at all, because the model named enterprise platforms outside the list. The startup profile, by contrast, surfaced Trello prominently.

• Prompt wording. Recommendation sets from two paraphrases of the same intent overlapped 0.58 on average, lower than the 0.67 overlap between reruns of the identical prompt. Rewording moved the results more than rerunning did.

• Application versus API. The clearest single result: through the API with search, Claude cited about 11 sources per answer; through the Claude application on standard settings, the same model family returned zero citations, because the application did not search for these prompts. The recommended brands also shifted. Linear, a tool that appeared in only 11% of the baseline Claude API answers (neutral instruction, search on), appeared in 77% of the Claude application answers. Notion moved from weak in the API to prominent in the application. Across all three interfaces, however, monday.com held the top share of voice, so the leader was stable while the middle of the list reshuffled.

Two limits on these numbers. The buyer-profile conditions are a proxy for application memory, not a real memory implementation. And the application-versus-API comparison is only clean for Claude, because it holds the provider roughly constant; the ChatGPT result changes the provider and model as well, so I treat it as a separate product observation rather than a harness-only comparison. The citation metric also counts only structured citations, so it understates the ChatGPT application, which cited sources inline (frequently techrepublic.com) without exposing them as separate links.

The takeaway is consistent with the five factors: within one category, the interface and its settings changed which brands appeared, how many appeared, and whether any sources were cited, without changing the underlying model.

My recommended reporting standard

Based on these factors, I recommend that every AI visibility report identify the complete system that produced each response. At minimum, document:

1. Interface: Consumer application, API, or agentic application.

2. Model: Exact model version when available.

3. Retrieval: Whether web search or another retrieval tool was available and whether its use was required.

4. Session state: New or existing session, memory setting, language, and region.

5. Instructions: System or developer instructions controlled by the auditor.

6. Prompts: Full prompt set, intent groups, and paraphrases.

7. Execution: Generation settings, number of runs, and collection date.

8. Evidence: Raw responses and citations, subject to privacy and licensing limits.

Without this context, a visibility score is difficult to reproduce and easy to misinterpret. These details explain why two tools can report different scores for the same brand. They also make before-and-after measurements more credible.

Conclusion

API and application tests answer different questions, so they should not be treated as substitutes for each other.

For repeatable experiments, model comparisons, and controlled retrieval tests, I recommend using a fixed API configuration. To understand what customers experience, test the consumer application. A complete audit uses both methods and reports the results separately.

The model name alone does not define an AI answer. The interface, tools, state, routing, and prompt all contribute to the result. AEO teams should measure the complete system and document how the measurement was made.

Leave a comment

Share