How Avatarin Built a 24/7 Retail Agent with GPT-Realtime (and the Prompts They Left Out)

Reporting on Avatarin's GPT-Realtime retail deployment tells a good story. It's also incomplete. Coverage of the project shows what the agent does; it doesn't show the prompt architecture, the guardrail layers, or the integration work that makes it actually reliable at 2 a.m. when no human is watching. That gap matters. Plugging GPT-Realtime into a retail environment without that scaffolding doesn't produce Avatarin. It produces a chatbot that confidently quotes the wrong price, stutters when a customer interrupts, and has no idea what to do when inventory data times out. This article fills that gap. You'll get the layered system prompt structure, the fallback and guardrail patterns, an integration blueprint, and ready-to-adapt templates. The goal isn't to retell a marketing story. It's to give you the implementation manual that didn't get published.

What the Coverage Doesn't Show

Write-ups on deployments like Avatarin's are results documents. They tell you the outcome. They skip the part where someone had to decide what happens when a customer asks about a product that's out of stock in one color but available in another, and the inventory API takes 800ms to respond. That's not a criticism of how these projects get reported. Results-focused coverage is supposed to do that. But it creates a dangerous impression for anyone who reads it and thinks, "Great, I'll just wire up GPT-Realtime and we're done." Here's the contrast in concrete terms. A demo-grade prompt for a retail voice agent looks like this:

You are a helpful retail assistant. Answer customer questions about products, prices, and availability.

That prompt will work in a demo. In production, with real customers and real edge cases, it will fail in at least six ways: it doesn't define what to do when data is unavailable, it doesn't constrain response length for voice output, it doesn't establish a fallback path, it doesn't handle ambiguous product references, it doesn't set a tone that matches brand voice, and it doesn't prevent the model from speculating when it doesn't know something. A production-grade system prompt looks more like this:

You are a friendly and knowledgeable retail assistant named "ShopBuddy" for Acme Retail.

ROLE:
Your job is to provide accurate, concise information about products, pricing, and stock availability. You do not offer personal opinions or recommendations. You do not speculate. If you are not certain, you say so and offer to escalate.

KNOWLEDGE BASE:
You have access to real-time product and inventory data via the tool calls available to you. Before stating any price or stock level, you must retrieve that data from the tool. Never quote a price or stock status from memory.

RESPONSE FORMAT:
Keep all responses short and conversational. This is a voice interface — long answers break the conversation. If an answer requires more detail, offer to send a summary to the customer's phone or email.

WHEN YOU DON'T KNOW:
If a tool call fails or returns no result, say: "Let me check on that — one moment." Wait for a retry. If the retry also fails, say: "I'm having trouble pulling that up right now. Can I connect you with a team member?"

TONE:
Warm, efficient, and direct. Never filler phrases like "Great question!" or "Absolutely!" Just answer.

The difference between those two prompts is the difference between a demo and a deployable product.

Core Prompt Architecture for Always-On Retail Voice

A 24/7 retail voice agent isn't a single prompt. It's a stack. Understanding each layer helps you diagnose what breaks when something goes wrong.

Layer 1: The System Prompt

This is your agent's constitution. It defines role, constraints, tone, response format, and escalation rules. It runs on every turn. It's also the layer most teams underinvest in because it's not glamorous work, but it's where reliability comes from. The system prompt for a production retail agent needs at least five explicit sections:

Role definition. Who is this agent? What store? What's it authorized to do?
Tool use rules. When must it call an API before answering? What's prohibited without a lookup?
Response format constraints. Voice output needs short sentences. Brief, direct replies work far better than long explanations for most retail queries.
Escalation paths. What triggers a handoff to a human? Frustrated sentiment, three failed queries in a row, or a request the agent can't fulfill should all route to a person.
Persona and tone rules. Specific enough that it sounds consistent across thousands of conversations.

Layer 2: Speech-to-Text

The model never hears the customer. It reads a transcript. The quality of that transcript determines whether the downstream prompt even has a chance of working. Accent handling, background noise, and product names that don't appear in standard training data (brand-specific SKU names, for example) are the most common failure points. Build in a confidence-threshold check: if STT confidence is below a set threshold, have the agent ask for clarification rather than guess.

Layer 3: Tool Calls / API Retrieval

This is where the agent fetches real data before it generates a response. Price. Stock level. Order status. The system prompt must make tool calls mandatory for any factual claim. "Never quote a price from memory" is the exact instruction you want in writing, because without it, the model will occasionally answer from training data, and training data doesn't know what your current sale price is.

Layer 4: Response Generation

GPT-Realtime generates the reply based on the system prompt plus the retrieved data. The critical constraint here is length. Voice listeners don't tolerate long responses the way readers do. Answers that work fine in a chat widget become exhausting when spoken aloud. Keep generation instructions tight: short sentences, one idea at a time, no lists.

Layer 5: Text-to-Speech

The generated text becomes audio. Formatting artifacts (asterisks, parentheses, numbered lists) break TTS output. Add a formatting rule to your system prompt: "Never use markdown, bullet points, or special characters. Write in plain spoken sentences only."

Guardrails, Fallbacks, and Latency Patterns That Keep Conversations on Track

Most retail voice agent failures fall into three categories: the agent says something wrong, the agent says something confusing, or the agent goes silent. Good guardrails prevent all three.

Preventing Wrong Answers: The Data-Before-Claim Rule

The single most effective guardrail for a retail agent is requiring a tool call before any factual statement. Here's how to encode that in a system prompt:

CRITICAL: You must never state a price, stock level, or product specification without first calling the inventory tool. If the tool is unavailable, say: "I'm having trouble confirming that right now. Would you like me to have a team member follow up with you?"

This one instruction eliminates the most damaging class of error: confidently wrong pricing. A customer who hears the wrong price and shows up expecting it will be unhappy in a way that's hard to recover from.

Handling Interruptions

GPT-Realtime supports real-time audio, which means customers can interrupt mid-sentence. Your prompt architecture needs to account for this explicitly. The model needs to know that a new utterance mid-response restarts the conversation context, not appends to it. In practice, this means building your system prompt to treat each turn as potentially starting fresh:

If the customer interrupts or changes the subject mid-conversation, immediately stop addressing the previous question and respond to the new one. Do not refer back to the previous question unless the customer does.

Fallback Chains

A fallback chain is a sequence of escalating responses when the agent can't answer. Design it explicitly rather than letting the model improvise. A three-level fallback chain for retail:

Level 1 (data unavailable): "Let me check on that — one moment." [Retry tool call]

Level 2 (retry failed): "I'm having trouble pulling that up right now. Can I try a different way to help you?"

Level 3 (unresolvable): "I want to make sure you get accurate information. Let me connect you with someone who can confirm this for you."

Each level sounds natural. None of them fabricates an answer.

Latency Management

Latency in a voice agent is felt immediately. Even pauses of a few hundred milliseconds are perceptible to listeners, and longer ones break conversational flow. A few practical approaches:

Keep your system prompt as short as possible while preserving all required instructions. Shorter prompts reduce time-to-first-token.
Use filler phrases as a latency bridge. Instruct the agent to say "One moment" or "Let me check" while waiting for a tool call to return, rather than going silent.
Cache common lookups. If your most popular products have their prices fetched many times per day, a short-lived cache reduces API round-trip time significantly.
Set a timeout on tool calls. If an API hasn't responded within a reasonable window, trigger the Level 2 fallback rather than waiting indefinitely.

If you're thinking about what happens when an API or platform has a wider outage, the principles in building resilient AI workflows for platform failures apply directly here.

Integration Blueprint: POS, Inventory, and CRM

The integration layer is where most teams underestimate the work. Connecting APIs is the easy part. Designing a data flow that handles partial failures, slow responses, and format mismatches without breaking the conversation is the actual challenge.

Step 1: Speech-to-Text

Choose a STT service with a low word error rate on your customer demographic. Test specifically on product names, SKU references, and brand-specific terminology. These are the words most likely to be transcribed incorrectly, and an incorrectly transcribed product name sends the entire downstream query off track. Consider building a custom vocabulary or hotword boost for high-frequency terms specific to your store.

Step 2: Intent and Entity Extraction

Before hitting your product database, the agent needs to extract structured data from the transcript. Product name, variant (color, size), query type (price, availability, order status). You can do this with a lightweight extraction prompt before the main response generation:

Extract the following from the customer's message:
- Product name (or description if no name given)
- Variant details (color, size, model — if mentioned)
- Query type: one of [price, availability, order_status, general_info, other]

Output as JSON. If a field is not mentioned, return null for that field.

Customer message: {transcript}

This gives your integration layer clean structured data to query against, rather than passing raw natural language to your database.

Step 3: API Integration

Use a middleware layer between GPT-Realtime and your backend systems. Don't call your POS or inventory database directly from the model's tool calls if those systems aren't designed for the request volume a 24/7 voice agent generates. The middleware handles rate limiting, caching, format translation, and error logging. Each tool your agent can call should have:

A clear function name and description the model can understand
A defined input schema
A timeout value
A defined error response the model knows how to handle

Step 4: Response Formatting for Voice

Data coming back from a POS system is not conversational. A raw API response might say {"price": 29.99, "currency": "USD", "in_stock": true, "quantity": 14}. Your agent needs to convert that into "That shirt is $29.99 and we have it in stock." The system prompt instructions for response format handle this, but you need to validate that the model isn't leaking JSON or technical language into its output during edge cases.

Prompt Templates You Can Deploy Today

These templates are starting points, not finished products. Adapt the persona name, tone, and fallback language to match your brand. Each template follows the same structure: system prompt snippet, example user input, and expected output.

Greeting and Introduction

SYSTEM PROMPT SNIPPET:
When a conversation starts, greet the customer warmly and offer to help. Keep the greeting under 15 words. Don't list everything you can do — just invite them to ask.

USER INPUT EXAMPLE:
[Customer approaches kiosk / starts conversation]

EXPECTED OUTPUT:
"Hi there! I'm ShopBuddy. What can I help you find today?"

Product Inquiry

SYSTEM PROMPT SNIPPET:
When a customer asks about a product, call the product_lookup tool with the extracted product name and any variant details before responding. If multiple matches return, ask a clarifying question about the most likely distinguishing attribute (usually color or size).

USER INPUT EXAMPLE:
"Do you have running shoes in a size 10?"

EXPECTED OUTPUT (after tool call returns results):
"We have three running shoe styles in size 10. Are you looking for road running, trail, or casual?"

Price Check

SYSTEM PROMPT SNIPPET:
Before stating any price, call the price_lookup tool. If the product is on sale, state both the original and sale price. Never estimate or round prices.

USER INPUT EXAMPLE:
"How much is the blue Patagonia jacket?"

EXPECTED OUTPUT (after tool call):
"The blue Patagonia jacket is $179. It's currently on sale from $229."

Stock Availability

SYSTEM PROMPT SNIPPET:
Call the inventory_check tool before answering any stock question. If an item is out of stock, check for alternatives in the same category before reporting unavailability. Offer to notify the customer when it's back in stock if the store supports that feature.

USER INPUT EXAMPLE:
"Is the medium gray hoodie in stock?"

EXPECTED OUTPUT (after tool call, item out of stock but alternative found):
"The medium gray isn't available right now, but we have it in charcoal in medium. Would that work?"

Order Status

SYSTEM PROMPT SNIPPET:
To check order status, ask for the customer's order number or the email address used for the purchase. Call the order_status tool with that information. Do not speculate about shipping timelines — report only what the tool returns.

USER INPUT EXAMPLE:
"I want to check on my order."

EXPECTED OUTPUT:
"Sure — do you have your order number handy, or the email address you used when you ordered?"

FAQ

How do you write prompts for real-time voice AI that don't sound robotic?

The main culprit is over-specification. When you tell a model exactly what phrase to use in every situation, it starts to sound scripted. Instead, specify the tone and constraints, then let the model generate the actual phrasing. "Respond as a warm, efficient retail assistant. Skip filler affirmations. Get to the answer in the first sentence" produces more natural output than a list of approved phrases. You can also add a few example exchanges showing the conversational register you want, and the model will match it without being locked into specific wording.

What guardrails prevent GPT-Realtime from giving wrong prices or stock information?

The data-before-claim rule is the core guardrail: no price or stock statement without a tool call returning current data first. Beyond that, add an explicit instruction that the model must never estimate or interpolate from memory. Pair this with a fallback for when the tool call fails, so the model has a safe path that isn't "make something up." Validation at the middleware layer, checking that returned prices fall within expected ranges, catches the cases where a data error rather than a model error is the problem.

How do you handle interruptions and context switching in a retail voice agent?

GPT-Realtime's audio mode handles the raw interruption detection, but your prompts need to tell the model what to do when that happens. The key instruction is to treat an interruption as a full context reset for the topic, while retaining session context (what the customer has been browsing, preferences expressed earlier). A prompt section that explicitly says "if the customer changes subject, address the new topic immediately without finishing the previous response" prevents the jarring experience of an agent that keeps answering a question the customer already moved past.

Can GPT-Realtime integrate directly with an existing point-of-sale system?

Not without a middleware layer. Most POS systems don't expose APIs designed for the request pattern of a voice agent. You'll need a middleware service that accepts tool call requests from the model, translates them into the format your POS expects, handles authentication, and returns a normalized response the model can work with. The middleware also gives you a place to add caching, rate limiting, and error logging without touching the POS system itself.

What latency should I expect when running a 24/7 retail voice agent?

Several factors stack: STT processing, GPT-Realtime time-to-first-token, tool call round-trip time, and TTS generation. The OpenAI Realtime API is designed for low latency, but tool calls add time proportional to how fast your backend systems respond. A well-optimized setup with cached common queries and a fast middleware layer can feel responsive. A setup with cold API calls to a slow inventory system will not. Test end-to-end latency with your actual backend before assuming the AI layer is the bottleneck, because often it isn't. On cost: OpenAI publishes current Realtime API pricing on their platform page, and it should be evaluated against your actual conversation volume rather than estimated in advance.

Building This Right Takes More Than One Good Prompt

Avatarin's 24/7 retail agent works because someone spent real time on the parts that don't make it into write-ups. The layered system prompt, the mandatory tool call rules, the fallback chain, the integration middleware, the voice-optimized formatting constraints. None of that is exotic. All of it takes deliberate work. Once the architecture is right, iteration is fast. The prompt is the lever. Change the system prompt, test a conversation, measure the output. That cycle, done carefully, is how a retail voice agent goes from "impressive demo" to "actually reliable at 2 a.m." And if you want to see what's possible with voice AI beyond a cloud-hosted model, the work being done on running complete voice models locally is worth tracking, especially for deployments where latency or data privacy makes cloud calls undesirable. If you want a starting point that's already structured for retail voice, Ultra Prompt's Retail & E-commerce prompt pack has templates built around exactly this kind of always-on deployment. Less time on the scaffolding means more time on the part only you can do: tuning the agent to your customers, your products, and your brand.

Ultra Prompt