AI Voice Service Agent Prototype
Personal prototype: a voice agent answers a small-business call, reads live company data, drafts a job via tools, and stops until a person confirms.
Technologies
- Voice AI
- Conversational AI
- LLM Tool Calling
- Service APIs
- Structured Data
- Human-in-the-Loop
Key Achievements
- Task-oriented inbound call flow for a small service business
- Answers grounded in current company data from APIs, not model memory
- Job requests created only as drafts, via validated tools
Project Links
Why I built this
I wanted to see whether a voice agent can take an inbound service call — “are you free Thursday, can you quote a repair” — without becoming a chatbot that invents opening hours. The interesting part is not speech-to-text. It is: live data, structured writes, and a human still owning the commitment.
This is a non-production personal prototype.
Scope
The sample is one inbound call into a small service business. The agent can:
- Figure out whether the caller wants information or wants work booked
- Fetch current company, service, or availability data from APIs
- Ask only for the missing job fields
- Create a draft job request through a validated tool
It cannot accept the job, promise a price or a slot, or confirm back to the caller until an operator says so. Telephony, recording consent, and production IAM are out of scope.
Core workflow
- Caller rings the service number.
- The agent identifies the ask.
- It reads from company APIs when the answer has to be true today.
- It gathers missing job fields with short questions, not a form recitation.
- A tool call writes a structured draft: contact, service, location, timing, notes, conversation reference, review status.
- An operator reads transcript + fields, edits, accepts or rejects.
- Only then does confirmation go to the customer.
If the API is down or the request is outside what the business does, the agent clarifies or escalates. It does not fill the gap from training data.
Design choices
The model is not the CRM. Services, coverage, and availability change. Anything the caller will act on is retrieved. “I think they cover that suburb” is a bug.
Writes go through a narrow tool. The job schema is the contract: required fields, types, source of the conversation. Free-form model output does not land in the business system. That makes the write testable and keeps the LLM off unrestricted APIs.
Read and write are different trust boundaries. Looking up opening hours is not the same operation as creating work. Draft-only writes plus a review queue is the control point for speech errors, incomplete addresses, jobs the company cannot take, and anything contractual.
Escalation is a successful path. Low confidence, missing data, or a request that needs judgement should leave the happy path. Pretending otherwise is how you get a polite, wrong booking.
I used coding agents on this prototype the same way I would on a small internal tool: specify the flow and tool schemas, generate, then check against the workflow, authz, and failure cases. Accepting the first generated handler would miss the point.
What I learned
Voice needs shorter turns and stricter gathering than text chat. People will not sit through eight questions; the agent has to know which fields block a draft.
Grounding is not a prompt paragraph. If the retrieve step is optional, the model will skip it under time pressure.
The review queue is where the prototype earns its keep. Intake automation is useful even when the model is not allowed to speak for the business.
Not production
A real phone agent still needs a telephony vendor, consent and recording rules, identity, per-tool authn/z, latency and barge-in behaviour, prompt-injection controls, an eval set of real calls, monitoring, and a defined on-call review process. This repo explores the workflow, not that operating environment.