Chetto is a multimodal AI system that turns everyday WhatsApp business conversations into structured financial records and tasks, that live scattered across chat threads®
As AI-powered workflows became core to the product, keeping output quality consistent got harder. Small changes to prompts, model versions, extraction logic, or orchestration could shift behaviour improving some cases while quietly regressing others that manual testing missed.
I built a regression-safe evaluation framework to test AI workflow changes before deployment. By using GPT-based semantic matching instead of exact-match comparison, it judges whether outputs are still correct even when the wording differs a reliable basis for experimentation, QA, and controlled rollouts.
Evaluating AI systems is fundamentally different from traditional software. Conventional outputs are deterministic and pass or fail on exact assertions; AI workflows produce many valid responses for the same input, so exact string matching flags correct answers as failures simply because they are phrased differently.
An automated framework that compares workflow outputs against expected behaviour using GPT-based semantic assessment measuring meaningful equivalence rather than literal text.
Teams run evaluation suites against different workflow versions, compare results, and catch regressions before anything reaches production.
A curated benchmark of representative inputs and expected outputs common scenarios, edge cases, failure-prone examples, and business-critical workflows held consistent across iterations and model versions.
For each case the workflow generates an output, the expected result is retrieved, and a GPT evaluator scores semantic similarity and task correctness as pass, fail, or a quality score focused on intent, not exact wording.
Benchmarks run across baseline and candidate systems production vs. candidate workflows, new prompts, model upgrades surfacing exactly where performance improved or degraded before release.
Repeatable, measurable evaluation lets teams test changes, compare versions, and validate improvements turning a qualitative review into a data-driven workflow.
Semantic correctness beats exact textual similarity evaluation must allow many valid responses rather than enforce rigid matching.
Benchmark quality matters as much as evaluation logic; a framework is only as good as the coverage of its test cases.
AI evaluation is a product capability, not a one-time project it is essential infrastructure for reliability and safe scaling.
Business operations on WhatsApp arrive as receipts, invoices, and PDFs and increasingly as voice notes dictating orders on the move. An agentic pipeline turns all of it into structured, queryable records.
Vision and audio models read each artifact; reasoning agents extract vendor, amount, date, and line items, then route them into the ledger with a fallback chain that keeps accuracy high on messy, real-world inputs.
Most of a business’s money and intent hides inside images and audio. Receipts are blurry, invoices are inconsistent, and orders are spoken rather than typed none of it labelled or structured.
An agentic extraction pipeline: images run through vision captioning and OCR; audio is transcribed and parsed for order intent; a reasoning layer structures everything into clean records.
When primary OCR is unsure, the agent falls back to stronger models pushing extraction accuracy from a shaky 60% to a dependable 97%+.
Every image is described in plain language first “a printed receipt totalling ₹1,250” giving downstream steps meaning to work from instead of raw pixels.
Primary OCR pulls vendor, amount, and date; when a photo is dim or crumpled it escalates to advanced models, lifting accuracy from 60% to 97%+.
Voice notes are transcribed and parsed for order intent items, quantities, and amounts so spoken orders become structured tasks alongside written ones.
A reasoning layer turns free text into queryable JSON amount, vendor, date, category and routes it straight into the dashboard.
Meaning-first captioning makes every later step more robust than working from raw pixels.
A fallback chain beats a single model graceful escalation is what makes accuracy dependable.
Audio is a first-class input; treating it as an afterthought loses real operational detail.
Every business runs differently, so Chetto lets users define the particular tasks they want created and the specific settings that shape them. But applying a new configuration straight to live data is a leap of faith.
The simulation framework lets users try their preferences on their own data first seeing exactly what a setting would produce before it goes live, so they can adjust until it fits and then implement with confidence.
Users need their own tasks, rules, and settings not fixed defaults. But there is no safe way to know how a new preference will behave on their real data until they apply it, and by then it is already affecting live records.
A simulation mode where users configure the tasks and settings they want and run them against a copy of their own data, with nothing changed for real.
They see exactly what the configuration would produce, tune it until it matches what they expect, and only then apply it to their live workspace.
Users define the particular tasks they want created and the specific rules and settings that shape them, instead of relying on one-size-fits-all defaults.
The configuration is simulated against the user's real records, so the preview reflects their actual business, not generic sample data.
Results show exactly what the new settings would produce, so users can compare against current behaviour before changing anything.
Once the preview looks right, the same configuration is applied to the live workspace, with no surprises and nothing irreversible.
Letting users preview on their own data builds trust faster than any amount of documentation.
People configure more boldly when changes are visible and reversible first.
A safe sandbox turns “I'm not sure” into “show me” and then “apply”.
