Optimizing LLM Agents for Kenya's Job Seekers (50% Shorter Conversations, Full Swahili Support)
The TL;DR
Compass is an AI career-guidance platform by Tabiya that interviews job seekers about their work history and maps what they've done to a formal skills taxonomy — including invisible work like informal trading, caregiving, and community roles. For the Kenya launch, the multi-agent LLM pipeline had a problem: conversations were long, repetitive, and English-only. In a market where many users pay per megabyte of data and speak Swahili first, every unnecessary turn was a user lost.
Key Results
Over five milestones I built an evaluation harness first, then used it to drive persona-aware conversation refactors, full Swahili enablement, and CV integration — cutting worst-case conversations from 70 to 35 turns and LLM calls by 20%, while holding skill-discovery quality at 85%+ overlap with baseline. 11 merged PRs across a Python + TypeScript codebase.
The Problem: An Agent That Talked Too Much, in the Wrong Language
Compass's core flow is "skills elicitation": a chain of LLM agents (welcome → collect experiences → explore skills → farewell) interviews the user and extracts structured experience and skill data. It worked — but nobody could say precisely how well, and users were feeling the friction.
What the Baseline Runs Revealed
My first deliverable was measurement, not fixes. I instrumented the E2E test suite with an automated metrics collector and ran repeated conversations across user personas. The numbers:
- 32.4 turns per conversation on average — and 70 turns in the worst case (a formal worker giving verbose, CV-style answers)
- 251 LLM calls per conversation, driving both cost and latency
- 11% question repetition rate — the agent re-asked things users had already answered
- "Okay" started 27% of all agent questions; phrase diversity was just 15.4%
- A hidden bug: the farewell agent was consuming 83% of processing time, making 64 LLM calls after the conversation had ended
- English only — no path for the millions of Kenyan users who are more comfortable in Swahili or code-switch between the two
The client's ask was specific: 20% faster conversations, 30% less repetition, Swahili support — without degrading the quality of the skills discovered. That last constraint is what makes this hard. Anyone can make a chatbot terser; keeping extraction quality while cutting turns requires knowing exactly what each turn contributes.
The Solution: Evaluation Harness First, Then Optimize
You can't refactor prompts safely without a regression net. So the project was sequenced to build the net before touching the flows.
Phase 1: Baselines, Observability & CI Quality Gates
The Foundation
- Built a baseline metrics collector hooked into the E2E chat executor — turn counts, per-phase and per-agent timing, LLM call counts, semantic repetition rate, skill overlap
- Added correlation IDs and structured logging fields (session, turn, agent type, LLM call duration) across the backend, with a sensitive-data checklist so no PII ever hits the logs
- Created golden transcripts per persona and wired them into GitHub Actions: PRs fail if turn count, repetition, or skill overlap regress past thresholds
Phase 2: Persona-Aware Conversation Refactor
The data showed two very different users. An informal worker ("I sell vegetables at the market") needs scaffolding and examples. A formal worker reciting their CV needs the agent to acknowledge what it already heard and stop re-asking. I built a persona detector that classifies users from verbal cues — in both English and Swahili — and adapts prompts, probing depth, and exit criteria per persona. Combined with fixing the farewell agent and consolidating the skills-exploration loop, this took the worst-case formal-verbose persona from 70 turns to under 35.
Phase 3: Swahili Enablement
Localization Is Architecture, Not Translation
- Evaluated candidate models for Swahili quality/cost and standardized on Gemini 2.5 Flash
- Built a synonym-mapping module: 50+ Swahili and Sheng job terms (formal like mhasibu/accountant, informal like mama mboga-style vendor roles) normalized to the ESCO skills taxonomy before retrieval
- Handled code-switching — real Kenyan users mix English and Swahili mid-sentence
- Enforced language consistency in prompts (no drift back to English) and shipped Swahili golden transcripts into the same CI gates
Phase 4: CV Integration
Uploaded CVs were dead-end data — extracted as plain text bullets that never reached the conversation. I built a structured extractor (title, company, dates, work type, responsibilities), a mapper that pre-populates the agent's state with deduplicated, provenance-tagged experiences, and prompt adaptations so the agent opens with "I see from your CV that you worked as X at Y — tell me more" instead of asking everything from scratch.
Phase 5: Hardening & Handover
Edge-case suites for the CV pipeline (corrupt files, contradictory data, mid-conversation uploads), post-secondary education collection so degree-derived skills rank alongside work skills, retry and graceful-degradation paths, and a full handover package. Along the way I also shipped SSE streaming for chat responses, an API-gateway migration (Pulumi IaC), and MongoDB index optimizations.
The Technical Implementation
The persona detector illustrates the approach: deterministic where possible, bilingual by design, and defaulting to the safer path for informal workers. Simplified from the production module:
class PersonaType(Enum):
INFORMAL = "informal" # task-oriented, informal workers
FORMAL = "formal" # structured, formal employment vocabulary
# English + Swahili cues, curated with local terminology review
FORMAL_INDICATORS = [
"position", "department", "responsibilities", "reported to",
"cheo", # title/position
"idara", # department
"mshahara", # salary
"mkataba", # contract
]
INFORMAL_INDICATORS = [
"daily work", "helped out", "hustle", "odd jobs",
"kazi", # work/tasks
"biashara ndogo", # small business
"kuuza", # selling
]
def detect_persona(user_messages: list[str]) -> PersonaType:
text = " ".join(user_messages).lower()
formal_score = sum(term in text for term in FORMAL_INDICATORS)
informal_score = sum(term in text for term in INFORMAL_INDICATORS)
if formal_score > informal_score and formal_score >= 2:
return PersonaType.FORMAL
# Default to INFORMAL: simpler language and more
# scaffolding never hurts; skipped questions do.
return PersonaType.INFORMAL
# Downstream: prompts, probing depth, and exit criteria
# all branch on the detected persona — and every change is
# gated by golden-transcript thresholds in CI.The Outcome
Every improvement below is measured by the same harness that established the baseline — not estimated, and protected against regression on every PR since.
Results
| Metric | Before | After | Change |
|---|---|---|---|
| Worst-Case Conversation | 70 turns | ≤35 turns | -50% |
| Average Turn Count | 32.4 turns | ≤27 turns | -17% |
| LLM Calls per Conversation | 251 | ≤200 | -20% |
| Question Repetition Rate | 11% | ≤8% | -27% |
| Phrase Diversity | 15.4% | 35%+ | 2.3x |
| Languages Supported | English only | English + Swahili | 80%+ parity |
Emmanuel demonstrated a rapid understanding of our complex system. He didn't just execute the basic requirements; he proactively suggested forward-looking and sustainable code edits that significantly improved our Python and TypeScript codebase... He is reliable, incredibly quick, and documents his work thoroughly. I would absolutely hire him again and highly recommend him to any client looking for a dedicated and highly skilled AI developer.
The Lesson
You can't optimize an LLM agent you can't measure — and you can't localize one by translating strings.
The single highest-leverage deliverable in this project was the evaluation harness. It found a bug nobody knew existed (83% of processing time spent after the conversation ended), it made every prompt refactor safe to ship, and it turned "the bot feels better" into numbers a client can sign off on. The Swahili work taught the second lesson: real localization lives in the retrieval layer, the taxonomy mapping, and the persona logic — the translated prompts are the easy 10%.
Key Takeaways
- Harness before optimization: baseline metrics + CI quality gates make aggressive prompt refactors safe
- Personas beat averages: the same flow that suits an informal trader wastes 40 turns on a formal worker — detect and adapt
- Localization is an architecture problem: synonym mapping to the taxonomy and code-switch handling mattered more than translation
- Instrument everything: the biggest performance bug was invisible until per-agent timing existed
Have an LLM agent that needs to get faster, cheaper, or multilingual?
I optimize production AI systems with evaluation-first discipline — measured improvements, not vibes. Book a consultation.
💳 Free call via Cal.com • 💼 Paid sessions: $35/30min or $70/hour on Upwork
For custom work and projects, visit my Upwork profile