AI Call Quality Analyst в ТРТ Тбилиси

A stealth-mode AI-powered Cloud-Native Health-Tech company is looking for an AI Call Quality Analyst.

The Business Domain revolves around providing Better Care for the US Patients; meaning, it’s about Treating Patients well. The company was founded more than five years ago.

It’s not vaporware, their platform supports US Physician Networks (IPAs) by enabling Smarter, Risk-Adjusted, and more Predictive Care that improves real patient outcomes.

Compensation and Benefits:

Paid Time Off

The company has Unlimited PTOs Policy and compensated New Years Holidays on top of that. The misuse of the policy isn’t welcomed, though it’s definitely possible to take at least two weeks – and fully compensated – vacation, or more.

Corporate Hardware

The company provides Corporate Hardware for employees who completed their Probationary Period, as well as proven their value.

Means of Communication

We use Google Workspace, Slack, Zoom, and similar collaboration tools for messaging, meetings, and document sharing across the organization. We don’t use Microsoft Outlook, Microsoft Team and similar software.

Cloud-Native

There’s real ability to make most solutions in a Cloud-Native and Third-Party Integrations manner, rarely spinning something self-hosted (e.g., over GKE). Fully Serverless Approach, based on Cloud Functions and Cloud Run is definitely not welcomed.

Team Building

The company partially compensates Team Building events, when multiple teammates are located in nearby countries.

Responsibilities

Own the quality of our AI-driven patient communications - define what "good" means, catch where it breaks, and drive the fixes across the agent, the flow, and the checks.

Key Focus Areas

• Conversation QA: review AI voice/chat interactions and score them against a defined behavioural rubric.

• Evaluation & monitoring: own the automated checks and the human review loop; keep the quality signal trustworthy.

• Continuous improvement: turn findings into concrete changes to the agent, the conversation flow, and the test suite.

What You'll Own:

• Define and maintain the quality rubric - the behavioural checks (off-script, hallucination, resolution, emergency handling, "cringe"/weird behaviour) and their pass/fail criteria.

• Monitor production calls - review a smart sample, run human annotation, and measure & correct the drift of the automated LLM judge.

• Root-cause and route every defect - decide whether it belongs to the agent prompt, the conversation flow, the data pipeline, or the telephony config.

• Drive fixes with the AI team - bring prioritized, reproducible findings to engineers, propose the prompt change where the defect sits in a prompt, and add regression/autotest cases before each release.

• Align changes with the business - agree flow and script changes with clinical/ops stakeholders, balancing patient experience, compliance, and outcomes.

• Report quality in plain language - trends, dashboards, and release-readiness calls for non-technical stakeholders.

Required Experience:

• 3+ years in QA / quality analysis of customer communications (contact-center, CX, or conversational AI) - monitoring and scoring live interactions against rubrics, giving actionablefeedback, and owning it through to verified implementation (not just reporting – confirming the fix shipped and worked).

• Experience running an evaluation loop for LLM/agent quality - LLM-as-a-judge, human annotation, regression on golden sets. Comfortable reading transcripts and structured logs (JSON) and using basic SQL / data literacy to slice metrics.

• Hands-on prompt engineering - writing, versioning and iterating production prompts: system prompts for LLM voice/chat agents, and evaluator prompts for LLM-as-a-judge checks.

• Hands-on experience evaluating voice and/or chat AI agents (RetellAI or similar LLM-agent platforms). Practical understanding of conversation flows, intents, and failure modes - barge-in, ASR/TTS errors, voicemail handling, escalation and fallback paths.

• Strong analytical and communication skills - translate technical findings into business impact and back and own the quality bar across teams without formal authority (bridge business ↔ AI team).

• Experience in Healthcare, Health-Tech, and MedTech is a major advantage

Languages

• English - C1 or higher; ability to present work and lead discussions with US-based teammates, customers, and stakeholders.

• Fluent Russian for collaboration with Russian-speaking teammates.

• Spanish - a plus; our patients are served in English and Spanish.

Компания, занимающаяся разработкой облачных технологий для здравоохранения на базе искусственного интеллекта, ищет талантливого AI Call Quality Analyst для долгосрочного сотрудничества. Это не виртуальный продукт: их платформа поддерживает сети врачей (IPA), обеспечивая более интеллектуальное, адаптированное к рискам и прогнозируемое лечение, которое улучшает реальные результаты лечения пациентов.

Вы присоединитесь к международной команде первоклассных профессионалов, которые с энтузиазмом создают продукты, улучшающие качество медицинских услуг.

Обязанности

Отвечать за качество наших коммуникаций с пациентами, основанных на ИИ — определять, что означает «хорошее качество», выявлять нарушения и обеспечивать внедрение исправлений на уровне оператора, сценария взаимодействия и контрольных механизмов.

Ключевые направления работы

• Контроль качества диалогов: анализировать голосовые и текстовые взаимодействия с ИИ и оценивать их по установленным критериям поведения.

• Оценка и мониторинг: отвечать за автоматизированные проверки и цикл проверки людьми; обеспечивать достоверность показателей качества.

• Непрерывное совершенствование: преобразование выявленных проблем в конкретные изменения, затрагивающие агента, хода диалога и набора тестов.