Review existing LLM/RAG test automation workflows, identify gaps, and design an improved testing architecture
Implement automated test pipelines that utilize RAGAS for retrieval/generation evaluation and LLM-as-a-judge for subjective quality and safety checks
Integrate testing pipelines with AWS services such as S3, Lambda, CloudWatch, OpenSearch, RDS, and SQS
Define and manage evaluation rubrics, set metric thresholds, implement regression alerting systems, and generate reporting dashboards
Ensure scalability, reproducibility, and continuous quality monitoring within CI/CD pipeline environments
Collaborate with AI/ML engineers, DevOps teams, and product stakeholders to align testing metrics with project goals
Stay updated on advancements in LLM and RAG evaluation frameworks, integrating relevant improvements into the testing processes
Troubleshoot, document, and resolve automation framework issues to ensure robust performance and reliability
Contribute to knowledge sharing, documentation, and training sessions related to the RAG testing framework

Requirements

3+ years of proven experience in LLM and RAG evaluation frameworks, including RAGAS and prompt-based judging automation
Knowledge of AWS cloud services, particularly compute, storage, orchestration, and monitoring solutions
Familiarity with vector databases and principles of semantic similarity metrics
Background in Python or Java, with the capability to apply these in automated testing environments
Understanding of KPI formulation and the ability to translate KPIs into actionable, automated testing logic
Skills in defining and improving testing pipelines within complex AI systems
Excellent command of written and spoken English (B2+ level)

Nice to have

We offer/Benefits

With us you can:

Work on a flexible schedule remotely or from any of our comfortable offices or coworking spaces in Ukraine
Receive the necessary equipment to perform your work tasks
Change projects and technology stacks within EPAM
Gain experience in various business domains (Insurance, E-commerce, Healthcare, Finance, Travelling, Media, Artificial Intelligence, and more)
Relocation opportunities may be available for eligible candidates, depending on the role and openings at other EPAM locations
Participate in volunteer, charity programs and communities (both technical and interest-based)

We focus on your professional growth:

You can plan your individual career path together with your manager
Receive regular feedback from colleagues
Improve your English for free with certified teachers (Speaking Clubs, client interview preparation courses, etc.)
Get the opportunity to undergo free training and certification in AWS, GCP, or Azure Clouds
Use the internal E-learn training program (18,200+ specialized training and mentoring programs)
Access corporate accounts on LinkedIn Learning, Get Abstract and other partner resources
Study at EPAM Solution Architecture School with the instructors who are practicing architects
Develop as a leader, join Delivery Management, Resource Management, Leadership Essentials school and more
Participate in internal communities (500+ meetups, technical discussions, brainstorming sessions, online events and conferences annually)

What we offer:

Vacation and sick leave (including a sick leave without a medical certificate)
A wide range of Voluntary Medical Insurance programs providing both medical treatment and various preventive options (including sports activities)
Medical insurance for family members at corporate rates
Company support during significant life events (childbirth or adoption, marriage, etc.)
Support for psychological comfort: discounts on services from mental health specialists or coaches, thematic training
E-kids program - a free programming language training program for EPAMers' children

AI Engineering & 9 others