Automated Testing in Python & 9 others
EPAM Systems
Software Engineering
Belarus
Posted on Nov 19, 2025
Responsibilities
- Audit the current LLM and RAG test automation framework, identify gaps, and propose improved architecture
- Implement automated testing pipelines incorporating RAGAS metrics and LLM-as-a-judge methods for quality assessment
- Integrate workflows with AWS services (S3, Lambda, CloudWatch, OpenSearch, RDS, SQS) for seamless operations
- Define clear evaluation rubrics, metric thresholds, regression alerts, and reporting dashboards to improve monitoring
- Ensure reproducible, scalable testing solutions within CI/CD pipelines
Requirements
- 3+ years of experience in LLM and RAG evaluation, including RAGAS and prompt-based judging automation
- Strong competency in AWS services (compute, storage, orchestration, monitoring)
- Background in vector databases and semantic similarity metrics
- Ability to establish KPIs and translate them into automated testing logic
- Proficiency in Python or Java
Nice to have
- Familiarity with LangChain, LlamaIndex, or similar RAG frameworks
- Understanding of CI/CD processes
We offer/Benefits
We connect like-minded people:
- Delivering innovative solutions to industry leaders, making a global impact
- Enjoyable working environment, whether it's the vibrant office or the comfort of your own home
- Opportunity to work abroad for up to two months per year
We invest in your growth:
- Leadership development, career advising, soft skills and well-being programs
- Certifications, including GCP, Azure and AWS
- Unlimited access to LinkedIn Learning and Get Abstract
- Free English classes with certified teachers
We cover it all:
- Monetary bonuses for taking part in the referral program
- Comprehensive medical & family care package
- Seven trust days per year (sick leave without a medical certificate)
- Discounts from 800+ partners (sports activities, restaurants, stores and services)