Develop and implement automated testing frameworks and strategies for data engineering workflows

Work with technologies including Databricks, PySpark, Scala, and Spark SQL

Create automated tests to validate data accuracy, consistency, and reliability across systems

Integrate automated testing into Azure DevOps CI/CD pipelines for efficient testing, building, and deployment

Perform performance testing to assess the scalability and effectiveness of data processing solutions

Partner with data engineers and stakeholders to collect requirements and align testing approaches with business objectives

Establish monitoring for automated tests and deliver regular updates on test coverage, defects, and data quality metrics

Minimum of 3 years’ experience in Data Quality Engineering or a related discipline

Comprehensive understanding of automated testing frameworks and data engineering principles

Hands-on experience with Databricks, PySpark, Scala, and Spark SQL

Knowledge of Terraform and Azure Event Hubs

Ability to connect testing frameworks with Azure DevOps and manage source control using Git

Experience with pipeline configuration and deployment through Terraform

Familiarity with Jira for managing tasks and issues

Strong analytical and troubleshooting skills for resolving complex data and testing challenges

Proficient English communication skills, both written and spoken, at B2+ level or higher

Experience using Power BI for data visualization

Familiarity with Azure Data Lake and Spark Streaming

Background in industries with large-scale or data-intensive environments

Skill in implementing Infrastructure as Code (IaC) solutions