Design and deploy automated testing frameworks and strategies for data engineering processes

Integrate with technologies such as Databricks, PySpark, Scala, and Spark SQL

Build automated tests to verify data accuracy, consistency, and reliability throughout platforms

Incorporate automated testing into Azure DevOps CI/CD pipelines for streamlined testing, building, and deployment

Conduct performance testing to evaluate the scalability and efficiency of data processing solutions

Collaborate with data engineers and stakeholders to gather requirements and align testing strategies with business needs

Set up monitoring for automated tests and provide ongoing reports on test coverage, defects, and data quality issues

At least 3 years of experience in Data Quality Engineering or a related field

Thorough understanding of automated testing frameworks and data engineering concepts

Practical experience with Databricks, PySpark, Scala, and Spark SQL

Knowledge of Terraform and Azure Event Hubs

Ability to integrate testing frameworks with Azure DevOps and manage source control using Git

Experience with pipeline configuration and deployment using Terraform

Familiarity with Jira for task and issue management

Strong analytical and troubleshooting skills for complex data and testing problems

Effective English communication skills, both written and spoken, at B2+ level or higher

Background with Power BI for data visualization

Experience with Azure Data Lake and Spark Streaming

Exposure to large-scale or data-intensive industries

Ability to implement Infrastructure as Code (IaC) solutions