Design and build automated testing frameworks and strategies for data engineering pipelines

Focus on integration with Databricks, PySpark, Scala, Spark SQL, and related technologies

Develop and implement automated tests to validate data integrity, quality, and reliability across all data platforms

Integrate automated testing processes within Azure DevOps CI/CD pipelines for streamlined testing, building, and deployment

Conduct performance testing to evaluate the efficiency and scalability of data processing systems

Collaborate with data engineers and stakeholders to gather requirements and align testing strategies with business needs

Set up monitoring for automated tests and provide regular reports on test coverage, defects, and data quality issues

At least 3 years of experience in Data Quality Engineering or a similar field

Strong understanding of automated testing frameworks and data engineering concepts

Proficiency with Databricks, PySpark, Scala, and Spark SQL

Experience with Terraform and Azure Event Hubs

Ability to integrate testing frameworks with Azure DevOps and manage source control using Git

Knowledge of pipeline configuration and deployment using Terraform

Familiarity with Jira for task and issue tracking

Strong analytical and troubleshooting skills for resolving complex data and testing issues

Fluent English communication skills, both written and spoken, at B2+ level or higher

Experience with Power BI for data visualization

Familiarity with Azure Data Lake and Spark Streaming

Background in data-intensive industries or large-scale data environments

Proficiency in implementing solutions using Infrastructure as Code (IaC)