Create and implement automated testing frameworks and strategies for data engineering workflows

Integrate solutions with Databricks, PySpark, Scala, Spark SQL, and other relevant technologies

Develop automated tests to ensure data accuracy, consistency, and reliability across platforms

Embed automated testing into Azure DevOps CI/CD pipelines for efficient testing, building, and deployment

Perform performance testing to assess the scalability and efficiency of data processing systems

Work with data engineers and stakeholders to collect requirements and align testing approaches with business objectives

Establish monitoring for automated tests and deliver regular updates on test coverage, defects, and data quality concerns

Minimum of 3 years’ experience in Data Quality Engineering or a related discipline

Comprehensive understanding of automated testing frameworks and data engineering principles

Hands-on experience with Databricks, PySpark, Scala, and Spark SQL

Familiarity with Terraform and Azure Event Hubs

Ability to connect testing frameworks with Azure DevOps and manage source control using Git

Knowledge of pipeline setup and deployment through Terraform

Experience using Jira for tracking tasks and issues

Strong analytical and problem-solving skills for addressing complex data and testing challenges

Proficient English communication skills, both written and spoken, at B2+ level or higher

Experience working with Power BI for data visualization

Knowledge of Azure Data Lake and Spark Streaming

Background in industries with large-scale or data-intensive environments

Skill in implementing Infrastructure as Code (IaC) solutions