In this role, you will harness your expertise in cloud-native technologies like Databricks, Azure DevOps, Delta Lake, and Spark Structured Streaming within a Microsoft Azure environment. You will play a pivotal part in designing, implementing, and maintaining scalable, performant, and transparent data-processing pipelines for critical financial data, pushing the boundaries of system scalability and efficiency while collaborating across teams.

If you are eager to take on an exciting challenge in a collaborative and innovative environment, we want to hear from you!

#LI-DNI

Responsibilities

Develop, maintain, and optimize efficient and scalable data pipelines using Python and Apache Spark
Build streaming and batch solutions leveraging Delta Lake tables and Spark Structured Streaming
Collaborate closely with business partners, IT experts, and architects to gather requirements, design solutions, and refine processes
Participate in peer code reviews to ensure code quality and adherence to best practices
Utilize relational data models and SQL to manage, query, and process data effectively
Implement automated testing strategies for data solutions to ensure high reliability and maintainability
Monitor, troubleshoot, and resolve complex data processing issues to support operational stability
Document technical designs, processes, and decisions with clarity and precision
Proactively identify areas for improvement, recommending and implementing solutions
Engage with product owners and team members to prioritize and deliver high-impact solutions

Requirements

3+ years of experience in a software engineering or data engineering role working with large-scale data systems
Significant experience in Python, Apache Spark, and Delta Lake
Knowledge of relational data models, SQL, and using them for complex data handling
Expertise in cloud-native technologies within the Microsoft Azure ecosystem including Databricks and Azure DevOps
Familiarity with automated testing practices and the ability to design reliable test strategies
Proficiency in analyzing complex problems, breaking them into actionable steps, and delivering elegant solutions
Fluency in English and strong communication skills to work effectively within distributed and collaborative teams

Nice to have

Understanding of Spark Structured Streaming for real-time and near real-time data processing solutions
Showcase of working on financial data or similar high-complexity data domains
Competency in identifying and resolving performance bottlenecks in distributed data systems
Familiarity with documenting data pipeline architecture and best practice implementation approaches