In this role, you will leverage deep expertise in cloud-native technologies like Databricks, Azure DevOps, Delta Lake, and Spark Structured Streaming within a Microsoft Azure environment to lead and guide teams in designing, implementing, and maintaining scalable, performant, and transparent data-processing pipelines for critical financial data. You will take ownership of technical initiatives, drive thought leadership, and push the boundaries of system scalability and efficiency while fostering cross-functional collaboration.

If you are ready to take on a leadership role in a collaborative and innovative environment, we want to hear from you!

#LI-DNI

Responsibilities

Lead the design, development, maintenance, and optimization of efficient and scalable data pipelines using Python and Apache Spark
Architect streaming and batch solutions leveraging Delta Lake tables and Spark Structured Streaming, ensuring high system performance and dependability
Collaborate closely with business partners, IT experts, architects, and engineers to gather requirements, define project goals, and design strategies
Mentor and support team members through code reviews, technical training, and collaborative problem-solving
Utilize relational data models and SQL to manage, query, and finesse large-scale data workflows
Develop and enforce automated testing strategies to ensure high-quality, reliable, and maintainable solutions
Monitor, troubleshoot, and anticipate complex data processing issues while implementing proactive measures to maintain operational stability across systems
Document technical designs, processes, and architecture best practices with precision and forward-thinking clarity
Identify and drive initiatives for continuous improvement, from operational efficiency to technical scalability, within data pipelines and team processes
Engage with product owners, senior leadership, and team members to prioritize and deliver impactful tools and solutions aligned with business goals
Take ownership of technical challenges and act as a primary point of contact for critical decisions and escalations within data engineering initiatives

Requirements

5+ years of experience in a software engineering or data engineering role working with large-scale distributed data systems
At least 1 year of relevant leadership experience
Advanced expertise in Python, Apache Spark, and Delta Lake
Deep familiarity with relational data models, SQL, and complex data transformations
Proven mastery of cloud-native technologies within the Microsoft Azure ecosystem, including Databricks and Azure DevOps
Strong experience in designing and maintaining reliable automated testing frameworks for data solutions
Exceptional analytical skills to break down complex problems, design actionable strategies, and deliver elegant solutions
Demonstrated ability to mentor and coach team members to elevate overall technical and professional outcomes
Fluency in English and excellent communication skills to lead and collaborate effectively with distributed cross-functional teams

Nice to have

Deep understanding of Spark Structured Streaming for real-time and near real-time data processing use cases
Significant experience working with financial data or similarly complex data domains requiring high accuracy and efficiency
Proven track record in identifying and resolving performance bottlenecks in distributed systems, with a focus on both technical and organizational solutions
Strong proficiency in documenting long-term data pipeline architecture and implementation approaches, specifically tailored for scalability and best practice alignment
Prior experience in leading technical decision-making and steering large-scale data engineering projects to successful completion