In this role, you will bring extensive expertise in cloud-native technologies such as Databricks, Azure DevOps, Delta Lake, and Spark Structured Streaming within a Microsoft Azure environment to steer the strategic direction of our data processing initiatives. You will oversee the design, implementation, and evolution of scalable, performant, and transparent data pipelines for critical financial data while establishing and driving technical standards across the organization.

This role demands a proactive leader who can inspire teams, foster innovation, and maintain a relentless focus on operational excellence, scalability, and efficiency.

If you are ready to take the helm as a chief leader in a dynamic, fast-paced, and forward-thinking environment, we want to connect with you.

#LI-DNI

Responsibilities

Drive the strategic vision and technical direction for the design, development, maintenance, and optimization of enterprise-scale data pipelines using Python and Apache Spark
Shape architectural strategies for streaming and batch solutions leveraging Delta Lake tables and Spark Structured Streaming while ensuring high performance, availability, and system dependability
Collaborate with senior executives, business leaders, architects, and engineers to translate business goals into scalable technology solutions and actionable priorities
Champion mentorship and professional development for teams, fostering a culture of continuous learning and technical excellence
Establish and enforce standards for relational data models, SQL workflows, and complex data transformations, ensuring alignment with organizational goals
Develop robust governance frameworks for automated testing strategies, ensuring high-quality, reliable, and scalable data solutions throughout the organization
Design proactive monitoring systems to anticipate, troubleshoot, and resolve complex data-processing issues while driving operational resilience across projects
Spearhead the documentation and communication of technical designs, system processes, and architecture best practices to align teams and stakeholders with long-term vision
Identify, prioritize, and execute transformational initiatives focused on continuous improvement, innovation, and optimization across systems and processes
Engage regularly with product owners, senior leadership, cross-functional stakeholders, and external partners to craft impactful solutions and drive measurable business results
Act as a decisive leader for mission-critical technical challenges, ensuring timely and effective solutions while maintaining executive confidence and organizational alignment

Requirements

7+ years of experience in software engineering or data engineering roles, with a proven history of overseeing large-scale distributed data systems
2+ years of relevant leadership experience, including managing cross-functional teams and driving technical strategy
Expert-level proficiency in Python, Apache Spark, and Delta Lake with a demonstrated ability to solve complex technical challenges
Comprehensive expertise in relational data models, SQL, and data transformation techniques for scaling large, complex data workflows
Proven mastery of cloud-native technologies within the Microsoft Azure ecosystem, encompassing Databricks, Azure DevOps, and other related platforms
Extensive experience in designing governance frameworks and maintaining reliable automated testing processes for large-scale data solutions
Exceptional analytical and strategic thinking skills to conceptualize, communicate, and implement scalable solutions across diverse technical and business domains
Demonstrated ability to mentor and coach a team of engineers while enhancing organizational performance through technical guidance and leadership
Fluency in English and outstanding communication skills to drive collaboration across distributed cross-functional teams as well as with senior leaders

Nice to have

In-depth expertise in Spark Structured Streaming for developing and managing real-time and near-real-time data solutions
Comprehensive experience working with financial data or analogous domains requiring high accuracy, efficiency, and governance
Proven ability to diagnose and resolve performance bottlenecks in distributed systems while simultaneously optimizing organizational workflows
Strong capability in documenting strategic, long-term data pipeline architectures tailored for enterprise scalability and aligned with industry best practices
A distinguished track record in leading technical decision-making processes and successfully steering large-scale data engineering projects to completion