Senior Data Engineer (Python, Databricks)
EPAM Systems
This job is no longer accepting applications
See open jobs at EPAM Systems.See open jobs similar to "Senior Data Engineer (Python, Databricks)" FinTech Australia.Software Engineering, Data Science
Portugal
Posted 6+ months ago
Senior Data Engineer (Python, Databricks) Description
DESCRIPTION
Currently, we are looking for a Senior Data Engineer with Python and Databricks expertise to make our team even stronger.
Responsibilities
- Develop, monitor, and operate the most used and most critical curated data pipeline - Sales Order Data (incl. Post-order information, e.g. shipment, return, payment). This pipeline is processing hundreds of millions of records to provide high-quality datasets for analytical and machine learning use-cases
- Consul with analysts, data scientists, and product managers to build and continuously improve "Single Source of Truth" KPI for business steering such as the central Profit Contribution measurement (PC II)
- Redevelop old legacy pipelines to new, advanced and standard versions that are easy to maintain and scalable for future demands
- Leverage and improve a tech stack that includes Python, Databricks, Kubernetes, Spark, and Airflow
Requirements
- Fluency in Python programming language
- Good hands on experience with Databricks
- Expertise in Apache Spark along with Spark streaming
- Good understanding & hands-on experience with CI/CD
- Rich working experience with Github
- Upper-Intermediate level of English, both spoken and written (B2+)
- Team player (easy & respectful communications, shares responsibilities for the team overall success)
- Communication skills
- Organizational skills
Nice to have
- Delta-Lake
- Expertise in SQL
- Fluency working with AWS landscape
- Ability to build Apache Airflow pipelines
- Presto
- Superset
- Starburst
- Oracle & Exasol
We Offer
- Competitive compensation depending on experience and skills
- Variety of projects within one company
- Being a part of a project following engineering excellence standards
- Individual career path and professional growth opportunities
- Internal events and communities
- Flexible work hours
This job is no longer accepting applications
See open jobs at EPAM Systems.See open jobs similar to "Senior Data Engineer (Python, Databricks)" FinTech Australia.