Lead Data Engineer
EPAM Systems
Software Engineering, Data Science
Ukraine
Posted on Saturday, April 27, 2024
Lead Data Engineer Description
DESCRIPTION
Our client is an American video game company.
Responsibilities
- Migrate current Data pipelines and DAGs from GCP BigQuery stored procedures into AWS Redshift
- Perform necessary data backfills and validate the quality and integrity during and post-migration
- Design, develop, and maintain data pipelines using AWS services such as Amazon Redshift (or any MPP solution), Amazon S3, dbt and Apache Airflow
- Utilize SQL and ETL/ELT solutions to transform and load data from various sources into data warehouses and data lakes
- Optimize data workflows and processes to ensure efficiency, scalability, and reliability
- Implement auto-updated mechanisms to keep data pipelines and systems up-to-date with the latest data
- Collaborate with cross-functional teams to understand data requirements and deliver solutions that meet business needs
- Monitor and troubleshoot data pipelines to identify and resolve issues in a timely manner
- Perform data assessments and analytics to provide insights and recommendations for data-driven decision-making
Requirements
- 5+ years of experience as a Data Engineer
- Advanced SQL, PLSQL, TSQL for data transformation and optimization
- Experience with writing ETL/ELT code, managing and deploying DAGs using DBT or Airflow
- Hands-on experience working with AWS RedShift, if not have you worked with similar Cloud EDW systems
- Upper-Intermediate level of English, both spoken and written (B2+)
Nice to have
- Familiarity with Google Cloud Platform (GCP) and Google Cloud BigQuery for data storage and analytics
- Knowledge of AWS migration tools for transitioning data and workloads from Google Cloud Platform to AWS
- Experience with analytics and data assessment techniques to evaluate data quality, integrity, and usability
- Proficiency in BI testing methodologies and tools for validating business intelligence reports and dashboards
- Background in data analytics engineering with a focus on building scalable and efficient data solutions
We Offer
- Competitive compensation depending on experience and skills
- Individual career path
- Unlimited access to LinkedIn learning solutions
- Sick leave and regular vacation
- English classes with certified English teachers
- Flexible work hours