Write batch processing pipelines, collaborating with ingestion teams to create new pipelines

Troubleshoot and fix production issues, testing the created code, and implementing unit and integration tests

Do data modeling for new data products and read data from Kafka and process it in batch mode

Deploy solutions to production and ensure smooth operation of the pipelines

Collaborate with cross-functional teams to understand business requirements and translate them into technical solutions

Ensure data quality, integrity, and security throughout the data processing lifecycle

Contribute to the development of best practices and standards for Big Data engineering and analytics

A minimum of 3 years of experience in Data Software Engineering, with a focus on Big Data technologies and platforms

Expertise in Snowflake, Amazon Web Services, Python, and SQL

Strong experience with Apache Hadoop, Apache Kafka, Apache Spark, and Apache Impala

Experience in writing batch processing pipelines using Spark Scala, and deploying solutions to production

Strong skills in data modeling for new data products

Good understanding of data ingestion, processing, and storage concepts

Hands-on experience in testing the created code, implementing unit and integration tests, and troubleshooting production issues

Excellent communication skills and strong critical thinking capabilities to effectively convey feedback and insights

Fluent spoken and written English at an Upper-Intermediate level or higher, enabling effective communication

Experience with real-time data processing technologies such as Apache Flink or Apache Storm

Knowledge of containerization technologies such as Docker and Kubernetes

Experience in working with NoSQL databases such as MongoDB or Cassandra

Familiarity with machine learning concepts and frameworks such as TensorFlow or PyTorch

Experience with data visualization tools such as Tableau or Power BI