Senior Data Engineer

Job Summary:

For one of our clients, we are seeking an experienced Data Engineer to join our dynamic team. The ideal candidate will have a solid background in data engineering, with expertise in Python, ETL tools, Airflow, and Hadoop. You will be responsible for building, maintaining, and optimizing data pipelines that handle high-volume transactions exceeding 50 millions per day.

If you're excited about solving challenging problems, delivering polished user experiences, and contributing to a fast-paced and innovative environment, we'd love to meet you.

Key Responsibilities:

• Design, develop, and maintain scalable data pipelines and data models for large transaction systems.
• Utilize Python and ETL tools to extract, transform, and load data efficiently from various sources.
• Implement and manage workflows using Apache Airflow to automate data processing tasks.
• Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and ensure data quality and availability.
• Optimize data processing performance and reliability in a Hadoop environment.
• Monitor and troubleshoot data pipeline issues to ensure operational efficiency and reliability.
• Document processes, data flow, and architecture for future reference and compliance.
• Stay current with industry trends and emerging technologies in data engineering.

Required Qualifications:

• 5+ years of experience in data engineering or a similar role.
• At least 3 years of hands-on experience with Apache Spark and/or AWS Glue. 
• Strong English communication skills and the ability to work collaboratively in a team environment.
• Strong proficiency in Python for data processing and manipulation.
• Hands-on experience with ETL tools (e.g., Talend, Informatica, Apache NiFi).
• In-depth knowledge of Apache Airflow for workflow management.
• Experience with big data technologies such as Hadoop and its ecosystem (e.g., HDFS, Hive, Spark).
• Proven experience in building and maintaining big transaction systems (handling over 50 million transactions per day).
• Familiarity with cloud platforms (e.g., AWS, Azure, GCP) is a plus.
• Experience with real-time data processing technologies such as Kafka and Flink is highly preferred.
• Excellent analytical and problem-solving skills.