Senior Data Engineer

For one of our client, we are seeking an experienced Data Engineer to join our dynamic team. The ideal candidate will have a solid background in data engineering, with expertise in Python, ETL tools, Airflow, and Hadoop. You will be responsible for building, maintaining, and optimizing data pipelines that handle high-volume transactions exceeding 50 million per day.

Key Responsibilities

- Design, develop, and maintain scalable data pipelines and data models for large transaction systems.

- Utilize Python and ETL tools to extract, transform, and load data efficiently from various sources.

- Implement and manage workflows using Apache Airflow to automate data processing tasks.

- Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and ensure data quality and availability.

- Optimize data processing performance and reliability in a Hadoop environment.

- Monitor and troubleshoot data pipeline issues to ensure operational efficiency and reliability.

- Document processes, data flow, and architecture for future reference and compliance.

- Stay current with industry trends and emerging technologies in data engineering.

Requirements

- 8+ years of experience in data engineering or a similar role.

- Strong proficiency in Python for data processing and manipulation.

- Hands-on experience with ETL tools (e.g., Talend, Informatica, Apache NiFi).

- In-depth knowledge of Apache Airflow for workflow management.

- Experience with big data technologies such as Hadoop and its ecosystem (e.g., HDFS, Hive, Spark).

- Proven experience in building and maintaining big transaction systems (handling over 50 million transactions per day).

- Familiarity with cloud platforms (e.g., AWS, Azure, GCP) is a plus.

- Excellent analytical and problem-solving skills.

- Strong communication skills and the ability to work collaboratively in a team environment.