Responsibilities:
The candidate will be responsible for building and operating the company's end-to-end data system, including:
Track P Pipeline & Platform
- Design and develop data pipelines from Bronze to Silver to Gold layers.
- Build batch and streaming pipelines, handling late-arriving data, CDC (Change Data Capture), and schema drift.
- Optimize Spark performance, including partitioning, Z-ordering, and compaction.
- Develop shared frameworks and reusable components (e.g., joins, utilities, data quality pre-checks).
Track A Analytics & Reporting
- Understand business requirements and KPI definitions.
- Design Gold and Diamond data models for analytical purposes.
- Build complex reports and transformations using SQL and PySpark.
- Implement standardized incremental and backfill workflows.
Track R Data Reliability & Monitoring
- Build and maintain Data Quality frameworks (e.g., row count checks, schema validation, metric consistency).
- Perform cross-table and cross-source data validation.
- Implement anomaly detection and alerting mechanisms.
- Conduct root-cause analysis and reduce recurring data issues.
Requirements:
- 25 years of experience in Data Engineering or related roles.
- Strong proficiency in SQL, Spark SQL, and PySpark.
- Hands-on experience with Databricks/Spark environments.
- Solid understanding of data modeling concepts: fact/dimension tables, grain, star schema, SCD.
- Strong analytical thinking and problem-solving skills, with the ability to troubleshoot pipeline issues and optimize data processes.
What we offer:
- Competitive Compensation: Enjoy a salary package tailored to your skills and experience, along with performance-based bonuses.
- Comprehensive Benefits: We support your well-being with allowances, and assistance with work visa processing.
- Work-Life Balance: Unwind with generous holiday and New Year bonuses.
- Top-Tier Equipment: Stay productive with the latest tools, including a MacBook and iPhone.
- Thriving Culture: Immerse yourself in a dynamic, inclusive work environment that fosters growth.
