AI Data Engineer
- Posted 20 days ago
- Be among the first 20 applicants
Job Description
- Competitive package with strong career growth potential
- Permanent position with stable work environment
Our client is a well-established organisation investing heavily in modern cloud, data, and AI capabilities to enable digital transformation and innovation across multiple business functions.
Job Description
- Design, develop, and support large-scale batch and real-time data pipelines on modern cloud platforms.
- Build robust data integration solutions across APIs, databases, files, and event-driven systems.
- Develop and maintain knowledge repositories and retrieval-based AI solutions to support emerging GenAI use cases.
- Provide technical leadership on platform architecture, code quality, deployment standards, and operational excellence.
- Drive platform reliability through monitoring, troubleshooting, release management, and continuous improvement initiatives.
- Degree in Computer Science, Engineering, Information Systems, or a related discipline
- At least 5 years of experience in data engineering or cloud data platform delivery.
- Strong hands-on experience with Databricks, Apache Spark (PySpark), and SQL, with exposure to technologies such as Python, Kafka, Delta Lake, and Microsoft Fabric considered advantageous.
- Proven track record designing and supporting production-grade batch and streaming pipelines, including orchestration, monitoring, troubleshooting, and performance optimisation.
- Experience implementing Knowledge Base and RAG solutions, including embeddings, vectorisation, retrieval mechanisms, and GenAI applications.
- Good understanding of CI/CD, data governance, cloud security, access management, and platform operations, with strong stakeholder communication skills.
- Competitive package with strong career growth potential
- Opportunity to work on cutting-edge cloud, data, and AI transformation programmes.
- Exposure to enterprise-scale platforms, stakeholders, and high-impact business initiatives.
More Info
Key Skills
About Company
Skills:
Orchestration, Cloud Security, Pyspark, Apache Spark, Kafka, Sql, Databricks, Data Governance, Python, Platform Operations, Retrieval mechanisms, Performance optimisation, Knowledge Base and RAG solutions, Batch and streaming pipelines, Access Management, Troubleshooting, Embeddings, Vectorisation, GenAI applications, Delta Lake, CI/CD, Microsoft Fabric, Monitoring
Skills:
Pyspark, Apache Spark, Kafka, Sql, Git, Databricks, Python, RAG solution stacks, embeddings, CI CD workflows, vector databases, cloud-native observability, Delta Lake
Skills:
Data Governance, Data Lineage, Python, Metadata Management, Pandas, Data Transformation, Distributed Systems, FastAPI, Rest Apis, Apache Iceberg, LangGraph, Pydantic, Trino, Observability, Monitoring, LangChain, Parquet, DuckDB, Microservices patterns, Data ingestion, dbt, CI/CD, tracing, SDKs, Event-driven architectures
Skills:
Python, Sql, Spark, Hadoop, Data Modeling, Machine Learning, Cloud Computing, Apis, Data Warehousing, ETL Processes
Skills:
Pyspark, Kafka, Databricks, Sql, Delta Lake, Microsoft Fabric