Lead Data Engineer

Experience Required - 8+ Years.

Work Locations - Bengaluru, Chennai, Hyderabad, Pune.

Notice Period - max 30 Days.

Primary Skills –

Spark Structured Streaming, Apache Spark, Scala, Apache Kafka, Databricks, Any cloud, preferably GCP.

In depth coding knowledge in Spark Structured Streaming.

Kafka producers and Spark structure streaming consumers.

• Strong technical knowledge on Big Data Technologies ( Data bricks, Spark Streaming , GCP , Scala )
• Strong technical skills in Spark Structure streaming, Scala, python, data bricks for lake flow engineering and orchestration, Cloud SQL, Git.
• Spark internals and execution architecture.
• Spark execution concepts - including query planning, DAG generation, shuffle operations, partitioning strategies, and memory management.
• Understanding of Spark optimization techniques such as broadcast joins, caching/persistence strategies, partition pruning, adaptive query execution, and skew handling
• Databricks platform expertise and optimization techniques - cluster sizing strategies, job optimization, Delta Lake internals, workflow orchestration, and performance tuning.
• Advanced batch and streaming ingestion patterns.
• Data Ingestion and Architecture Patterns - Exposure to ingestion frameworks and modern data integration patterns.
• Performance and Optimization Experience - troubleshooting performance bottlenecks, optimizing large-scale data pipelines, or implementing cost and performance improvements.
• Performance tuning and troubleshooting at scale.
• Data architecture and design decision-making.
• Technical leadership, mentoring, and stakeholder communication.

Key Responsibilities

• Possess very strong communication skills and stakeholder management.
• Hands on project technical expertise and involve 80% in technical work and 20% management activities
• Carries very strong Development project experience and manage a team of 10 to 15 technical project team.
• Involve in Technical architecture review meetings and understanding the proposed technical architecture for the project requirements
• Provide required technical mentoring to delivery team
• Efficient coordination with Onshore Tech Lead on the deliverables and ensure to meet quality & schedule adherence
• Report the Monthly Metrics (Internal & Client)
• Generate all the required reports.
• Review and arrange training for the team depending on the need basis.
• Part of deployment activities
• Good analytical and Problem-solving skills