Key Responsibilities

Lead the design, architecture, and implementation of Azure Databricks solutions for data processing and analytics.
Develop and optimize ETL/ELT pipelines using Apache Spark, Python (PySpark), Scala, and SQL.
Work closely with data engineers, data scientists, and business stakeholders to ensure efficient data workflows.
Implement data governance, security, and compliance best practices in Databricks and Azure Data Lake.
Optimize performance tuning, cost management, and scalability of Databricks workloads.
Automate CI/CD pipelines for Databricks notebooks using DevOps tools like Azure DevOps, Terraform, or GitHub Actions.
Troubleshoot and resolve issues related to big data processing, Spark jobs, and cloud storage.
Stay updated with Azure and Databricks best practices and mentor junior team members.
Required Skills & Experience:

10+ years of experience in data engineering, big data, or cloud-based data solutions.

Strong hands-on experience with Azure Databricks, Apache Spark, and Delta Lake.
Expertise in Python (PySpark), Scala, SQL, and data transformation techniques.
Deep understanding of Azure Data Services (Azure Data Lake, Azure Synapse, Azure Data Factory, etc.).
Experience with data modeling, ETL/ELT design, and distributed data processing.
Proficiency in DevOps practices, including CI/CD pipelines, Infrastructure as Code (IaC), and version control (Git).
Familiarity with MLflow, Unity Catalog, and Databricks SQL is a plus.
Strong analytical, problem-solving, and communication skills.

Preferred Qualifications

Databricks certifications (e.g., Databricks Certified Data Engineer Associate/Professional).
Experience with containerization (Docker, Kubernetes) and orchestration tools.
Knowledge of real-time data streaming using Kafka, Event Hubs, or similar technologies.
Exposure to Machine Learning (ML) workflows in Databricks is a plus.

Data and Analytics - Lead