Data Engineer

Position: Data Engineering Experience: 6+ Years Salary: Up to 27 LPA Roles and Responsibilities: - Design, develop, and maintain production-grade data pipelines using Databricks. - Build scalable data processing solutions using Delta Lake and Databricks SQL. - Develop and manage Databricks Workflows for orchestration, scheduling, monitoring, and dependency management. - Create reusable ingestion and transformation frameworks for enterprise-scale data processing. - Develop data pipelines to ingest data from REST APIs and external systems. - Implement API ingestion frameworks supporting authentication, pagination, retries, rate limits, and incremental loads. - Transform and structure raw API data for analytics and reporting use cases. - Design and implement dimensional data models including fact and dimension tables. - Write and optimize advanced SQL queries for transformation, validation, aggregation, and analytics. - Develop reusable Python utilities, libraries, and automation scripts for data engineering operations. - Implement data quality frameworks, validation checks, reconciliation processes, and monitoring mechanisms. - Configure and manage governed data assets using Unity Catalog. - Support data governance, lineage, metadata management, and access control implementation. - Monitor and troubleshoot production data pipelines and perform root-cause analysis for failures. - Optimize Databricks workloads for scalability, reliability, and cost efficiency. - Collaborate with architects, analysts, DevOps teams, and business stakeholders to deliver scalable data solutions. - Support CI/CD and deployment practices for data engineering workloads. Skills Required: - Strong hands-on experience with Databricks in production environments. - Expertise in Delta Lake, Databricks SQL, Databricks Workflows, and Unity Catalog. - Strong experience building REST API-based data ingestion pipelines. - Advanced SQL skills including joins, CTEs, window functions, aggregations, and query optimization. - Strong Python programming experience for data engineering and pipeline development. - Experience with dimensional data modelling and analytical data structures. - Hands-on experience implementing data quality frameworks and automated validation checks. - Strong understanding of data ingestion, orchestration, transformation, and production support. - Experience troubleshooting data pipeline, integration, and performance issues. - Knowledge of data governance, security, metadata management, and access controls. - Experience with Medallion Architecture (Bronze, Silver, Gold layers) is preferred. - Exposure to Delta Live Tables (DLT) is preferred. - Experience with Azure services such as Azure Data Factory, Azure Data Lake Storage, Azure Functions, and Azure Synapse is an advantage. - Familiarity with Git, Azure DevOps, and CI/CD practices is preferred. - Exposure to telemetry, AI/engineering-tool data integration, and observability frameworks is an added advantage. Other Details: - Work Location: Remote - Shift Timing: Starts at 12:30 PM IST - Employment Type: Full-time - Budget: Up to 27 LPA - Opportunity to work on enterprise-scale data engineering and governed analytics platforms. About the Client: Our client is a global technology and AI-focused enterprise specializing in advanced data, analytics, and enterprise AI solutions for large-scale organizations across multiple industries. With a strong focus on innovation, the organization develops and deploys production-grade AI and machine learning systems, leveraging modern cloud, data engineering, and automation technologies. The company is known for its research-driven approach, continuous investment in emerging technologies, and expertise in building scalable AI-powered platforms for enterprise transformation. Disclaimer: About job