Azure Data Engineer
- Posted a month ago
- Over 200 applicants have applied
Job Description
- Designing and building data pipelines to collect, process, transform, and store data from various sources. This involves understanding the data flow, selecting the appropriate technologies, and creating scalable and efficient data processing workflows.
- Ensure that the data pipeline architecture is reliable and produces high-quality data. This involves monitoring the data pipeline, identifying and resolving issues, and implementing data validation and testing mechanisms.
- Work closely with data scientists, data analysts, software engineers, and other stakeholders to understand their data requirements and ensure that the data pipeline architecture meets their needs.
- Optimizing the data pipeline architecture to ensure efficient and scalable data processing. This involves identifying bottlenecks, optimizing data processing workflows, and selecting appropriate technologies and tools.
- Ensure security and privacy of data in the pipeline architecture. This involves implementing appropriate security measures, such as data encryption, access controls, and auditing.
- Documentation of the data pipeline architecture, data processing workflows, and data sources. This involves creating technical documentation, data flow diagrams, and reports that communicate the performance and quality of the data pipeline to stakeholders.
- Qualification and Experience:
- Qualification:
- A bachelor's in computer science, data science, software engineering, or a related field
- Experience:
- 5+ years of relevant experience in data engineering, software development, or data architecture, in a cloud-based environment.
- Technical skills in data engineering, software development, and data architecture. This includes experience with big data technologies.
- Proficiency in one or more programming languages such as Python, Java, or Scala, as well as experience with scripting languages like Bash or PowerShell.
- Data management skills, data modelling, data governance, and data security. Analytical skills and experience with data analysis, data profiling, and data quality assessment.
- Project management methodologies, such as Agile or Scrum, and be able to manage projects independently.
More Info
Job Type:
Industry:
Function:
Employment Type:
About Company
Skills:
Pyspark, Kafka, Azure Databricks, Sql, ELT, Git, Azure Data Factory, Python, Azure DevOps, Etl, Microsoft Entra ID, CI CD, Unity Catalog, ADLS, Delta Lake
Skills:
Pyspark, Adf, Databricks, Python, Sql
Skills:
Azure Synapse Analytics, Azure Databricks, SQL Data Warehousing Concepts, Azure Data Lake Storage ADLS, ETL ELT Pipeline Development, Big Data Processing Fundamentals, Azure SQL Database, Python PySpark, Azure Data Factory ADF
Skills:
Python, Power Bi, Pyspark, PowerShell, Azure Databricks, Data Modeling, Bash, Perl scripting, Scrum Methodology, Sql, Agile Methodology, Azure Data Factory, Azure Data Lake, Gitlab, Devops Tools, Big data concepts, CI/CD pipelines, Data pipelines
Skills:
Etl, Sql, Data Warehousing, Python, Azure Data Factory, Azure Synapse, Cloud Architecture, ADLS Gen2