Azure DevOps and Data Engineer

Azure DevOps and Data Engineer
Fecha de publicación: 04 Enero 2023 BS Degree in Electrical Engineering, Computer Science or equivalent The Hands-On Data Engineer will be responsible for the build of the Enterprise Data platform. Setting up the data pipelines using Azure Data Factory ADF Gen2 that are parametrized, scalable, robust and resilient and build pipelines to validate, ingest, normalize and apply business-specific process on the healthcare data. Build Azure Data Lake leveraging Databricks technology to consolidate all data across the company and serve the data to various products and services. Will work with the Data Architect, product management, information security and infrastructure teams to deliver software and technology products that meet requirements and produce desired business outcomes. Will work with the Data Architect to do the necessary DevOps both manually on the Portal and using ARM Templates and Visual Studio.

Fecha de publicación: 04 Enero 2023

BS Degree in Electrical Engineering, Computer Science or equivalent

The Hands-On Data Engineer will be responsible for the build of the Enterprise Data platform.

Setting up the data pipelines using Azure Data Factory ADF Gen2 that are parametrized, scalable, robust and resilient and build pipelines to validate, ingest, normalize and apply business-specific process on the healthcare data.

Build Azure Data Lake leveraging Databricks technology to consolidate all data across the company and serve the data to various products and services.

Will work with the Data Architect, product management, information security and infrastructure teams to deliver software and technology products that meet requirements and produce desired business outcomes.

Will work with the Data Architect to do the necessary DevOps both manually on the Portal and using ARM Templates and Visual Studio.

Lugar:

Permanent remote from any city in Mexico

Skills:

MUST HAVE hands on experience and in-depth understanding of the HDFS, Spark, ETL and the Big Data concepts

Actividades:

Deseable:

Minimum 5+ years Hands-on experience in Azure

Minimum 3+ years Hands-on experience in Databricks

MUST TO HAVE hands-on experience with Azure Databricks, Azure Data Factory, ADLS, Notebook, either Jupyter or Zeppelin – Mandatory

Hands on experience in PySpark, Scala, SQL and Python.

MUST HAVE hands on experience and in-depth understanding of the HDFS, Spark, ETL and the Big Data concepts

Hands on experience with Visual Studio, ARM Template and Azure DevOps and deployment both via the portal and using

Hands on experience with Azure Data Factory (ADF), understanding the concept of parametrized ADF pipeline.

Hands on experience with the ADLS and understanding different zones and the concept of staging and merging.

Hands on experience with Delta Lake in Databricks, understanding of staging, curated and staging and merging using the Delta Lake.

Hands on experience with SQL Server being able to comfortably query and manipulate the data using SQL.

Understanding of the basics of the security in Azure using AAD, Roles and Permission, azure service principal key vault etc.

Experience with the Healthcare Industry is a plus

Experience with Azure Data Explorer (ADX) is a plus

Exposure to other cloud like GCP and AWS is a plus

Beneficios:

Payroll 100%

Christmas bonus 15 days

Vacation 12 days

Vacation bonus 50%

Career plan

Major medical expenses

Life insurance