Data engineer (Azure)

Roles & Responsibilities

Data engineer (Azure) – Synapse and Pyspark, Python, Datawarehouse and Azure Data Explorer, Azure Devops

Job Scope

• Design, review and development of Pyspark scripts. Testing, troubleshooting of data pipelines, orchestration
• Implement and maintain datalake
• Establishing connections to source data systems such as on-prem databases, IOT devices, APIs
• Managing the collected data in appropriate storage / data-base solutions e.g. file systems, SQL servers, Big Data platforms such as Hadoop, HANA, etc. as required by the specific project requirements.
• Design, development of relevant data pipelines using pyspark, copy data activities for batch ingestion
• Performing data integration e.g. using database table joins, or other mechanisms at an appropriate level as required by the analysis requirements of the project
• Deployment of pipeline artifacts from one environment to the other using Azure Devops
• Preferred to have Devops expertise

Skills & Experience

Good to have

Others

Tell employers what skills you have

PySpark

Factory

Azure

Big Data

Pipelines

Hadoop

ETL

Data Integration

Data Governance

Data Engineering

SQL

Python

Visualization

Metadata

Power BI

Databases