Senior Data Engineer

Roles & Responsibilities

Job Responsibilities :

• Work across workstreams to support data requirements including reports and dashboards
• Analyze and perform data profiling to understand data patterns and discrepancies following Data Quality and Data Management processes
• Understand and follow best practices to design and develop the E2E Data Pipeline : data transformation, ingestion, processing, and surfacing of data for large-scale applications
• Develop data pipeline automation using Azure, AWS data platform and technologies stack, Databricks, Data Factory
• Understand business requirements to translate them into technical requirements that the system analysts and other technical team members can drive into the project design and delivery
• Analyze source data and perform data ingestion in both batch and real-time patterns via various methods; for example, file transfer, API, Data Streaming using Kafka and Spark Streaming
• Analyze and understand data processing and standardization requirements, develop ETL using Spark processing to transform data
• Understand data / reports and dashboards requirements, develop data export, data API, or data visualization using Power BI, Tableau, or other visualization tools

Qualification :

We are looking for experience and qualifications in the following :

Mandatory Skills : PySpark, Databricks

Tell employers what skills you have

Tableau

PySpark

Apache Spark

Data Analysis

Azure

Big Data

AWS

Data Transformation

Data Management

Azure Data Factory

Azure Machine Learning

ETL

Data Quality

Databricks

Data Engineering

SQL

Python

Apache Kafka

Power BI

Data Visualization