Software Engineer

Roles & Responsibilities

Responsibilities :

• Provide L3 support for all Apache Spark and Airflow related issues for the Data Platform.
• Design, develop, and maintain complex workflow, data pipelines, processes, and system architecture using Java, Python, PySpark, Apache Spark, Airflow and AWS technologies for efficient data processing and orchestration.
• Work closely with AWS services like S3, Glue, EMR, Redshift, and other related technologies to design and optimize data infrastructure.
• Implement data integration and transformation processes to ensure optimal performance and reliability of data pipelines.
• Optimize and fine-tune existing data pipelines / Airflow to improve efficiency, scalability, and maintainability.
• Troubleshoot and resolve issues related to data pipelines, ensuring smooth operation and minimal downtime.
• Develop and maintain documentation for data pipelines, processes, and system architecture.
• Stay updated with the latest industry trends and best practices related to data engineering and AWS services.

Requirements :

Tell employers what skills you have

PySpark

Airflow

Apache Spark

Scalability

Amazon AWS

Pipelines

Reliability

Staging

Data Engineering

EMR

SQL

Python

Java

AWS Lambda

S3

Data Warehousing

Software Development