Lead Data Engineer - AURUMJIN CONSULTING PTE. LTD.

Roles & Responsibilities

• Design, develop, and implement Spark Scala applications and data processing pipelines to process large volumes of structured and unstructured data.
• Integrate Elasticsearch with Spark to enable efficient indexing, querying, and retrieval of data.
• Optimize and tune Spark jobs for performance and scalability, ensuring efficient data processing and indexing in Elasticsearch.
• Collaborate with data engineers, data scientists, and other stakeholders to understand requirements and translate them into technical specifications and solutions.
• Design and deploy data engineering solutions on OpenShift Container Platform (OCP) using containerization and orchestration techniques.
• Optimize data engineering workflows for containerized deployment and efficient resource utilization.
• Collaborate with DevOps teams to streamline deployment processes, implement CI / CD pipelines, and ensure platform stability.
• Monitor and optimize data pipeline performance, troubleshoot issues, and implement necessary enhancements.
• Implement monitoring and logging mechanisms to ensure the health, availability, and performance of the data infrastructure.

Tell employers what skills you have

Machine Learning

Scalability

Scala

Big Data

Pipelines

Hadoop

ETL

Data Quality

Data Engineering

Logging

SQL

Containerization

Java

Orchestration

Databases