Observability Engineer

Observability Engineer - ( 24000139 )

Description

The engineer is responsible for designing, implementing, and maintaining Elasticsearch clusters and related components in enterprise distributed cluster environment. His role involves working with large-scale data sets and optimizing search and indexing performance.

Responsibilities

1. Cluster Design and Deployment: Designing and deploying Elasticsearch clusters based on specific requirements, including hardware, network, and security considerations.
2. Configuration and Optimization: Configuring Elasticsearch settings, including index mappings, shard allocation, and cluster settings, to optimize performance and ensure efficient resource utilization.
3. Data Ingestion and Indexing: Developing strategies for efficient data ingestion into Elasticsearch, including handling large data volumes, implementing data pipelines, and optimizing indexing processes.
4. Search and Query Optimization: Tuning search queries, relevance scoring, and aggregations to improve search performance and response times.
5. Monitoring and Troubleshooting: Implementing monitoring solutions to track cluster health, performance, and resource utilization. Identifying and resolving issues related to indexing, search, and cluster stability.
6. Scalability and High Availability: Planning and implementing strategies for scaling Elasticsearch clusters horizontally or vertically to handle increased data volumes and user traffic. Ensuring high availability and fault tolerance through replication and data redundancy.
7. Security and Access Control: Implementing security measures, such as authentication, authorization, and encryption, to protect Elasticsearch clusters and data from unauthorized access.
8. Collaboration and Documentation: Collaborating with cross-functional teams, including developers, DevOps, and data engineers, to integrate Elasticsearch into applications and systems. Documenting cluster configurations, deployment processes, and troubleshooting guidelines.
9. Performance Analysis and Optimization: Conducting performance analysis, identifying bottlenecks, and implementing optimizations to improve overall system performance and efficiency.
10. Stay updated with Elasticsearch: Keeping up-to-date with the latest features, enhancements, and best practices in Elasticsearch and related technologies.
11. Collaborate and work with the various stakeholder to develop technical solutions, plans and configuration for the observability capability and develop run book automation with the objective to achieve self-healing capability.
12. Perform integration and develop full interoperability capabilities with various operations management platform including change management, service management, privileged access management system, etc.
13. Provide support to existing monitoring system and bridge the transition to an AI-enabled observability platform.

The engineer plays a crucial role in ensuring the smooth operation and performance of Elasticsearch clusters, enabling efficient search and analysis capabilities for organizations dealing with large-scale data sets.

Qualifications

Requirements

• Possesses at least a Bachelor's Degree in Computer Science/Information Technology or equivalent.
• Solid experience on managing enterprise distributed cluster environment.
• At least 5 years of hands-on working experience in:
• Enterprise logging platform (such as ElasticSearch, Splunk)
• Automation scripting (such as Ansible, Terraform, Powershell, etc)
• Demonstrate strong knowledge in both application and infrastructure domain with ability to develop automation scripts.
• Possess good technical knowledge in implementing, troubleshoot, performance tuning of hardware, operating system and system services.
• Strong analytical skill, creative and thinking out of the box in problem solving.
• Excellent command of written and spoken English.
• A good team player and able to work effectively at all levels of an organization with the ability to influence others to move towards consensus.
• Proven ability to operate under pressure and meet challenging deadlines with minimum supervision.

Primary Location

: Singapore

Job

: Technology Solutions

Organization

: Group Operations & Technology Division

Schedule

: Permanent

: Full-time

Job Posting

: 08-Sep-2024, 3:49:52 AM