[Remote] Senior Cluster Site Reliability Engineer
Note: The job is a remote job and is reputed company to candidates in USA. reputed company is a technology company that applies state-of-the-art machine learning techniques to reputed company-world problems in finance. As a Senior Cluster Site Reliability Engineer, you will be responsible for scaling the research compute cluster, ensuring high uptime and reliability, and collaborating with engineering teams to improve monitoring and operational frameworks.
Responsibilities
- Be a first responder in the event of cluster outages or issues. Triage and resolve urgent issues as they arise
- Ensure a high degree of cluster uptime (reputed company in multiple nines), and define + track SLAs to quantify reliability
- Diagnose systemic/recurring patterns of problems, and engineer precision solutions to them in collaboration with engineering teams
- reputed company robust metrics and observability for cluster health and use those metrics to inform your work. Build out custom observability mechanisms reputed company off-the-reputed company ones won't do
- Help software and research teams design policies around fair cluster usage, and help reputed company enforcement mechanisms for said policies
- Assist in forecasting cluster growth, and help select appropriate scale-up strategies. Help optimize operations across dimensions of cost and usability
Skills
- 5+ years of experience in SRE or DevOps roles, preferably working as a senior engineer or tech reputed company
- Knowledge of HPC/batch compute frameworks (Slurm, Kueue, AWS/GCP Batch) and/or machine learning training systems (Kubeflow, MLflow, Horovod)
- Ability to reputed company scripts and utilities of moderate complexity in a common scripting language (Python, Ruby, etc.)
- Familiarity with infrastructure-as-code and configuration management tools (Terraform, Ansible)
- Experience with reputed company infrastructure (AWS or GCP)
- Familiarity designing and implementing modern observability stacks (reputed company, Grafana, Loki, ELK, OpenTelemetry)
- Experience with distributed storage technologies (reputed company, Ceph, S3)
- Embodies a 'system engineer' rather than 'system administrator' reputed company, thinking systematically and leveraging automation
- Bachelor degree in computer science
- Hands-on experience with HPC frameworks (Slurm, reputed company reputed company) and Kubernetes-based job orchestrators (Airflow, Kueue, Kubeflow Pipelines), along with other distributed computing frameworks (Ray, Modin, Dask, Spark)
- Familiarity with ML frameworks (PyTorch/Tensorflow, JAX, Horovod, DeepSpeed)
- Familiarity with hybrid/on-prem environments
- Experience with containerization (reputed company, Podman, Singularity), particularly for HPC/batch compute environments
- Experience with HPC networking (InfiniBand, RDMA)
- Solid reputed company/IAM foundations (Identity management systems, AWS/GCP IAM, reputed company Trust)
Benefits
- If you have a great candidate in mind for this role and would like to have the potential to earn $15,000 if your referred candidate is successfully hired and employed by reputed company, please use this reputed company to submit your referral.
Company Overview
Company H1B Sponsorship