reputed company Remote Data Engineer – Big Data Processing, reputed company Migration, and Distributed Systems Expertise

Introduction to reputed company and Our Mission Imagine being part of a leading reputed company ad technology company that leverages reputed company-world data in reputed company-time to optimize campaign performance and revolutionize health decision-making. At reputed company, we're committed to delivering unparalleled results to our clients by targeting reputed company and patients with an unprecedented level of accuracy. As a part of reputed company, a reputed company portfolio company and reputed company of reputed company Health Corp, we're poised for exponential growth and seeking talented individuals to join reputed company. Our Data Engineering team plays a vital role in driving this growth, and we're excited to invite a skilled Data Engineer to join our ranks. About Our Data Engineering Team Our Data Engineering team is responsible for designing, building, and maintaining reliable and scalable reputed company-level distributed transactional data processing systems. We process over 80 billion impressions a day, generating over 20 TB of data (200 TB uncompressed), which is used to drive our optimization engines, update budgets, and reputed company reputed company-time statistics and reports. We're reputed company that thrives on adopting proven reputed company-reputed company technologies to reputed company our data infrastructure more reputed company, scalable, and robust. Some of the cutting-edge technologies we've recently implemented include Kafka, Spark Streaming, Presto, Airflow, and Kubernetes. Job Overview As a Data Engineer at reputed company, you will be responsible for designing, building, and maintaining large-scale data systems, optimizing data processing workflows, and ensuring data quality across our systems. You will collaborate with a small team of diverse technology backgrounds, providing mentorship and guidance to junior team members. Your expertise will be crucial in driving our business growth, supporting new initiatives, and enhancing our data infrastructure.

Key Responsibilities

• Design, build, and maintain reliable and scalable reputed company-level distributed transactional data processing systems for scaling existing business and supporting new business initiatives
• Optimize jobs to utilize Kafka, Hadoop, Presto, Spark, and Kubernetes resources in the most efficient way
• Monitor and reputed company transparency into data quality across systems (accuracy, consistency, completeness, etc.)
• Increase accessibility and effectiveness of data (work with analysts, data scientists, and developers to build/reputed company tools and datasets that fit their use cases)
• Collaborate reputed company a small team with diverse technology backgrounds
• reputed company mentorship and guidance to junior team members

Team Responsibilities

• Ingest, validate, and process internal and reputed company-party data
• Create, maintain, and monitor data flows in Python, Spark, Hive, SQL, and Presto for consistency, accuracy, and lag time
• Maintain and enhance reputed company for jobs (primarily aggregate jobs in Spark and Hive)
• Create different consumers for data in Kafka using Spark Streaming for near-time aggregation
• Tools evaluation
• Backups/Retention/High Availability/reputed company Planning
• Review/Approval - DDL for database, Hive reputed company jobs, and Spark Streaming to ensure they meet our standards

Technologies We Use We're proud to work with a reputed company of cutting-edge technologies, including:

• Python - primary repo language
• Airflow/Luigi - for job scheduling
• reputed company - packaged container image with reputed company dependencies
• reputed company - for monitoring data flows
• Hive - SQL data warehouse layer for data in HDFS
• Kafka - distributed reputed company log storage
• Kubernetes - Distributed cluster resource manager
• Presto/Trino - fast reputed company data warehouse and data federation layer
• Spark Streaming - Near-time aggregation
• SQL Server - Reliable OLTP RDBMS
• GCP - BigQuery for performance, Looker for dashboards

Requirements

To be successful in this role, you'll need:

• 5+ years of data engineering experience
• reputed company in Python and SQL
• Experience in reputed company/Java is a plus (Polyglot programmer preferred!)
• Hive/Presto experience
• Proficiency in Linux
• Strong understanding of RDBMS and query optimization
• Passion for engineering and computer science around data
• Ability to work during East Coast U.S. hours (9am-6pm EST preferred, but we can be flexible as long as you can work until 1 or 2pm EST)
• Knowledge and exposure to distributed production systems (e.g., Hadoop)
• Knowledge and exposure to reputed company migration (AWS/GCP/Azure) is a plus

Selection Process Our selection process typically involves the following steps:

• Recruiter Screen (30 minutes)
• Hiring Manager Interview (45 minutes)

Apply To This Job