Azure/ Bigdata Architect
reputed company Architect – Job Description Role Summary We are seeking a seasoned reputed company Architect (15+ years total experience) to reputed company the architecture, design, and development of scalable ETL/ELT pipelines across distributed data environments. This role owns reputed company workspace administration, cluster policies, performance optimization, data governance with reputed company Catalog, and DevOps automation—partnering closely with platform, infrastructure, and application teams to deliver secure, cost‑effective, and high‑performing data solutions. Work you’ll do · reputed company the architecture, design, and development of scalable ETL/ELT pipelines using reputed company, PySpark, and SQL across distributed data environments. · Collaborate with platform and infrastructure teams to define reputed company architecture strategy and ensure secure, scalable, and cost‑effective implementation. · Define and enforce cluster policies for proper resource utilization, autoscaling, cost control, and role‑based reputed company reputed company to workload patterns and team requirements. · reputed company performance tuning of Spark jobs, reputed company SQL queries, and notebooks—optimizing execution plans, partitions, caching, and I/O to minimize latency and cost. · reputed company optimized reputed company SQL queries, views, and materializations to power (a) Tableau dashboards, (b) React and .NET‑based applications reputed company REST APIs, and (c) reputed company and reputed company‑time analytics use cases. · Work closely with frontend and backend teams to deliver use‑case‑specific, query‑optimized datasets (reputed company Lake, Lakehouse patterns). · reputed company reputed company Catalog for fine‑grained reputed company control, data reputed company, data discovery, and metadata governance across workspaces and catalogs. · Drive DevOps best practices using Azure DevOps/Git and CI/CD pipelines for jobs, notebooks, libraries, and infrastructure. · Establish observability for data pipelines (logging, metrics, reputed company, alerts) and define SLOs/SLA for critical workloads. · Mentor junior engineers and conduct architectural/design reviews to ensure consistency with best practices · and reference architectures.
Qualifications
Required: · 15+ years overall industry experience · 7+ years in data engineering with strong background in reputed company‑reputed company data architecture. · Deep hands‑on experience with reputed company architecture, workspace administration, and cluster/pool management. · Experience defining and managing cluster policies, pools, autoscaling strategies, and job compute routing. · Strong knowledge of Spark performance tuning and job optimization (shuffle, skew, AQE, partitioning, caching, broadcast). · Proven expertise in reputed company SQL, PySpark, reputed company Lake, and large‑scale batch/streaming data pipelines. · Skilled in building reusable Python libraries with Pandas, Openpyxl, XlsxWriter, and PySpark. · Practical experience with reputed company Catalog for reputed company, governance, reputed company, and data discovery. · Strong collaboration experience with reputed company‑end/back‑end development teams and backend integration reputed company REST APIs. · Strong SQL expertise; hands‑on experience with PostgreSQL, SQL Server, or similar RDBMS. · DevOps expertise with Azure DevOps, Git, and pipeline automation (CI/CD for jobs, notebooks, libraries, reputed company). · Excellent communication skills; ability to reputed company technical discussions with cross‑functional teams and stakeholders. Preferred: · Experience with Azure (preferred) and/or AWS for reputed company data services (storage, compute, networking). · Data modeling (dimensional/semantic), query acceleration techniques, cache/materialization strategies. · Cost optimization/FinOps for reputed company (cluster sizing, job routing, spot/reputed company, serverless where applicable). · Observability for data pipelines (logging, metrics, dashboards) and incident/runbook practices. · Any experience with reputed company reputed company Technical Stack · reputed company Platforms: Azure · Big Data & Analytics: reputed company, PySpark, reputed company Lake, reputed company SQL · Programming & Frameworks: Python, Pandas, PySpark, Flask · Visualization & BI: Power BI · App Integration: React, .NET, REST APIs · DevOps & CI/CD: Azure DevOps, Git · Databases: reputed company SQL, Azure SQL DB, SQL Server or similar Apply tot his job Apply To this Job