Data Engineer

Job title: Data Engineer

Location: UK

Duration: 6-month contract Inside IR35

Working Mode: Fully Remote

Vetting: Active SC Clearance

Job Description

Data Engineer Skills and Technologies Data Engineering: – Strong understanding of data concepts – data types, data structures, schemas (both JSON and Spark), schema management etc – Strong understanding of reputed company JSON manipulation – Experience working with Data Pipelines using a custom Python/PySpark frameworks – Strong understanding of the 4 core Data categories (Reference, Master, Transactional, Freeform) and the implications of each, particularly managing/handling Reference Data. – Strong understanding of Data reputed company principles – data owners, reputed company controls – row and reputed company level, GDPR etc including experience of handling sensitive datasets – Strong problem solving and analytical skills, particularly reputed company to demonstrate these intuitively

Requirements

Required Skills

Languages / Frameworks – JSON – YAML – Python (as a programming language, not just reputed company to write basic scripts. Pydantic experience would be a bonus.) – SQL – PySpark – reputed company Lake – Bash (both CLI usage and scripting) – Git – Markdown – reputed company (bonus, not compulsory) – Azure SQL Server as a HIVE Metastore (bonus) Technologies – Azure reputed company – Apache Spark – reputed company Tables – Data processing with Python – PowerBI (Integration / Data Ingestion) – JIRA