Job Summary
A company is looking for a Data Engineer to join their dynamic team.
Key Responsibilities
- Develop and maintain scalable data pipelines and systems using Scala, Python, Spark, and Databricks
- Design and implement unified data models with enterprise-wide consistency
- Optimize operations in Spark for handling large, unstructured, and nested datasets
Qualifications
- Extensive experience as a Data Engineer, focusing on Python programming in Spark/Databricks
- Proven expertise in designing and optimizing large-scale data pipelines, ETL processes, and data lakes
- Strong background in cloud platforms, ideally Azure, with hands-on experience in managing cloud-based data infrastructure
- Demonstrated ability to convert legacy systems into modern Spark/Databricks platforms
- Experience in troubleshooting and resolving complex data issues
Comments