Lead Data Engineer

Lead Data Engineer responsibilities include architecting and governing scalable batch and streaming data platforms, designing end-to-end reference architectures, and leading data governance architecture. The role requires strong proficiency in Python, Apache Spark, and Azure Databricks, as well as experience with streaming platforms, microservices, and cloud-native architecture principles. The Lead Data Engineer will also act as Technical Product Owner, owning one or more data domains or products, and partnering with business stakeholders to translate business objectives into architecture-aligned epics, user stories, and acceptance criteria.

Requirements

• Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field
• 8–12 years of experience in Data Engineering, with demonstrated architecture ownership on Azure-based platforms
• Strong proficiency in Python, with solid OOP, design patterns, and system design principles
• Deep expertise in Apache Spark (PySpark, Spark SQL) and Azure Databricks
• Strong hands-on and architectural experience with streaming platforms
• Proven experience designing microservices and event-driven architectures
• Strong experience deploying and operating workloads on Azure Kubernetes Service (AKS)
• Deep understanding of Delta Lake and large-scale lakehouse architectures
• Advanced SQL skills for analytics and optimization
• Strong experience with Azure ADLS Gen2, Databricks, Azure Functions, Service Bus, Key Vault
• Strong experience with Git, GitLab CI/CD, and release management
• Experience with Terraform and Pulumi for enterprise-grade IaC
• Strong knowledge of data modeling, distributed systems, and fault tolerance

Benefits

• Generous Paid Time Off
• 401k Matching
• Retirement Plan