Lead Data Engineer

Lead Data Engineer responsible for architecting, designing, and governing scalable batch and streaming data platforms using Python, Apache Spark, and Azure Databricks. Owns data governance architecture, defines standards for security, access control, lineage, and metadata. Provides technical leadership to data engineers and cross-functional teams.

Requirements

• Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field
• 8–12 years of experience in Data Engineering, with demonstrated architecture ownership on Azure-based platforms
• Strong proficiency in Python, with solid OOP, design patterns, and system design principles
• Deep expertise in Apache Spark (PySpark, Spark SQL) and Azure Databricks
• Strong hands-on and architectural experience with streaming platforms: Apache Kafka OR Apache Flink, Spark Structured Streaming
• Proven experience designing microservices and event-driven architectures
• Strong experience deploying and operating workloads on Azure Kubernetes Service (AKS)
• Deep understanding of Delta Lake and large-scale lakehouse architectures
• Advanced SQL skills for analytics and optimization
• Strong experience with Azure ADLS Gen2, Databricks, Azure Functions, Service Bus, Key Vault
• Strong experience with Git, GitLab CI/CD, and release management
• Experience with Terraform and Pulumi for enterprise-grade IaC
• Strong knowledge of data modeling, distributed systems, and fault tolerance