Senior Data Engineer – Azure Databricks

Blend is a premier AI services provider looking for an experienced Senior Azure Databricks Engineer with strong hands-on expertise in Python, SQL, and Apache Spark to design, build, and optimize scalable data pipelines and analytics solutions on the Azure cloud platform.

Requirements

• Design, develop, and maintain scalable data pipelines using Azure Databricks
• Implement ETL/ELT workflows using PySpark, Spark SQL, and Python
• Implement pipelines for data ingestion using Azure Data Factory
• Optimize Spark jobs for performance, cost, and scalability
• Work with structured and semi-structured data (Parquet, Delta, JSON, CSV)
• Build and manage Delta Lake tables (ACID, time travel, schema evolution)
• Integrate Databricks with Azure Data Lake Storage (ADLS Gen2)
• Develop complex queries and transformations using SQL
• Collaborate with Data Science teams to prepare data for modelling use cases
• Follow best practices for security, access control, and governance in Azure
• Ensure data quality, validation, and monitoring using testing tools
• Deployment of solutions to Production environments

Tagged as: Python, SQL