Principal Data Engineer

As a Principal Data Engineer, you will design, build, and maintain scalable data pipelines and infrastructure to support analytics, reporting, and data science initiatives.

Requirements

• Design and implement scalable data ingestion and transformation frameworks using Azure services
• Build and maintain robust ETL/ELT pipelines using Azure Data Factory and Azure Databricks
• Integrate data from diverse sources including on-premises systems, cloud storage, APIs, and streaming platforms
• Develop and optimize notebooks and workflows in Azure Databricks using PySpark, SQL
• Implement Delta Lake for efficient data storage, versioning, and ACID transactions
• Leverage Databricks features such as Unity Catalog and job orchestration
• Design and implement data models (star/snowflake schemas) for analytics and reporting
• Collaborate with architects to define data lakehouse architecture and best practices
• Implement data validation, profiling, and cleansing routines
• Monitor and optimize performance of Spark jobs and data pipelines
• Troubleshoot and resolve issues related to data latency, job failures, and resource utilization
• Implement role-based access control (RBAC), encryption, and secure data handling practices
• Maintain clear documentation of data flows, architecture, and operational procedures

Benefits

• 401k Matching
• Retirement Plan
• Generous Paid Time Off
• Health Insurance