Senior Data Engineer

Role Overview

The Senior Data Engineer will design, build, and optimize robust, scalable, and efficient ETL/ELT data pipelines using Python and PySpark. They will develop and manage processes for ingesting data from various sources and transform it into clean, usable formats for downstream consumption. The role involves implementing comprehensive unit and integration test coverage for data pipelines and establishing monitoring, alerting, and dashboarding solutions for data quality and pipeline health.

What You Will Do

Develop and optimize data pipelines using Python and PySpark, design and build robust ETL/ELT data pipelines, develop and manage processes for ingesting data from various sources, and implement comprehensive unit and integration test coverage for data pipelines.

Why It Might Be a Fit

The ideal candidate will have strong proficiency in Python and PySpark, experience with pipeline orchestration tools such as Airflow, and expertise in SQL for complex querying, data manipulation, and schema design. They will also have hands-on experience with Azure Databricks and proficiency with core Azure Analytics Services including Azure Data Factory, Azure SQL Server, and Azure Key Vault.

Requirements

• Strong proficiency in Python and PySpark for large-scale data processing and ETL development
• Experience with Pipeline Orchestration tools such as Airflow
• Expertise in SQL for complex querying, data manipulation, and schema design
• Demonstrable experience in designing, building, and maintaining robust ETL/ELT data pipelines
• Hands-on experience with Azure Databricks
• Proficiency with core Azure Analytics Services including Azure Data Factory, Azure SQL Server, and Azure Key Vault
• Experience implementing CI/CD pipelines from GitHub (including GitHub Actions) for automated testing (unit tests), build, and deployment processes
• Familiarity and practical experience with OpenShift (setup, deployment-ready configurations, and management)
• Experience with HELM for deploying applications on Kubernetes/OpenShift
• Experience in setting up and configuring Grafana for dashboards to monitor data quality and pipeline health

Benefits

• Competitive salary
• Equity
• Health insurance
• Paid time off
• Retirement plan
• Learning budget
• Parental leave
• Wellness programs
• Visa/relocation assistance
• Remote flexibility
• Stipends
• Bonus/commission