The Data Engineer plays a critical role in designing, building, and maintaining high-quality data pipelines on the Databricks platform. Success in the role requires writing efficient code, primarily in Python and SQL, while leveraging tools such as PySpark and Delta Lake.
Requirements
- Proficiency in data engineering principles, including the development and maintenance of data pipelines.
- Advanced coding skills in Python, SQL, and Scala, with significant experience working with Apache Spark.
- Hands-on experience with the Databricks platform, particularly with Delta Lake, Databricks Runtime, and Databricks Workflows.
- Familiarity with the Azure Cloud platform.
- Knowledge of the Gold Medallion architecture.
- Experience in data ingestion, transformation, and loading processes (ETL/ELT).
- Excellent communication skills, with the ability to explain complex data concepts to both technical and non-technical audiences.
- Strong problem-solving and analytical abilities.
- Experience with Mulesoft API platform is considered an asset.
- Background in creating ingestion pipelines from a variety of systems, such as HRIS, ERP, CRM, Microsoft SQL Server, and Apache Kafka.
- Experience with machine learning and data analytics.
- Knowledge of data governance and security best practices.
- Databricks certifications an asset
- Knowledge or experience in developing and integrating custom machine learning models using Azure Machine Learning, MLflow, and other relevant libraries
Benefits
- Paid time off
- 401(k) matching
- Retirement plan
- Health insurance
To apply for this job please visit jobs.atsautomation.com.
Follow us on social media