Senior PySpark Data Engineer

We are seeking a highly skilled and experienced Senior PySpark Data Engineer to join our dynamic data engineering team. The ideal candidate will have a strong background in building and managing large-scale data processing systems and a proven track record of working with cutting-edge Big Data technologies.

Requirements

• Design, develop, and maintain robust, scalable, and high-performance data pipelines using PySpark.
• Develop, schedule, and monitor complex data workflows using orchestration tools like Apache Airflow.
• Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and deliver high-quality data solutions.
• Optimize and tune Spark jobs for performance and efficiency.
• Implement data quality checks and ensure data integrity across all data pipelines.
• Design and implement data models for optimal storage and retrieval.
• Mentor junior data engineers and promote best practices in data engineering.
• Ensure compliance with data governance and security policies.
• Troubleshoot and resolve data-related issues in a timely manner.

Benefits

• Equal opportunity employer