We are a diverse and focused team motivated to solve the hardest problems in the automotive industry through machine learning. As an early data engineer, your work is critical to our success, ensuring data quality and reliability in every part of our ETL pipeline.
Requirements
- 3+ years as a data engineer
- Experience as a tech lead or mentor
- Proficiency in Python/Scala/Java/C++ and SQL
- 3+ years of experience with Spark or equivalent technologies
- 2+ years of experience with a workflow scheduler (Airflow, Prefect, Argo, etc)
- 2+ years of experience with distributed file-systems (HDFS, S3, etc)
- Familiar with the tools in open-source data ecosystem (Apache, CNCF, etc)
- Experience with incremental or real-time processing (Delta Lake, Apache Hudi, Kafka Stream, Spark Streaming, etc)
- Bonus: Experience with Kubernetes, Experience working with ML teams, Contributor to open source projects, Experience in the Automotive industry or a love of cars, Prior work in small, agile teams
Benefits
- 401k Matching
- Tuition Reimbursement
- Relocation Assistance
To apply for this job please visit www.viaduct.ai.
Follow us on social media