Role Overview
Join a company leading the way in AI and automation consulting, working on data pipelines, data warehousing, and data integration for various analytics and AI use cases. Collaborate closely with Delivery Managers, ML Engineers, and other stakeholders to ensure seamless data flow and accessibility.
What You Will Do
Build and maintain scalable batch and streaming pipelines (ETL/ELT) for analytics and AI, ingest data from OLTP databases, files, SaaS apps, and APIs into the lakehouse, prepare data for ML and GenAI use cases with ML Engineers, and ensure data quality, security, and governance.
Why It Might Be a Fit
You need to be a quick learner, get excited about innovation, and be on the constant lookout to master new technologies as they come up in the Data, AI & Cloud realms. You’ll contribute to technical design, mentor junior team members, and apply best practices such as version control, code review, testing, CI/CD, and documentation.
Requirements
- Degree in Computer Science, Engineering, or related field, or equivalent experience
- 3–5 years building production data pipelines
- Strong SQL and Python (including PySpark)
- Distributed processing: Partitioning, joins, and performance tuning (e.g., Apache Spark on Databricks)
- Lakehouse architecture: Medallion architecture and open table formats (e.g., Delta Lake, Apache Iceberg)
- Declarative pipelines and orchestration: e.g., Lakeflow Spark Declarative Pipelines, Lakeflow Jobs, Airflow, dbt
- Data ingestion: Batch, incremental, CDC, and API ingestion (e.g., Lakeflow Connect, Auto Loader, Fivetran, Azure Data Factory, AWS Glue)
- Batch and streaming: Streaming concepts such as watermarking and late data (e.g., Spark Structured Streaming)
- OLTP and OLAP: How transactional and analytical systems differ and integrate (e.g., PostgreSQL, SQL Server, Databricks Lakebase)
- Data modeling: Relational and dimensional modeling, including star schemas and SCDs
- Data warehousing: e.g., Databricks SQL, Snowflake, BigQuery, Redshift
- Data quality: Validation, monitoring, and alerting (e.g., pipeline expectations, Lakehouse Monitoring)
- Governance and security: Access control (RBAC/ABAC), PII masking, and lineage (e.g., Unity Catalog)
- Cloud platforms: At least one of AWS, Azure, or GCP
- DevOps and CI/CD: Git, automated testing, CI/CD, and infrastructure as code (e.g., Databricks Asset Bundles, Terraform)
- Soft skills: Strong problem-solving, communication, and collaboration
To apply for this job please visit bitsinglassca.bamboohr.com.


Follow us on social media