Senior Data Engineer

Abacus Insights is seeking a Senior Data Engineer to join its dynamic Tech Ops division. The successful candidate will work with customers, data vendors, and internal engineering teams to design, implement, and optimize complex data integration solutions in a modern cloud environment.

Requirements

• Bachelor’s degree in Computer Science, Computer Engineering, or a closely related technical field.
• 6 years of hands‐on experience as a Data Engineer working with large‐scale, distributed data processing systems in modern cloud environments.
• Strong ability to communicate complex technical concepts clearly across both technical and non‐technical stakeholders.
• Expert‐level proficiency in Python, SQL, and PySpark, including developing distributed data transformations and performance‐optimized queries.
• Demonstrated experience designing, building, and operating production‐grade ETL/ELT pipelines using Databricks, Airflow, or similar orchestration and workflow automation tools.
• Proven experience architecting or operating large‐scale data platforms using dbt, Kafka, Delta Lake, and event‐driven/streaming architectures, within a cloud‐native data services or platform engineering environment—requiring specialized knowledge of distributed systems, scalable data pipelines, and cloud‐scale data processing.
• Experience working with structured and semi‐structured data formats such as Parquet, ORC, JSON, and Avro, including schema evolution and optimization techniques.
• Strong working knowledge of AWS data ecosystem components—including S3, SQS, Lambda, Glue, IAM—or equivalent cloud technologies supporting high‐volume data engineering workloads.
• Proficiency with Terraform, infrastructure‐as‐code methodologies, and modern CI/CD pipelines (e.g., GitLab) supporting automated deployment and versioning of data systems.
• Deep expertise in SQL and compute optimization strategies, including Z‐Ordering, clustering, partitioning, pruning, and caching for large‐scale analytical and operational workloads.
• Hands‐on experience with major cloud data warehouse platforms such as Snowflake (preferred), BigQuery, or Redshift, including performance tuning and data modeling for analytical environments.

Benefits

• Competitive Leave & Benefits
• Comprehensive health coverage
• Equity for every employee – share in our success
• Growth-focused environment – your development matters here