Data Scientist

Role Overview

This role supports data integration, pipeline development, data modeling, governance, and documentation while collaborating with cross-functional teams to deliver high-quality, reliable data products for the agency.

What You Will Do

Designs, builds, and maintains secure, scalable data solutions across Azure Databricks and Azure Synapse Analytics. Plays a key role in implementing the agency’s data strategy by developing batch and streaming pipelines, optimizing data warehouse workloads, and enforcing data governance standards.

Why It Might Be a Fit

The position assists with requirements gathering, solution reviews, and technical support, and works closely with data architects, scientists, analysts, and project managers to translate business needs into technical designs.

Requirements

• Master’s degree in data science, analytics, informatics, computer science, mathematics, engineering, or related field
• Developing with Databricks notebooks in PySpark, Spark SQL, and R, including transformations, joins, aggregations, and window functions
• ACID transactions, MERGE upserts, schema enforcement/evolution, time travel, and table maintenance (OPTIMIZE, Z-order, VACUUM)
• Working with medallion (bronze/silver/gold) architecture and incremental ingestion via Auto Loader, Structured Streaming, or Lakeflow/DLT
• Using Unity Catalog for Workspace Catalogs, schemas, permissions, lineage, external locations, and storage credentials
• Right-sizing clusters and SQL warehouses and diagnose partitioning, shuffle, skew, and caching issues via Spark UI
• Scheduling and managing dependencies with ADF Pipelines, Databricks Workflows (Jobs), and using Git folders (Repos) for source control
• Designing MPP warehouse tables, including distribution (hash, round-robin, replicated), columnstore indexes, and partitioning
• Querying Parquet, Delta, and CSV files in ADLS Gen2 using OPENROWSET, external tables, and views
• Building star-schema facts, dimensions, stored procedures, and views, including SCD Type 1 and Type 2
• Developing with COPY INTO, PolyBase, and CETAS, and building parameterized Synapse/ADF pipelines (copy, data flows, triggers)
• Workload management, resource classes, statistics, materialized views, result-set caching, and pool scaling/pausing
• ADLS Gen2 folder/file-format design (Parquet/Delta), Entra ID access control, managed identities, and Key Vault
• Git and CI/CD in Azure DevOps or GitHub, including branching, PRs, and deploying Synapse artifacts and Databricks Asset Bundles
• Monitoring pipelines and workloads with Azure Monitor/Log Analytics and alerting, and troubleshooting failed jobs
• Strong written and verbal communication skills
• Using Microsoft lakehouse patterns, including interoperability and migration paths between Synapse and Databricks

Benefits

• Dental insurance
• Vision insurance
• Health insurance
• Paid time off
• Retirement plan
• Learning budget
• Parental leave
• Wellness program
• Visa sponsorship
• Relocation assistance