Principal Data Scientist – Agent Builder

Role Overview

The Search Conversational Experiences team builds Elastic’s new conversational and agentic platform that lets customers chat with their own data in Elasticsearch. As a Principal Data Scientist, you will help set the technical direction for how we evaluate, improve, and scale chat quality across Elastic’s agentic platform.

What You Will Do

Define the evaluation strategy for conversational and agentic search, lead the design of quality metrics and decision frameworks, build and compare improvements across retrieval and re-ranking approaches, and turn experimental results into product and business decisions.

Why It Might Be a Fit

8+ years of applied DS/ML experience, strong track record defining and leading evaluation for production AI/ML systems, hands-on ability with Python, PyTorch/Transformers, and experience influencing product and technical strategy through data.

Requirements

• 8+ years of applied DS/ML experience
• Deep expertise in IR, NLP, ranking, semantic search, RAG, or LLM-powered product experiences
• Strong track record defining and leading evaluation for production AI/ML systems
• Hands-on ability with Python, PyTorch/Transformers, Pandas, notebooks, reproducible experiments, versioned datasets, and clean, reviewable code
• Strong understanding of retrieval systems, including dense and sparse retrieval, re-ranking, vector search, query understanding, and evaluation metrics such as nDCG, MRR, Recall@k, precision, and latency/cost trade-offs
• Experience collaborating closely with engineering teams to move from prototype to production, including telemetry design, dashboards, CI guardrails, and quality regression tracking
• Practical Elasticsearch experience, or experience with similar search and distributed data systems
• Excellent written and verbal communication, with the ability to explain complex scientific and technical trade-offs to engineering, product, design, and leadership audiences
• A collaborative, low-ego style and a strong ability to mentor, raise standards, and develop transparency for others in a distributed team

Benefits

• Competitive pay based on the work you do here and not your previous salary
• Health coverage for you and your family in many locations
• Ability to craft your calendar with flexible locations and schedules for many roles
• Generous number of vacation days each year
• Increase your impact – We match up to $2000 (or local currency equivalent) for financial donations and service
• Up to 40 hours each year to use toward volunteer projects you love
• Embracing parenthood with minimum of 16 weeks of parental leave