Skip to content
Data & AI

Data Scientist - Evaluations, Chanakya

Sarvam AI

  • Bengaluru · On-site
  • Full-time · Mid level

Check my ATS match for this job →

Source: Sarvam AI careers page. You apply on the employer's own website.

About this job

About Sarvam

Sarvam is building the bedrock of Sovereign AI for India. The company is developing India's full-stack sovereign AI platform, building across research, models, infrastructure and applications with a singular focus on making AI genuinely work for India. Sarvam works with leading enterprises and public institutions and is backed by Lightspeed, Peak XV, and Khosla Ventures. Sarvam partners with India's leading brands, including Tata Capital, SBI Life, CRED, IDFC, and LIC.

About the Role

This is an intellectually demanding role. The Data Scientist anchors the evaluations function for this vertical — designing, building, and maintaining evaluation frameworks that measure model and system quality in operational context. You are not running standard benchmarks. You are building domain-specific eval harnesses for high-stakes use cases where a wrong answer carries real consequences.

You will work closely with the MLOps Engineer, the PM, and the deployment team. The evaluations you design are the mechanism by which the team determines whether what we've built is good enough to deploy — and whether it stays good after deployment.

What You'll Do

• Design and build evaluation frameworks for Sarvam's AI outputs across domain-specific requirements: document comprehension, command summarisation, geospatial reasoning, enterprise workflow automation, and others as they emerge

• Define quality metrics in collaboration with domain experts and clients; translate operational requirements into measurable, defensible signals

• Run structured evaluation cycles pre- and post-deployment; build dashboards that surface model quality in production

• Identify failure modes, edge cases, and distribution shifts — with the bias of someone looking for what's wrong, not confirming what's right

• Collaborate with the MLOps Engineer to operationalise eval pipelines — automated, triggered by deployment events, versioned, and reproducible

• Build and manage domain-specific datasets for fine-tuning, evaluation, and benchmarking — including human annotation workflows where needed

• Publish internal findings and quality reports that feed the product and engineering roadmap

What We're Looking For

• 3–6 years in data science, ML research, or applied AI; at least 2 years working with LLMs in production contexts

• Strong statistics and probability fundamentals — you understand what makes an evaluation valid and what makes it misleading

• Experience designing evaluation frameworks from scratch: custom metrics, inter-rater reliability, red-teaming methodologies

• Python proficiency; comfort with pandas, NumPy, HuggingFace datasets, RAGAS, EleutherAI Eval Harness, LangSmith, or equivalent

• Experience with prompt engineering, model fine-tuning, or RLHF in applied settings

• Ability to work with unstructured domain data: PDFs, doctrine documents, transcripts, and field reports

Bonus Points

• Prior work in high-stakes domains (healthcare, legal, defence, finance) where output quality carries real-world consequences

• Red-teaming or adversarial evaluation experience

Note: We are looking for people who can own the outcomes described here, not people who

match every line of this specification. If this problem excites you and you believe you can do this work, we want to hear from you.

Why Sarvam?

Sarvam is a fast-moving, high talent-density team building full-stack AI for India, working on problems that push the frontiers of AI with real population-scale impact.

• Work alongside researchers, engineers, builders, and business leaders who move fast and hold each other to a very high bar

• High ownership and high impact, from day one

• Everything we do is AI-first, from the way we build and ship to the way we think about problems

• You can work on problems that could change how an entire country learns, works, and communicates

If you want to work on problems at the frontier of AI in India, Sarvam is the place to be.

Skills mentioned

  • Python
  • Pandas
  • NumPy

Have these skills? Make sure they appear on your resume in the same words.

Similar jobs

See all jobs →
New On-site Mid level

Data Engineering & Agentic AI

Bosch

  • Location:Bengaluru
  • Experience:8+ yrs
  • Employment type:Full-time
  • Salary not disclosed

Job Description Key Responsibilities:Design and develop scalable and reliable data pipelines and data processing solutions.Read data sets from enterprise-wide applications and data lakes, deploy them into data warehouse, which can be accessed via spreadsheets, Power BI or other analytical platfor...

  • Power BI
  • Python
  • ETL
  • SQL
  • MySQL
  • Jenkins
  • +2 more

Bosch careers page

Apply now
New On-site Lead / Manager

ServiceNow Architect - AI

Bosch

  • Location:Bengaluru
  • Experience:12–15 yrs
  • Employment type:Full-time
  • Salary not disclosed

Job Description Roles & Responsibilities : Job Description Own the target architecture, technical standards and cross-domain design decisions for a scalable ServiceNow platform. Provide architectural direction for Workflow Studio, IntegrationHub, App Engine, portals, AI capabilities and enterpris...

Bosch careers page

Apply now
On-site Mid level

Research Engineer – Neuro-Symbolic AI (Knowledge Graphs) & Multimodal Assistant Systems

Bosch

  • Location:Bengaluru
  • Experience:3+ yrs
  • Employment type:Full-time
  • Salary not disclosed

Job Description Roles & Responsibilities : Conduct cutting-edge research in Neuro-Symbolic AI, focusing on the integration of formal knowledge representations such as knowledge graphs with modern machine learning and deep learning techniques. Design multimodal data pipelines integrating text, str...

  • Machine Learning
  • Deep Learning
  • NLP
  • PyTorch

Bosch careers page

Apply now
Hybrid Mid level

AI & Backend Software Engineer – Intellectual Property Rights(Python / FastAPI / GenAI)

Bosch

  • Location:Bengaluru
  • Experience:3+ yrs
  • Employment type:Full-time
  • Salary not disclosed

Job Description Roles & Responsibilities : We are looking for a hands-on software engineer with intrinsic motivation to build and run Generative AI workflows for Bosch's global Intellectual Property (IP) organization. You will own workflows from start to finish. You will work directly with IP att...

  • Python
  • FastAPI
  • Generative AI
  • LLM
  • Flask
  • Django
  • +8 more

Bosch careers page

Apply now
On-site Senior

Senior Power BI Developer & Consultant

Bosch

  • Location:Bengaluru
  • Experience:8–10 yrs
  • Employment type:Full-time
  • Salary not disclosed

Job Description Shape the future of data-driven reporting and decision-making We are looking for a Power BI Reporting Developer & Consultant who combines strong technical expertise with a business-oriented mindset. In this role, you will design and develop professional reporting solutions while w...

  • Power BI
  • Excel
  • SQL
  • Data Visualization

Bosch careers page

Apply now
On-site Mid level

Power BI Developer

Bosch

  • Location:Bengaluru
  • Experience:3–5 yrs
  • Employment type:Full-time
  • Salary not disclosed

Job Description Roles & Responsibilities : Shape the future of data-driven reporting and decision-making We are looking for a Power BI Reporting Developer & Consultant who combines strong technical expertise with a business-oriented mindset. In this role, you will design and develop professional...

  • Power BI
  • Excel
  • SQL
  • Data Visualization

Bosch careers page

Apply now