Skip to content
Other

SDE2 - API Platform

Sarvam AI

  • Bengaluru · On-site
  • Full-time · Mid level

Check my ATS match for this job →

Source: Sarvam AI careers page. You apply on the employer's own website.

About this job

About the Role

Sarvam's model APIs (LLM, ASR, TTS, Vision, Translate) are how developers and enterprises ship products on our foundation models. Our OpenAI-compatible LLM APIs route every request across multiple model deployments, with streaming, failover, metering, billing, and rate-limiting all part of the same stack. Today it's FastAPI and Python; we're in the middle of rewriting it in Go.

We're hiring a Software Engineer (SDE 2) to own core parts of this platform. The features. The reliability. The performance. This is an IC role.


What You'll Do

  • Building and evolving the LLM API platform, from the moment a request hits the edge to the response that goes back out

  • The Python→Go rewrite: building services in Go alongside the existing Python stack, without compromising reliability

  • Streaming infrastructure: SSE for Chat Completions and Responses, long-lived connections, clean handling of cancellations and client disconnects

  • Routing and failover: weighted provider chains, retries with backoff, and latency-driven admission control across model deployments

  • Protocol compatibility: keeping the API faithful to the OpenAI spec across inference servers (vLLM, SGLang, Dynamo) that each behave slightly differently

  • The commercial layer: metering, billing, and rate-limiting, engineered to the same bar as the model APIs themselves

  • Performance and reliability for a multi-tenant platform serving long-context agentic and coding workloads

  • Observability (logging, metrics, tracing), so the team can debug production confidently and quickly

  • Integration and load testing: the harness that lets us ship fast without breaking customer-facing APIs

  • Close partnership with the Inference and MLOps teams on production readiness for new model launches

  • Design-first, documentation-first engineering culture within the team: RFCs before code, decisions written down, no tribal knowledge


What We're Looking For

  • 3+ years building production backend systems

  • Strong Python in production: asyncio, FastAPI or similar, and services you've built and operated under real load

  • Working knowledge of Go, or a strong willingness to learn it: you'll work in both languages through the rewrite

  • Experience building low-latency, multi-tenant backend services

  • Hands-on experience with streaming systems: SSE or WebSockets, long-lived connections, backpressure

  • Strong PostgreSQL and Redis fundamentals: schema design, query performance, caching, and rate limiting

  • Comfort running services on Kubernetes in production

  • An ownership mindset: you test before you ship, debug production with metrics, logs, and traces, and write the RCA afterwards


Bonus Points

  • Experience serving or integrating LLMs in production, especially OpenAI-compatible APIs

  • Familiarity with inference servers (vLLM, SGLang, Dynamo) and latency metrics such as time to first token and inter-token latency

  • Experience building metering, billing, or usage-based pricing infrastructure

  • Time at an early- or growth-stage startup: comfort with ambiguity, speed, and ownership


Why Sarvam?

Sarvam is a fast-moving, high talent-density team building full-stack AI for India, working on problems that push the frontiers of AI with real population-scale impact.

  • Work alongside researchers, engineers, builders, and business leaders who move fast and hold each other to a very high bar

  • High ownership and high impact, from day one

  • Everything we do is AI-first, from the way we build and ship to the way we think about problems

  • You can work on problems that could change how an entire country learns, works, and communicates

If you want to work on problems at the frontier of AI in India, Sarvam is the place to be.

Skills mentioned

  • LLM
  • FastAPI
  • Python
  • PostgreSQL
  • Redis
  • Kubernetes

Have these skills? Make sure they appear on your resume in the same words.

Similar jobs

See all jobs →
New On-site Mid level

Executive Secretary - PS/RP-IN

Bosch

  • Location:Bengaluru
  • Experience:5+ yrs
  • Employment type:Full-time
  • Salary not disclosed

Job Description Executive Calendar, Meetings & Stakeholder Management - Manage calendars, meetings, workshops and related logistics; coordinate agendas and communication with internal/external stakeholders while maintaining confidentiality Travel & Logistics Management - Coordinate domestic/inter...

  • Stakeholder Management
  • SAP

Bosch careers page

Apply now
New On-site Mid level

House Manager

Zeta

  • Location:Bengaluru
  • Experience:5+ yrs
  • Employment type:Contract
  • Salary not disclosed

About Zeta Build the future of banking. Zeta is a next-generation banking technology company providing cloud-native, fully stackable processing and core banking platforms for issuers. With a focus on scalability, compliance, and innovation, Zeta empowers financial institutions to modernize their...

Zeta careers page

Apply now
New On-site Director & above

Associate Director, Risk Management

Razorpay

  • Location:Bengaluru
  • Experience:10–12 yrs
  • Employment type:Full-time
  • Salary not disclosed

Razorpay is one of India’s leading full-stack financial technology companies, powering the way businesses move, manage, and grow money. Founded in 2014 by Harshil Mathur and Shashank Kumar with a simple vision - to simplify payments for Indian businesses - we’ve since grown into a fintech powerho...

  • Payroll
  • Salesforce
  • Onboarding

Razorpay careers page

Apply now
On-site Lead / Manager

Group Manager - HSE (Health, Safety & Environment) and Sustainability

Bosch

  • Location:Bengaluru
  • Experience:16+ yrs
  • Employment type:Full-time
  • Salary not disclosed

Job Description Roles & Responsibilities: In charge of the PAN India operational and site management aspects of environmental, health, safety, and sustainability/ESG strategies of BGSW. Also, accountable for executing Bosch Group HSE and Sustainability Vision and EHS policy in operations, promptl...

  • Negotiation

Bosch careers page

Apply now
On-site Senior

Executive Assistant/Senior Executive Assistant

Zeta

  • Location:Bengaluru
  • Experience:2–8 yrs
  • Employment type:Full-time
  • Salary not disclosed

About Zeta Build the future of banking. Zeta is a next-generation banking technology company providing cloud-native, fully stackable processing and core banking platforms for issuers. With a focus on scalability, compliance, and innovation, Zeta empowers financial institutions to modernize their...

  • Project Management

Zeta careers page

Apply now
On-site Lead / Manager

Senior Manager, Banking Alliances

Razorpay

  • Location:Bengaluru
  • Experience:6+ yrs
  • Employment type:Full-time
  • Salary not disclosed

Razorpay is one of India’s leading full-stack financial technology companies, powering the way businesses move, manage, and grow money. Founded in 2014 by Harshil Mathur and Shashank Kumar with a simple vision - to simplify payments for Indian businesses - we’ve since grown into a fintech powerho...

  • Payroll
  • Salesforce

Razorpay careers page

Apply now