In March 2023, an engineer at Grab named Wenbo Wei started looking at what LLMs could do inside the company. By then ChatPT had taken the world by storm already.
He had one annoying problem in front of him. His team ran Grab’s ML Platform, and their Slack channels mostly had the same questions asked almost every day. The on-call engineer spent more time answering users than building anything meaningful.
So he tried the obvious fix. Take the platform’s Q&A documentation, put a chat box in front of it (like their private ChatGPT), and let the LLM answer.
Naive RAG Approach
Their internal docs ran to more than 20,000 words. At the time, GPT-3.5-turbo accepted only 8,000 tokens(compare that to 1M tokens context window today), which Grab’s post describes as roughly 2,000 words.
So he cut the docs down. And then cut some more. To make it fit he had to get under 800 words, and by then the details people actually asked about were gone. He added semantic search on top. Still not good enough to hand to engineers.
Most of us would stop here. Idea failed, close the ticket, move on.
How GrabGPT was born
He stopped fixing the documentation bot and looked at what was already in front of him.
A working chat interface, the open-source chatbot-ui. Grab’s Google login. And access to Catwalk, the model serving platform his own team ran. Grab had no internal ChatGPT at all.
Over one weekend he put the three together.
That was his complete implementation actually. No new servers, no team, no project plan. Open source gave him the interface. Google login meant nobody created an account. Catwalk meant nobody built serving infrastructure. It ran on a private route instead of the public internet.
He called it GrabGPT.
Then people showed up
Three hundred signed up on day one. Six hundred more on day two. Nine hundred more by the end of the week. By month three it crossed 3,000 users, with 600 people on it daily. When Grab wrote the story down in May 2025, almost the whole company was using it.
No launch, no marketing, no mandate from above. So why did it spread?
Grab gives four reasons, and none of them is about the model being good. It ran on a private route, so company data stayed inside. It worked in every region, including the ones where public ChatGPT is blocked. It was not tied to one provider, so OpenAI, Claude and Gemini sat behind the same box. And every conversation was auditable, which is what data security and governance needed before saying yes.
Read those four again. Data staying in-house. Access. Not getting locked to one vendor. An audit trail. Same four arguments we have been having about internal tools for ten years. GenAI added nothing new to that list.
LLM-Kit - Starter kit for GenAI Apps
One chat app in front of a few providers is easy at 300 people. Harder at 3,000. It breaks completely once other services start calling models from production code.
In November 2024, Grab published LLM-Kit, a starter kit for teams building LLM applications. You asked for one and got a working repository: API server, config, a sample agent with tests, CI files, staging and production infrastructure already written in Terraform. FastAPI, LangChain, LangSmith, Vault, Amazon EKS, GitLab CI, Redis, PGVector and Datadog underneath.
They measured the saving at about one and a half weeks per team, before anyone wrote a first feature, and used it to onboard hundreds of GenAI applications.
Three months later came the piece that matters most for platform people: the AI Gateway. Wenbo Wei is one of its four authors, which tells you the two pieces of work were connected.
Grab AI Gateway
This diagram is my reconstruction for teaching. Grab has published the gateway’s parts and how a request flows through it, but not this exact picture.
Grab describes the gateway as a deliberately small layer between the caller and the provider, stepping in only when it has to. It does six things.
Hands out keys. Exploration keys are short-term, belong to a person, reach staging only. Production keys are long-term and belong to a service. Permission is granted per request path, so a key gets one provider or one feature and nothing more. On the way out, the gateway swaps your internal key for the real provider credential, so no team ever holds a provider secret.
Limits each key. Its own request rate per key, on top of the provider’s global limit. One badly behaved service cannot drink the whole company’s quota.
Speaks one language. Grab picked the OpenAI request format as the standard and translates to whatever the provider expects. Changing model becomes changing one parameter.
Reroutes traffic. Between similar models and between regions, so reserved capacity gets used and quota limits get worked around.
Writes everything down. Request, response, tokens, path, model name, all into the data lake. Security and compliance query it like any other table.
Counts the money. Cost per request after the reply returns, async work picked up by a daily job, totals on dashboards with alerts.
One detail I like: Chimera notebooks mount an exploration key on their own, and Catwalk deployments get the gateway settings as environment variables. A data scientist at Grab never sets up an API key. The platform hands it over.
That gateway now fronts 50+ models and 300+ onboarded use cases, with more than 3,000 Grabbers having asked for a key. Grab’s own open models run through Catwalk’s vLLM offering and sit behind the same door as OpenAI.
Three things are still on their list: a model catalogue with token limits, quota and pricing; protection against prompt injection and bad output; and rate limits counted in tokens and money rather than requests. Most teams find all three the hard way.
And then agents
By July 2026, LLM-Kit had been rebuilt.
Grab now reports more than 500 services on the internal agent framework, over 50 registered MCP servers, and one gateway carrying every model call in the company at billions of tokens a month.
A team asks for an agent, fills a form, and gets a GitLab repository already wired: a LangGraph reasoning loop with a 30 second step timeout and retries; tools discovered at runtime over MCP, so adding a capability does not mean redeploying; Postgres with pgvector set up automatically; Glean search across Confluence, TechDocs, Jira and internal drives; an evalshub endpoint with ROUGE, BLEU and LLM-as-judge and golden tests already written; OpenTelemetry tracing through FastAPI, LangChain and MCP; plus OIDC, Vault, distroless images and gRPC contracts.
That used to take two weeks. Now about an hour.
Read the list again. Timeouts. Retries. Service discovery. Secrets. Tracing. CI. If you have run production services, none of that is new to you.
Put the two versions of LLM-Kit side by side and you can see what actually changed in two years. LangChain became LangGraph. LangSmith and Datadog became OpenTelemetry. Hardwired tools became MCP. Evaluation got promoted. Vault, OIDC, GitLab CI, pgvector and FastAPI carried straight over.
What AI platform engineers should take from this
Nobody at Grab designed this platform in 2023. A chat tool proved people wanted it. A gateway brought model access under control. LLM-Kit made shipping repeatable. Agents came last, on top of everything else. That order is the useful part.
Here is what I would carry into my own platform:
Build the gateway before anything else. It is the smallest piece that gives you access control, cost visibility and an audit trail from the day you switch it on. Routing, fallback and budgets all hang off it later.
Standardise the request format before you argue about models. Grab picked the OpenAI schema. That single choice is why swapping a provider became a config change instead of a rewrite.
Put the audit trail in on day one. It is what got GrabGPT approved company-wide. Retrofit it in year two and you are rebuilding under a deadline.
Hand teams a running repository, not a library. LLM-Kit gives you CI, secrets, tracing and infrastructure already wired. That is what took setup from two weeks to an hour. A library with good documentation would not have.
Ship evaluation inside the template. An eval endpoint that arrives with the repo gets used. One that arrives as a separate project does not.
Meter tokens and money, not just requests. Grab’s own roadmap admits request-rate limits are the wrong unit for LLM traffic. Start where they are heading, not where they started.
Most of this is already your job. Timeouts, retries, service discovery, secrets, tracing, CI. Two things are genuinely new: MCP for finding tools at runtime, and evaluation as a first-class endpoint. Learn those two properly and the rest of your experience transfers directly.
And keep your failed experiments. The documentation bot never shipped. It still showed Wei where the wall was, and it left behind the pieces he used that weekend.
Sources
Wenbo Wei, From failure to success: the birth of GrabGPT, 19 May 2025
Bjorn Jee, Daniel Tai, Siddharth Pandey, Wenbo Wei, Grab AI Gateway: connecting Grabbers to multiple GenAI providers, 19 Feb 2025
Boon Zhan Chew, Kendrick Tan, Swati Joshi, Yu Jie Ang, Supercharging LLM application development with LLM-Kit, 29 Nov 2024
Kendrick Tan, Jeffery Lean, Yi Sheng Tay, Agent platform: how we help Grab build and run AI agents at scale, 24 Jul 2026