Uber used RAG and AI agents to build its in-house Text-to-SQL, saving 140,000 hours annually in query writing time. 📈
Here’s how they built the system end-to-end:

The system is called QueryGPT and is built on top of multiple agents each handling a part of the pipeline.

1. First, the Intent Agent interprets user intent and figures out the domain workspace which is relevant to answer the question (e.g., Mobility, Billing, etc).

2. The Table Agent then selects suitable tables using an LLM, which users can also review and adjust.

3. Next, the Column Prune Agent filters out any unnecessary columns from large tables using RAG. This helps the schema fit within token limits.

4. Finally, QueryGPT uses Few-Shot Prompting with selected SQL samples and schemas to generate the query.

QueryGPT reduced query authoring time from 10 minutes to 3, saving over 140,000 hours annually!

Link to the full article: https://www.uber.com/en-IN/blog/query-gpt/?uclick_id=6cfc9a34-aa3e-4140-9e8e-34e867b80b2b

How Much GPU Memory Needed To Server A LLM ?

This is a common question that consistnetly comes up in interview or during the disscusiion with your business stakeholders.

And it’s not just a random question — it’s a key indicator of how well you understand the deployment and scalability of these powerful models in production.

As a data scientist understanding and estimating the require GPU memory is essential.

LLM's (Large Language Models) size vary from 7 billion parameters to trillions of parameters. One size certainly doesn’t fit all.

Let’s dive into the math that will help you estimate the GPU memory needed for deploying these models effectively.

𝐓𝐡𝐞 𝐟𝐨𝐫𝐦𝐮𝐥𝐚 𝐭𝐨 𝐞𝐬𝐭𝐢𝐦𝐚𝐭𝐞 𝐆𝐏𝐔 𝐦𝐞𝐦𝐨𝐫𝐲 𝐢𝐬

General formula, 𝐦 = ((𝐏 * 𝐬𝐢𝐳𝐞 𝐩𝐞𝐫 𝐩𝐚𝐫𝐚𝐦𝐞𝐭𝐞𝐫)/𝐦𝐞𝐦𝐨𝐫𝐲 𝐝𝐞𝐧𝐬𝐢𝐭𝐲) * 𝐨𝐯𝐞𝐫𝐡𝐞𝐚𝐝 𝐟𝐚𝐜𝐭𝐨𝐫

Where:
- 𝐦 is the GPU memory in Gigabytes.
- 𝐩 is the number of parameters in the model.
- 𝐬𝐢𝐳𝐞 𝐩𝐞𝐫 𝐩𝐚𝐫𝐚𝐦𝐞𝐭𝐞𝐫 typically refers to the bytes needed for each model parameter, which is typically 4 bytes for float32 precision.
- 𝐦𝐞𝐦𝐨𝐫𝐲 𝐝𝐞𝐧𝐬𝐢𝐭𝐲 (q) refer to the number of bits typically processed in parallel, such as 32 bits for a typical GPU memory channel.
- 𝐨𝐯𝐞𝐫𝐡𝐞𝐚𝐝 𝐟𝐚𝐜𝐭𝐨𝐫 is often applied (e.g., 1.2) to account for additional memory needed beyond just storing parameters, such as activations, temporary tensors, and any memory fragmentation or padding.

𝐒𝐢𝐦𝐩𝐥𝐢𝐟𝐢𝐞𝐝 𝐅𝐨𝐫𝐦𝐮𝐥𝐚:

M = ((P * 4B)/(32/Q)) * 1.2

With this formula in hand, I hope you'll feel more confident when discussing GPU memory requirements with your business stakeholders.

#LLM

Best article on GenAI getting started
https://blog.bytebytego.com/p/where-to-get-started-with-genai

The matrix calculus for Deep Learning. Very well written. https://explained.ai/matrix-calculus/