元NVIDIAエンジニアが語る、AIが1000倍安くなる理由

My job is to make the tokens as cheap as humanly possible. I will achieve that and I will do it through every layer in the stack available to me. I love the supply side levers. I will use every chip. I'll use every source of power and I will use every piece of land in the United States that's, you know, suitable for this. We still treat the agent as a person that is expensive to consult and you should ask them when you have a hard question. That's not the way to think about intelligence. It's incredible that the machine can think and we should [music] try to get that into as many hands as as many people as possible.

I think it's important early in these conversations to just say the thing like literally what you're building and what it does today. So maybe just orient us there with a with a brief description like literally what the system is that you're building and why it should exist.

Exactly. The theme of our company is abundance. We want to deliver this new commodity of intelligence to as many people as possible at a at a cost that is sustainable for almost every industry. We think that whenever you make something 10 times cheaper, it's a new product category and uh we aspire to do that for tokens. We think it's so profound that the machine can think and now our job is to make as many machines as possible in the world work towards thinking.

to start with? Absolutely. Token cost today my north star is I want to have the lowest cost per token in the industry and do that by a mile. I don't think tokens are the final unit of uh of work or intelligence but they are what we use today and so it's very straightforward. I think after tokens you start to move more towards um more outcomes which is like a vague direction. Uh you can imagine for example today when you consume tokens through an agent you don't actually control how many tokens the agent reasons for. It can reason for a certain amount of time or it can call a certain number of tools and increasingly I think we will have agents do some unit of work take as many shots on goal as they can and however many tokens they use to get there is going to be kind of a dependent variable depending on the task. So you think about like agents that selfadminister a token budget as opposed to a company setting a budget for how many tokens engineers can spend per month.

What was the unique opening that that you saw that's maybe the market's not being efficient in its attempt to tackle this? So I think there's two things that are tailwinds for our company. One is got to be the rise of open source. I had to talk about that first. I think we are starting to see an increasing number of our customers and the broader market care about owning intelligence. They they want to have control sovereignty over the thing that they depend on. Uh and so that created a much more robust market for customized models or even just like these vanilla open source models that no one can ever take away from you. You always have the weights. You always have the right to deploy them however you like. uh in that world there's been a reasonably robust market for the past couple years uh serving these models at large scale. The challenge is all those companies, you could take your pick, base 10, fireworks together, they all focus on low latency inference, and they were pulled in that direction by one very important customer, uh, cursor. And [clears throat] I think that that was the right choice about a year ago, and as of six months ago, it started to look like maybe low latency wasn't the only thing you wanted from an agent. You wanted more persistence, more long horizon tasks. And now, it's to me very obvious that the future of agentic inference is long horizon tasks. You're going to run the machine for hours or days at a time. It doesn't matter if it spits out tokens at 100 tokens per second. Maybe 10 is just fine. That comes with corresponding advantages and efficiency.

When you're waiting on it, you absolutely deserve the fastest answer possible. My trick is I don't want you to be waiting on it. I want it to be proactive. I want it to be in the background. One way to say it is like the best latency is no latency at all. When you wake up in the morning, the work's already been done overnight. you didn't even have to ask for it. Uh that's the dream. We're not quite there yet. But more importantly, I think the more you're in the loop as you prompt agents and wait for a response, in fact, you're the bottleneck uh in helping the in having the agent do more or less work. What we'd like is the agent to operate on more human time scales. You don't manage your colleagues every 5 minutes. You ask them to do a high level task and you come back and check in maybe every day, but more likely once a week. And that to me is the future of human agent collaboration, more like human time skills. say more about the early indications that this is happening and therefore you should be building this company.

You know, I love this market because it's unbounded. There's no human in the loop. So, you can consume as many tokens as you like in the background. Uh versus human attention span. If you tell me to consume 10x as many tokens at codeex or at cloud code, I'm actually not sure if I can anymore. I'm already in a loop and locked in coding for most of the the day that I'm at the laptop. What is undowned is how many tokens can be consumed in the background or proactively. So longterm, I think, you know, we're going to end this year at maybe 50/50 background and uh and real-time workloads, but I see this going to 9010 in favor of background.

Most deep research, most questions where you want to have a definitive answer over not 100 sources, not a thousand sources, but 10,000 sources or more. If you want to build an authoritative index of information like for example one of our customers parallel web systems seeks to do. They want to build an index over the whole internet and they want to monitor the internet in real time for changes. That is the kind of crazy exabyte scale task that you need a very different kind of intelligence or scale of intelligence to achieve. Deep research is a top category for us and then increasingly we see cyber security following this direction. If you think about, yes, there's so much code you can generate, but there's uh exponentially more ways to break that same code than it is to generate that code. And there are some great customers out there who are working very hard to uh find agents that can break any piece of software and proactively patch them. So when Fable first came out, for example, or Mythos first came out, basically there was this push in the cyber security community to run Fable against every line of code we've ever written and look for bugs in 20 different ways. uh meaning you're looking for both memory errors, you're looking for business logic errors and looking for like network vulnerabilities, all these things. And these are all actually things that you would write specialized agents for. You wouldn't just have Fable look at the source code once, you'd have it actually set up environments where you can pen pentest these applications. And at some point, people started to make this joke that security has become proof of work. When you want secure software, it's really a question of how many dollars did you spend on anthropics APIs trying to break into your software. uh that is the best indication for how secure it is because that's the best tool in the world. And increasingly we found that the frontier of intelligence here is quite jagged. It's not the case that Fable finds a supererset of all bugs in software. You would find some bugs with a very small model that you don't find with the large model. Uh you'd find some bugs with Haiku that you would find with Fable and vice versa. So it encouraged this very diverse approach to sampling and trying to build cyber security agents that break software autonomously such that you can patch them. If you were to get sort of like speculative and imaginative about the sorts of things that longunning very cheap very longunning agents can enable. We talked about some very practical examples deep research um cyber security etc. But if you if you get a little bit dreamier about the use cases new product category that this sort of inference will unlock and I guess the question is just like so what like what if you're maximally successful dream a little bit about what that might enable.

My estimation is yes, we totally can.

You have to be willing to spend tokens without any promise of return. That's the unlock. The long lens view to take on this is that we have we have a form of intelligence that can tackle any verifiable problem. Any verifiable problem means most software. It means a lot of formal like math proofs and similar. And it could also mean scientific discovery. These are all relatively verifiable problems. And all those things currently have a dollar cost attached to them essentially. That's a hidden one. It's like how many tokens could you possibly harness to make this work? And we have actually started to bring it within view a dollar cost for these long horizon tasks that is reasonable. It's not millions, it's thousands and maybe it could be hundreds or even tens of dollars in the near future to have a definitive answer to any scientific question to any research problem. RAMP is the only platform built to make your finance team leaner, faster, and better, saving businesses 5% annually on average, so you can stay focused on growth. RAM customers grow revenue 3.2 two times faster than the average American business. Visa, Verscell, Kerser, Stripe, Notion, 11 Lab, Shopify, and 70,000 other businesses all now run on RAMP. Mine does too, and so should yours. Learn more at ramp.com/invest. OpenAI, Cursor, Anthropic, Perplexity, and Verscell all have something in common. They all use work OS. To achieve enterprise adoption at scale, you have to deliver on core capabilities like SSO, skim, arbback, and audit logs. Instead of spending months building these missionritical capabilities yourself, you can just use Work OS APIs to gain all of them on day zero. That's why so many of the top AI teams you hear about already run on Work OS. Work OS is the fastest way to become enterprise ready and stay focused on what matters most, your product. Visit works.com to get started. Felix by Rogo is a personal finance agent that turns a single prompt into [music] finished client ready work using your firm's own templates, context, and standards. Send Felix an email like, "Take [music] these comments and turn them for me." Or, "Udate my tracker with the context of these emails." And Felix sends back finished [music] PowerPoint decks, Excel models, and sourced research. Felix works the way your team already does, delivering [music] work quickly and accurately around the clock. Learn more at robo.ai/felix. [music]

pretty much the questions we can ask. uh the models are on the cusp of basically taking even a high level question and chasing it down every possible follow-up you can have the model essentially take that on its own

and we will solve the token budget problem

I put basically the entire category of human taste into that category we have not solved human taste yet and I don't know that it fundamentally can be I'm excited to be surprised here but um we are focused on very quantitative uh problems we leave the quality of writing we the uh the beauty of art to to people.

ultimately to have this giant token factory, extremely lowcost intelligence supplier of extremely lowcost intelligence.

Talk through what your master plan is to approach this challenge that's so different from what others are thinking about doing.

Just describe what that means for for the lay person.

Simple as that. There's been a long history of how we evolved at tensor core over time that we'll get into.

That's a great question. I actually I cannot say that there is a divine truth of the inverse that explains why matrix multiplies seem to be the atomic unit of computation. But uh one way I've heard it described to me is well it's a really succinct way to mix two blocks of numbers together and have them interact in some interesting way. That's as much as I can say about it. It is really convenient that linear algebra turns out to be a very compact representation of arbitrary relationships in data. So Nvidia great graphics company obviously has had market share dominance in GPUs and and gaming graphics for quite some time. And then starting in like the mid2010s they started to actually start these like skunk works projects to make the graphics processor more suitable for machine learning tasks that they were tracking. I remember actually reading some of the like lab notebooks of some of my managers when I was at Nvidia. they would visit these small ML conferences like ICML or NURPS at the time and they would just take note of these papers like oh this deep learning thing seems to be catching on and what's really interesting is that these grad students are using gaming Nvidia GPUs in order to train their large models we should double click on this and figure out what's going on here and by 2015 2016 at least Jensen had the conviction to to kind of double down on hey this usage of our models is only of our chips is only going to grow let's start allocating more and more precious silicon die area to this capability that seems to be emerging. Let's put the first version of tensor cores on the chip. So, we're talking about, you know, taking this gaming chip which is designed for painting pixels on a screen and adapting it to do metric multiplies and it was early and you would be competing against the graphics teams essentially when you ask for more silicon area and any chip company. There's always competition for that. It is something that that the designers guard so carefully. you don't ever want to invest in the wrong technology because that's opportunity cost that you could have allocated to some other functionality. And so we we kind of like fought and tooth and nail and got just a tiny bit of diary maybe like 5 10% something like that for the first generation of these chips uh to get some some amount of acceleration for basic convolutions which were the fundamental operation for computer vision models in the day. Uh, and then we had a software team that was trying to squeeze all the performance we could out of the chip. And I think on that software team, which is where I work, that's what actually taught me the most about um, just the ethos that Nvidia has around they have this term called speed of light. They always chase the speed of light for any piece of hardware that they make. It is so ingrained in every engineer's mind that if the machine can do it, we're going to push the machine to the frontier until it does what we think is.

it's a it's a frugality that permeates the company. And so coming out of this time there, you get this experience of what it's like to develop more efficient usage of the underlying hardware through software.

And so so link that to, you know, today's environment.

Can you explain technically why the trade-off between throughput and latency is unbreakable? Why can't we have both from the same hardware? It's quite foundational in almost every system that you could ever possibly look at. There's always a trade-off between getting a small amount of data through the system as quickly as possible and leaving a lot of buffer uh room for that or trying to run wide and slow like narrow and fast or wide and slow is like a classic trade-off in all computer science. But for GPU specifically, I think there's one thing to focus on which is there's this concept of like batching on the GPU. We want to group many users work together into a batch that we can uh run all at once on the GPU. That's the parallel processing of the GPU. We'd like to have a lot of parallel work to do. The thing is though, you're doing net more work when you run a large batch of compute together. And so you might be filling all the units, but every step along the way as you carry a a batch of work through the GPU, there's more work to be done. And so any individual token or any individual user's request in that batch, it's going to spend a longer time on the GPU being carried with other people's traffic. Maybe the way to say it is um you know, if you want to get downtown and SF, you can take the bus or you can take a private transit. And the private transit is going to have its own direct path as the crow flies or you know, using exactly the roads that you want from point A to point B. a bus, it's going to have to serve many more people and it it has to fundamentally uh do something that works for everyone

and it's a great analogy and so step one for what you're trying to do is like create the best possible bus on top of Nvidia GPUs. Like that's step one of your optimization.

and have them connect their results together at the end. Reduce their results back together at the end. And this is a great great way to cut the minimum latency of of an operation. You're each GPU is now doing 1/8 as much work, let's say, uh, and therefore it can finish faster but not eight times faster. It's sublinear scaling. You'll use eight times more hardware, but you won't get eight times the speed. You might get like four to fivex the speed. You're not going to get strong scaling. And this is because of communication overhead. It's because every GPU is going to be a little bit less efficient working on a smaller tile of work than a larger tile of work. And so, it's the only way to speed up if you want the minimum latency possible. You can do that, but it is not the choice I would make. For example, I would prefer to use a different parallelism scheme like expert parallelism or pipeline parallelism. And we may do interesting things to overlap and hide the communication latency in a way that you would have less ability to do that for a low latency server.

And only latency performance

So Nvidia is excellent at low latency inference. And I'm telling you that we don't really care that much about low latency inference. So where does that leave us? Well, I think I'm not holding my breath for other companies broadly to figure out NVLink quickly. It's a challenging technology to figure out. It's hard to scale. It's hard to productionize. And so, if I do have some other vendors chip and it is good at the foundational compute components, it can still do metric multiplies really well. It just can't communicate those results across its peers quickly. Well, maybe there's a room for that other chip in my stack as a really really good compute per dollar option. And that's what I actually optimize for in most cases is how many flops does this chip have and how much is it going to cost me per hour to operate to own and operate. Uh and so there are other chips that definitely rank higher than Nvidia on flops per dollar, but they may not have as much interconnect. And so it's my job to figure out what parallelism scheme am I going to use that's going to make this chip suitable for inference. It's not going to be tensor parallelism. Nvidia is basically mandatory for that. But other techniques may work well for me. So before we leave the latency part of the story, can you comment on companies like Cerebras or others that can perform incredibly fast operations? I'm curious like what you think about those approaches, those companies, what might happen in the future. What is your prediction for the future of very low latency focused hardware? Cerebrus Grock uh and a couple others that are coming out of stealth now I think have made a very interesting bet on not just building another GPU but actually building a different kind of accelerator that focuses on a different memory hierarchy. Uh they want to maximize the amount of SRAMM on the chip and use that as very very fast memory for for weights and KV cache. So SRAM versus DRAM there's two ways to make memory for a chip. One is to integrate the memory on the logic die itself. Like meaning you tell TSMC, I want this many megabytes of of storage on my chip. Uh and there's a way to build that. TSMC has a standard cell library you can use and you can just print out a bunch of cells of SRAM. The problem with SRAMM is it takes a lot of area on the silicon die. Um, so if you want to build a large die like let's say the Nvidia Blackwell at 800 mm square. If you made that whole DS RAM, it would be in the maybe like singledigit gigabytes, it's not a crazy amount of of data storage. Compare that to if you're willing to take a different process technology entirely. So not TSMC anymore, but now Micron SKH Highix Samsung. They build DRAM, which is a whole different way to build memory that's more focused on capacitors than transistor cells. So, SRAM, the standard way to build SRAMM is what's called the 6T transistor cell. It's a stable transistor arrangement that allows you to write a bit to it and then it holds that state in that bit regardless of whether you keep applying. Well, you had to apply some power, but uh it it's holding that bit without any sort of like active management. It's static. Now, dynamic RAM, DRAM, it's dynamic because what you do to write some data is you write a charge onto a capacitor and as soon as you write that charge into that capacitor, the charge is dissipating. it's been leaking. And so the dynamic part of DRAM is that you must every 50 milliseconds or so refresh every bit you've written. So you're constantly juggling billions of balls in the air essentially billions of bits have to be managed by a memory controller which is reading and refreshing every bit on the DRM. Now the benefit of that is you can get much much higher density and it's a whole different process technology. There's a ton of different trade-offs. Hence why we split the DM manufacturing into an entirely different company like Micron SKX and Samsung. These are the best companies in the world to do this. They build DM. And if you take DM from those companies and you stack it uh into many layers and you kind of print them or or solder them around the main logic die that you get from Nvidia, you can now get hundreds of gigabytes uh like Blackwell has 288 GB of HPM capacity around the logic die. And the logic die itself maybe only has like 500 megabytes of of SRAM. So it's possibly multiple orders of magnitude, three orders of magnitude difference in density for DRAM versus SRAMM. Okay, so let's go back to Cerebras. What are they doing? Well, they see this problem, there's not really an obvious way to increase SRAM density on the chip. But thing with SRAMM is because it's so physically close to the logic gates that actually do the computation, the arithmetic logic units are right next to the SRAMM that they're going to pull from, the compute units that are doing the matrix multiplies can pull data from SRAMM at just mind-boggling speeds. You know, Serbis quits pabytes per second, 21 pabytes per second further away for scale engine 3. And so compare that to HBM on an Nvidia black wall is u you know 10 terabytes per second or so in that range. So once again, many orders of magnitude difference, more capacity, but proportionally less bandwidth essentially.

So there you go you have a path to a thousand tokens per second

Can you explain KV cache just like in

Yep. And and this is why sometimes people would observe like deep in a conversation things start to degrade because there's some sort of like technical problem.

And so these extremely fast, extremely low latency approaches ultimately are limited by by this factor.

Crisis and Grock and maybe a couple others, you should think of them as accelerators. What they are really good at is being used in conjunction with an more traditional GPU like device that critically has this offchip memory built in. You want offchip memory for capacity and onchip memory for speed. We want to hybridize these two things. So if you take uh transformers in the limit, you take a transformer to a million context length. What ends up happening is you have this you know computebound stage which is the actual matrix multiplies for the uh what we call the MLP which is where most of the model's knowledge world knowledge is encoded and then you have the attention layer which is where we're kind of dynamically adapting to the current conversation. Attention in the limit is usually memory bound and the MLP in the limit is computebound at large enough batch size. And I would say the original sin of transformers is that you've taken this extremely fundamentally memory bound layer and juxtaposed it right next to a computebound layer. It is very difficult to have a single chip that is good at both compute operations and memory operations. The GPU is quite balanced in this regard, but you have to choose one or the other. Cerebrus has a very fast memory access for something like a matrix multiply and it's really good to host the the MLP the the weights essentially on the Cerebrus chip but the GPU has the capacity to scale to really long context lengths and so you would like to put the uh attention possibly on the GPU and the MLP on the cerebrus chip

yeah, you've been so good at explaining some of the core concepts just for people that again aren't aren't deeply familiar with what this innovation was in 2017

uh learning arbitrary sequence data. And the most interesting sequences of data that we produce on a regular basis is language

Uh, and transformers benefit from a lot of the computer vision work, too. For example, one of [clears throat] the challenges in computer vision was we had a hard time going from hundreds of thousands of parameters, which you get for like linear models like support vector machines or other legacy machine learning models. Those had, you know, on the thousands of parameters. Then we got to deep learning and got to tens of millions of parameters with computer vision. The biggest models were you know around like 150 million parameters was a huge model for computer vision. And now we routinely talk about trillions of parameters and transformers are the link to go from millions to trillions of parameters.

does it stand a reason then that you think transformers will just stick around because that's the thing that we're good at getting more of those two things. Well, data is an open question, but comput. Yeah, transformers are so they're just such great sponges, you know, like you you you increase the compute available to a transformer by 10x and you'll get you'll get some log improvement somewhere. Uh and and so far the scaling laws really work. They're really quite beautiful. And to [clears throat] the point about I guess what do transformers do really well? They extend to almost any data set you can throw at them. They're extremely powerful general learners. And I think what's especially useful about transformers over other techniques that we've tried to replace attention is transformers represent any pair wise relationship that you want. Any token in the sequence can attend to any other token in the sequence. So if there's any relationship that's in the sequence at all, you're going to find it with transformer. Now it may be the case that you don't need all toall modeling. You don't need every token to look at every other token. But if you need to, transformers give you that option. And until we know a better way to kind of prune that space down uh a better way to kind of have information modeling be more selective attention is a very very good operation. This is another kind of trick that we learned in the computer vision days. Like one of the old Karpathy sayings is that you know if you have a new data set that you want to train a model for. Your first goal should be to overparameterize the the model and try to overfitit the data that you have to prove that there is a relationship that you can model or memorize that your learning algorithm works uh that you can instill knowledge into the model. Once you can overfit then you can compress and the compression is how you get generalization. You don't want to actually memorize the data that you have in front of you. you want to generalize and therefore once you overfit the data set then you can kind of work backwards and try to find the general patterns that fit into the smallest parameter count possible.

generic Yeah. Everyday Joe. Exactly. So the feature of data to me is giving the model a hard verifiable task and letting it run in this gym where it's kind of isolated and it just has a a problem that it can make progress on and get measurement of whether it made progress on that problem or not. You can imagine coding problems are in this category. math problems are also in this category and um increasingly more and more we have we can just give the agent a computer essentially and have it act like it's a human worker and just give it feedback on whether it's making progress towards the target outcome that environment becomes the data. I think this is not a super differentiated take but uh it's been really really productive from what I've seen so far. And you think that just goes on for a really long period of time or is that another like if I think about the internet as this one big block like this is another big block that will have its you know day in the sun and we'll kind of get it all and and then we'll have to move on to something else.

Okay. Okay. Now, so I like that we've veered off in different little side cars here, but coming back to your initial task of making existing hardware more efficient.

Sounds good. So, yeah, I mentioned kernels. It's surprising people think kernels are done. There are great people like Triau who write excellent kernels and they're they form the bedrock of all of our um modern deep learning is built on flash attention. Modern transformers are built on flash attention. But if you deviate from the happy path at all, if there's a new model that comes out that has a slightly different way to embed positional information like the change of the rope system. Suddenly the kernel that we had is not suitable for this new model and we may have to make a a patch to this kernel. I wouldn't say we're in the phase where we had to invent new kernels from scratch, but having the ability to quickly modify existing GPU kernel, sorry, a kernel, by the way, is a it's a general term for any program you run on the GPU. And so, historically, kernels tend to be put into a library where every kernel has a very very scoped purpose. Typically, you have a kernel for a matrix multiply. You have another kernel for even something as simple as addition. You want to add two tensors together, that's another kernel.

And then increasingly we've started to fuse those kernels together. So if I do a matrix multiply and then I want to add it to another matrix that I've also multiplied maybe those two become one kernel and I just fuse the operations where instead of writing the data out to DRAM and then reading it back in just to do the addition maybe I can just do this uh easily.

I don't want to speak for Tree, but what he taught me was uh you shouldn't write kernels by hand anymore necessarily. I like to say we write kernels in the whiteboard. We go to the whiteboard, we describe what we think the machine should be doing, then we succinctly describe that in in natural language to the a model. And then the model is able to do the execution of okay, here is my input and output. here is the strategy of how we want to dispatch this work onto the GPU. I'm gonna go implement this.

Exactly. And that I think I'm not sure exactly why models are not superb at doing this. I don't think this is like our remote or anything like that. I'm sure in 6 months time we'll have much better models uh on kernel engineering and I'm sure the labs would tell you that they already do a lot of their kernel engineering uh in a fully automated way. And so software as an edge,

if I think about software as maximally near speed of light, efficient usage of of an underlying piece of hardware

That's right. The rising tide of something like Mythos or GBT 5.6 Soul that lifts all boats. It really does. Um I actually don't think there's a point in specializing to say we work on making the model better for just kernel engineering. I think that's actually not not the most meaningful subset of of like coding in general,

and then if there's a certain amount of energy available inside of a chunk hunk of coal like what percent of it we can harness and use

right

like what percent do you think we're at like How how efficiently can we use an existing piece today?

because of heat.

but in practice you don't spend the majority of your time in a transformer in that happy path where you're doing a large batch m matrix multiply

uh our job is to basically build the engine around the chip such that we are feeding the GPU these large batches of work at all times

And it's all comes down to like this is a very new paradigm of programming.

is like a drug market or something right now. Like there's all sorts of fascinating things happening to get as many of them as possible because everyone's so short. Y

but then also to talk about what the market is like for like not the bleeding edge chips like if I if I

what's that market like let us into that world

on the Nvidia side. And it's all because the scarcity is so high and demand is just off the charts. Now, for other chips, I wouldn't even call them inferior. I I like to say there's no bad chips. There's really bad pricing. And uh I will make any chip work at the right price. That's like kind of one of the ethoses of the company. And let's talk about AMD. AMD, I think great chips overall. The challenge is that people don't um understand how to program them very well. So, you know, I've been talking to you about how we have such a great kernel team. We're so serious about squeezing the performance out of the hardware. Nvidia is pretty good at doing that for their own chips. Frankly, there's some alpha that we can squeeze out, but actually there's a lot more to be done on other chips because the vendor does a little bit less work than Nvidia does to make the best kernels out of the box or or even better for me there is alpha and just like other people have this perception that AMD is not as as good as Nvidia. That's music to my ears. I'm very happy for them to sleep on this chip and for me to buy as much as I can.

And so it's fundamentally an arbitrage for you. Like if you can be much better at eking out performance from chips that have received less attention, you can then resell that at a margin and it could be a great business.

Like you look at memory stocks.

Historically it was like 2 3 4%. Now it's 19 20 21%. And it just sort of looks like if you're a student of market history, you get all these things through time that are sort of reached some crazy near-term peak and then and then collapsed back to long-term norms.

Yeah. Uh so you know a lot of people made a lot of money in Micron and Skhinx and companies like this but everyone feels like ah these you know on the long term like compute's a commodity and uh it will not represent a quarter or fifth of the entire market capitalization of the world and and so they're scared and that's the setup. Um everyone acknowledges that like there's a huge shortage but everyone sort of feels like ah we'll figure it out and these things will revert back down to their their normal place in capital markets. I'm curious what you think about about that narrative.

Coming back now to your take on hardware. And so the unit level is interesting to me like talked about chips, talked about racks, talked about, you know, clusters. I'd love to talk about data centers

Y

I think one of the themes in our conversation has come back to what is training versus inference like what is the difference between these two? categories and you know what was different about two years ago being training oriented and today being inferenceoriented and I think the most conservative players in the entire AI stack have got to be the infra players whether that's data centers or even more conservative is TSMC the chip infra people uh and so data centers historically were built like AI data centers they were built for training uh and training is the superset workload over inference you can make any training cluster work for inference but maybe not vice versa and what the difference there is networking uh how much do you invest in bandwidth between chips and how large of a cluster do you need? There's actually a diseconomy of scale to to data centers in some way. Like it's way more expensive and difficult to build a um you know 100,000 GPUs in one data center than it is to build 10,000 than it is to build 1,000. And and we now we just talk about you know how many megawatts or gigawatts do you have? And basically there's no way to build a gigawatt data center in the United States easily anymore. Even 100 megawatts is is increasingly hard. It's basically impossible unless you're a very special set of customers. Uh 10 megawatts is probably on the edge of what's possible today and 1 megawatt I would argue is plentiful. So there's this incredible lore on the market where you can find lots of aggregate power but it will not be concentrated and that was not interesting to anyone who's building training uh data centers because you just assume all be in one spot for no one wants to deal with cross data center training.

but increasingly we're seeing a few new thinkers realize that inference is going to be suitable for these distributed 1 megawatt data centers and uh we're we're quite in agreement with that and we are very happy to buy small pools of compute across the United States and use that as our inference fleet. give us a sense of literal physical size of uh 1 megawatt versus 10.

That you can buy, you know, you're a buyer to ek out the most per chip.

data centers

your ability to eat more out of it, and then small units of expression in a data center

which is bad.

you'd have basically zero buyers for a data center that has 95% up time

up time

partially uh it's actually two things one is that we have a really robust control plane that is going to be fine handling any single failure in any single data center as long as it's not correlated with other data centers and I can just move the workload somewhere else I'm cool with that um the failures happen at some rate and I am basically linearly happy with a data center that's 95% uptime versus 98% versus 99%. It's just linearly good or bad for me.

Now, you do need that async piece that I mentioned of, you know, we serve these long horizon agents because what happens when a request fails is that I'm going to have to go find a new GPU to put that request on. And that means that for that single turn of the agent's work, you know, it's working for an hour, but then it hits a roadblock because it's GPU got pulled away. At that moment in time, that agent is going to experience maybe like an extra minute or two or three, maybe even 10 of latency. But my argument is that my customers don't care because their agent was running for hours.

Doesn't matter. [laughter] It doesn't matter if like a single turn occasionally becomes uh a little bit longer. Yeah. So we tell our customers, look, our average throughput is going to be very competitive, but our P99, our 99th percentile latency, it's not going to be controlled. It cannot be. And in return, I'll give you unbeatable economics.

Talk about power as a category. What are you seeing that's interesting, innovative, where do you think this goes?

And if my chips are cheap enough, they're probably not going to be Nvidia racks. And if my chips are cheap enough, I don't mind the capital cost of having idle chips.

That's right.

We are building a factory. We're trying to build the best steel factory in the world. Uh but it will come through mini mills not through large monolithic steel plants.

Yeah.

to your user like your user is me

But you can imagine many other permutations of the business

across all this stuff the virtual scavenger how do you think about that question of like which which type of these businesses to be You're doing real life factorial basically. we have to be more pragmatic. I think that the capital we're we're looking at for owning everything is like you said, it's insane. Yeah, software has high leverage, so we have to start with software, but ultimately, you know, do we own power generation or can we get great power purchase agreements with uh utilities? I'm more inclined to pursue like letting other people specialize in the things that they're historically good at and then see if we can get to the scale. I think of it as like I want to get to the scale where I earn the right to take this under our wing. I absolutely think that there's efficiencies to be gained everywhere in the stack. If you can break the assumption that people I would be buying from, they made assumptions about who their customers would be. And I maybe break those assumptions. It's a pretty optimistic view. Uh I think it's only possible because we're actually trying to underwrite the largest market for compute in the history of computing. we're actually going to build so many billions, trillions of dollars of investment into inference. Uh, and because of that focus, it makes sense to build a lot of things that are custom for inference. And it's my job to seek all the places where that's possible. And then as as they become obvious to me and my and my partners, I will get my partners to build custom things for me. And if they can't do it for me, I will do it myself. Yeah. I think compute scaling is actually like very efficient. Uh as in like you give me more flops and I will use more flops. And I would say we're actually fairly judicious already with our use of flops. Uh if you look at a modern model, there are very few models that are more than 10% dense, meaning 10% of the possible number of experts you can activate are activated. And I think the frontier models are closer to like 1%. So fairly sparse already. I don't think that we're wasting too much on the MOE side. People have been working with for quite some time. They're pretty good at squeezing. Where we are not good is attention and its use of memory. Specifically, the KV cache is quite uncompressed right now. I think if you look at the entropy in a KV cache, it's nowhere near it's not earning its keep. Like we're storing many kilobytes of data in the KV cache per token. Um, and that's probably off by an order of magnitude or two. And I I don't know what the Frontier Labs do, but Deep Seek certainly publishes really interesting work to compress that further and further. And they're making good progress. And I think the fact that they're able to make order magnitude progress here every year or so signals that there's a lot more room to go. I guess this all on the micro scale. If you zoom out further, I think that we actually don't marshall our compute effectively at all. Like we have all this compute in the world. Nvidia is pumping out 5 million Blackwell chips this year. Where are they all going? Are they all being used at all all the time? I certainly doubt it. I think that at some level we just need better orchestration of compute across the world. Uh this is very difficult to do because a lot of the comput disappears into private pools of compute that will never see the light of day and those GPUs sit very sadly idle. Uh it's actually it pains me physically to see that those GPUs are just you know silicon and power went into that and it's just sitting idle and I want to fix that. how we or organize and orchestrate the world's compute as a shared resource and and pack it more efficiently. I would estimate that, you know, we all make fun of XAI for having, you know, some challenges with total flop utilization on its clusters, but um the reality for the rest of the world is it's far far worse. A ton of GPUs just sit in warehouses or sit in private pools allocated to a specific customer um just don't get utilized. That's way more effective. Yeah. Um, what about fabs? Like what do you think is the future of fabs themselves? Like I think everyone is wondering be able to how will they expand capacity basically? Will we do it here in the US? Yeah. Well, it's interesting. Everything grows in balance with each other, right? If we snap our fingers and double all those things, you might fix a TSMC bottleneck that there you're just going to run into another bottleneck. You make 20% more chips, then you have another bottleneck immediately. I will say though, it is interesting what they consider to be a mustd deliver. uh like what they consider to be like an invariant that their customers me are always going to want versus what I think of as like a more fluid relationship. I think that if a fab exposes more of their trade-offs to me, I'm able to make more intelligent decisions about what I think I can I can do. Can we talk about how you designed the system of your own business? But like bring me into the culture and how you structure a team and a business where this is the northstar. Curiosity. It's 100% curiosity. You know the one thing I cannot teach is love for performance, love for uh digging into every microscond that the machine is working and understanding what's happening on the machine at that time. That to me is the most important trait for a performance engineer and it's what I look for. I don't look for lots of AI experience. I don't look for, you know, CUDA experience at all. That's actually a huge red herring. I mean, CUDA as a concept or GP as a concept have evolved so much in the last 5 years. There's no point asking for 10 years of experience. I want to teach that, but I cannot teach the love for performance engineering. That is what I seek. in a line. I would say the labs pay an immense premium to be 3 to 6 months ahead of of everything else. Uh and I think that's probably still worth it. I think it makes perfect sense for open and anthropic to do what they do. You know there's a sensitive topic around distillation which I think is part a very core piece of the relationship between closed and open frontier. And you know I'd like to offer an alternative view on that which is there is the sense that distillation is theft that you are taking something from the frontier models when you distill on their outputs. And in fact, even if that's not your intent, even if you don't ever try to, you know, scrape data from anthropic, one thing I'll offer is that an increasingly large percentage of the artifacts we put out on the internet are AI generated. Even if you just look at GitHub alone, you know, what percentage of repos created in the last year do we think were created by cloud code? Um, do we consider that to be distillation? Because that's probably all we need. I would not be surprised if you could train a fable glass model only on the outputs of code you consider good on GitHub that's open source. And certainly if we take the position that users own the outputs of their interaction with AI and they choose to put that up on GitHub, which a lot of them do, we're going to have latent distillation for a long time. It seems fundamentally impossible for me. Like I I don't think it's fundamentally possible to prevent the diffusion of of information or model capabilities. It will happen. The question is just how fast. And so then the question becomes, do scaling and improvement laws hold forever or for a really long period of time? And if they do, then there's value to being three and six months ahead and that will just last as long as it lasts and they can charge a huge premium for those tokens relative to a very cheap open source token. Is that the right way to think about it? And so your hope of what the future looks like is what like what balance between closed and open, you know, what balance between model companies doing everything because they have the advantage of owning the stack or whatever. You know, Enthropic can do that, you know, is like the new Google could Google just do that or something. What do you hope the future looks like? every company every user even make the agent your your own. Uh I think we're we're very not that far away from that level of customization and capability. I want people to own their intelligence and I want that intelligence to be customized probably not through weight fine-tuning but probably through more in context learning. That's a more technical detail. But the underlying input to this abundance future is about is basically cheap tokens. My job is to make the tokens as cheap as humanly possible. I will achieve that and I will do it through every layer in the stack available to me. I love the supply side levers. I will use every chip. I'll use every source of power and I will use every piece of land in the United States that's you know suitable for this. And in return, people will have the incentive to explore what it's like to have abundant intelligence. We still treat the agent as a person that is expensive to consult and you should ask them when you have a hard question. That's not the way to think about intelligence. It's incredible that the machine can think and we should try to get that into as many hands as as many people as possible. Most of the ideas on chips, I would say. You know, when I talk about building custom chips and they ask me, "Oh, so what's different?" Basically, it's it's about sidestepping the HPM shortage and focusing on more extreme offload to other forms of memory such as flash. Um, I'm quite passionate about that idea. Everyone on my team knows that I keep banging the drum around like what would we have to change about the model architecture to make offloading KB cache to flash work at a much greater level. And um I'm whiteboarding that all the time. That's like in the community of like inference people. you know we have some divergent views on what you can do if you design a system around serving at you know one to 10 tokens per second which is our whole north star more broadly I think there is this like larger sense around you know what do you do how do people consume a trillion tokens per day like that's the world we want to create the capability for them to do that a trillion tokens well okay at openi pricing that's at least $5 million at the very least for 5.5 or 5.6 six. Yeah, I think the dollars was probably the most good metric. Yeah. Yeah. So, what's the world in which we consume what currently costs $5 million per person per day? Are you at all worried that just like the average person just can't and won't do that like doesn't do that now with their own brain? Like there actually isn't that much demand for intelligence in the world. You want to enable those people. What about the inverse question? Not what you think is craziest, but like what consensus thing you think is wrong? so the consequence of this is people lose their minds over geopolitics like what what happen if we lost access to DMC for any reason. And um my contrarian take is that it wouldn't be that bad. Supply would take a shock for sure, but the best processes that we have in the west uh like Intel not that far behind at worst like maybe 2x uh worse performance per watt and the gap is just far smaller than than you would make it out to be if you talk if you follow like the chipboard dialogue. What else is happening in the AI world that is not in your path? Meaning it's not like a component of this whole system that you would end up doing something in that interests you most. if you had a 100 entrepreneurs in a room, all of whom wanted to create some new compute startup. um and let's say they were specifically wanted to make hardware chips or systems or racks or whatever. because it seems like we're going to try everything and that will be great for the world. You know, some stuff will work. But if you had to give them advice on how to orient their business to be successful in this coming world, what advice would you give them? It's all about the bottlenecks on supply chain. So, you need to first convince me or convince an investor that you understand the like three to five bottlenecks that dictate modern chip supply. There's TSMC wafer capacity, there's HPM capacity, and there's um like advanced packaging, and maybe a fourth one would be power. Like, where will you get the power? How will you build these racks? Uh and I I want to hear like you should have a great answer to each of those four bottlenecks and how you're going to work around them because it's all arbitrage at the end of the day. You're building a chip because you think that Nvidia has made some choices that are difficult for them to change, which is true. Nvidia makes a lot of choices that are difficult for them to change. They're not perfect. They're just really well balanced. And so, you want to be spiky. You want to pick something and say, I think they've underpriced the impact of how short we're going to be on HBM. We're going to push really hard in this other direction instead. Which, you know, as a as an aside, I do think is probably the thing to attack most. so it's going to be a while until we They they've been burned on that many times. I think they're gonna make everything else more expensive. Think that iPhones will cut their memory. iPhones are going to go up in price and um we're just going to deal with it. Why doesn't Nvidia go all the way to the end and sell tokens? Do you think It's great to have competition amongst his buyers. The kindest thing I mean I my immediate first thought is like all the mentors that I've had over the years. It's a rare person who takes a lot of time out of their their schedule and um and you know makes it like their personal interest essentially to to make sure that you understand something that uh or teach you something or or like ingrain some value in you that they think that you're on the cusp of understanding but just push you over the line for understanding. uh a lot of the people in Nvidia that I mentioned earlier who instilled that like love of performance engineering in me but also my professors in college who I remember like my adviser in like sophomore year I was very impatient student so I show up at his office hours and say like I I want to build AI chips I know what I want to do why am I wasting time taking all these like other basic classes and networking and you know operating systems and he just looked at me and said like you know he laid out basically like the whole stack and showed me the depth of or the beauty of like understanding every piece in the puzzle like he he took my entire path of like trying to focus on one piece of the the system and said that you know it's so rare that someone can actually understand the entire stack from the gate level silicon all the way to building a great internet scale service and you know you should aspire to be someone who over the course of your lifetime achieves that level of understanding. Thank you so much for having me.