Introduction
On January 10, 2025, DeepSeek launched an LLM, R1, that the startup claims is on par with OpenAI's ChatGPT o1 for reasoning tasks. The app shot to the top of App Store rankings in record time, captivating not only the tech industry but the wider world. One claim was particularly striking: that the model had been trained for under $6 million (compared with the $100 million OpenAI reportedly spent on GPT-4). This sent ripples through the stock market and the media.
While the media and financial storm is hard to ignore, what interests us more at Adservio is understanding exactly what DeepSeek did and how. We'll start with the key facts and our first impressions, then explore what we know so far from their papers about the model architecture, training data, evaluation approach, and the techniques used. Finally, we'll look at attempts to reproduce their results and what comes next.
DeepSeek's new models: what are the facts?
Who built DeepSeek
Founded in May 2023, DeepSeek is a Chinese AI startup based in Hangzhou and Beijing, backed by the Chinese hedge fund High-Flyer. Both High-Flyer and DeepSeek were founded by Liang Wenfeng.
On January 10, 2025, DeepSeek launched its mobile app; on January 20, 2025, the company released R1's weights on Hugging Face and R1's inference code on GitHub.
What did DeepSeek actually build? DeepSeek built two types of models, plus applications to use them: V3, the latest version of a general-purpose language model, and R1, a reasoning model built on top of V3-Base. They also provide smaller versions that can run on a laptop.
There are two variants of V3. One is built on Llama (Meta's open-weight model) and the other on Qwen (Alibaba's open-weight model).
Although they released the weights for R1 and the code needed to run the model for inference, they did not release the training code or the code for all the hardware-level optimizations they applied.
Our first impressions using DeepSeek
Some of us at Adservio used DeepSeek through the company's website, while others used Ollama to run a smaller R1 model on their laptop. We then spent time using the model the way you would use any other, for tasks ranging from coding to reasoning questions.
Based on our experience over the past few days, here are some of our initial impressions and thoughts:
Multilingual performance was solid in English and Mandarin but less fluent in French, with occasional unintended Chinese or Arabic characters appearing and the occasional fallback to English during complex reasoning. Its reasoning style can be excessively verbose, at times it seemed to go in circles while working through a problem. We'd like to know more about how DeepSeek thinks about safety and privacy, especially from the user's perspective. Model instances are available in various sizes and can run on a range of consumer-grade hardware, including energy-efficient setups. The hosted version appears to have guardrails aligned with the Chinese government's worldview; the model itself may reflect perspectives consistent with that alignment. We have no visibility into the data used to train it (though it's worth noting the same is true for Llama, OpenAI, and Claude). This makes some governments and enterprises uneasy.
How can you use DeepSeek?
You can try the model on DeepSeek's website or its mobile app. Alternatively, you can use ollama run deepseek-r1:32b to run the smaller version of the model locally. For that, you'll need to download and install Ollama.
Cloud providers have also jumped in. You can deploy DeepSeek's models on GCP Vertex AI, AWS Bedrock, and Azure AI Foundry, or deploy it as an NVIDIA NIM.
DeepSeek's models are certainly intriguing enough to consider adding to your AI platform toolbox alongside other open-weight models, since application developers will want to experiment with or use different models for different purposes.
Can we trust DeepSeek's reported performance numbers?
Update: in early 2026, Hugging Face published a full reproduction of DeepSeek R1 through its openR1 project, training data, code, and validation pipeline included, confirming the reported results within one to three standard deviations on benchmarks like MATH-500.
We'd also like to understand whether the model was exposed to benchmark data during training, and whether the evaluation methods used in the paper are appropriate.
That said, we have no particular reason to believe the results aren't real.
What made waves was the figure of 2.788 million GPU hours (estimated at $5.576 million) for training (see the first table in that paper). The V3 paper clarifies the assumptions behind that price point, but it also offers caveats, stating that this only represents the final training run. Given how quickly the industry jumped on the coverage of this particular family of models, we suspect this figure has been repeated out of context many times in that coverage.
DeepSeek's technical components
R1 was trained using a combination of SFT and RL on top of V3-Base. These are transformers that were heavily optimized for specific hardware/software frameworks given the constraints imposed by the environment (specifically, US government export controls on NVIDIA H100 chips). DeepSeek also used a combination of new and old techniques in interesting ways. Let's start by looking at V3-Base.
V3-Base: mixture-of-experts and training costs
V3-Base uses a robust mixture-of-experts approach, similar to Mixtral but more efficient, trained with 671 billion total parameters versus 405 billion for Llama. Both models use FP8 quantization and a 128K context window, with comparable training token counts: 14.8 trillion for V3-Base versus 15 trillion for Llama.
The key difference: the V3 paper states 2.788 million GPU hours, versus 39.3 million cumulative GPU hours for Llama 3.1 405B FP8. This is where the nuance lies, the 2.788 million hours for V3 covered only the final complete training run, whereas Llama's number is a cumulative figure. So we don't know if there's a direct comparison to be made: V3 was trained on some data generated by an as-yet-unreleased R1,should V3's training cost include R1's in that case?
R1 was built on top of V3-Base using supervised fine-tuning (SFT) and reinforcement learning (RL) to build reasoning into the model, with a long chain-of-thought pattern, then further distilled into smaller dense models, with versions based on Llama and Qwen. They also released R1-Zero, which doesn't use SFT and has certain limitations such as readability and language mixing, though it shows intriguing reasoning behaviors, probably more interesting to researchers than to end users, hence the multi-stage training with cold-start data applied before RL to overcome these issues.
V3 was then built by using data created from R1's reasoning, verification, and reflection patterns to further improve V3-Base, creating a more complete model, V3.
All these models were trained using NVIDIA H800 GPUs. These are versions of the H100 GPU made for the Chinese market, and as mentioned earlier, they are limited to comply with US export controls. Specifically, H800 chips have half the chip-to-chip interconnect speed of H100s (about 400 GB/s versus 900 GB/s on NVLink).
The cost of training R1 is reported as $5.58 million. We know they're getting this wrong, but it's unclear by how much. The calculation comes from the V3 technical report, which is the training cost for DeepSeek V3. CNN gets it right by saying the cost applied to the base model, however, they don't help people understand the difference between the two.
R1 was trained on top of V3-Base, so the cumulative cost of training R1 would definitely be more than training the base model. The numbers in table one of the V3 technical report appear to be for one complete training run, likely the final complete training run. If you were to try to reproduce the training process, you'd likely need to do more than one complete training run.
There are also conflicting reports that DeepSeek has access to 50,000 A100 GPUs, which is more in line with what OpenAI reportedly used to train GPT-4 (25,000 A100s).
If you were to rent 50,000 A100 GPUs in the US today, you'd likely pay around $1.35/GPU-hour (if you could find that many available). That's roughly $11.34 million per week. In DeepSeek's case, it seems they may have used GPUs that their backer High-Flyer had previously acquired for high-frequency trading purposes.
Diving deeper into what makes DeepSeek distinctive
There are a number of sophisticated ways DeepSeek modified the model architecture, training techniques, and data to squeeze the most out of the limited hardware it had. Let's walk through these from the bottom up.
Optimizing for the available hardware
Optimizing for the available hardware, There are two key limitations of the H800s DeepSeek had to use compared to H100s. First, they have half the GPU-to-GPU interconnect bandwidth of H100s, and second, much smaller memory: 80GB versus 188GB.
Interestingly, DeepSeek seems to have turned these limitations into an advantage. "[T]he economical training costs of DeepSeek-V3 ... [were] achieved through our optimal co-design of algorithms, frameworks, and hardware," wrote the DeepSeek team. In other words, they made decisions that would let them extract the most from what they had available.
For example, they used FP8 to significantly reduce the amount of memory required. The V3 paper says "low-precision training has emerged as a promising solution for efficient training." However, before this work, FP8 was considered effective but not very efficient; DeepSeek demonstrated how it can be used efficiently. "In this work, we introduce an FP8 mixed precision training framework and, for the first time, validate its effectiveness on an extremely large-scale model. Through the support for FP8 computation and storage, we achieve both accelerated training and reduced GPU memory usage."
They further optimized for the constrained hardware at a very low level. The V3 paper also states "we also develop efficient cross-node all-to-all communication kernels to fully utilize InfiniBand (IB) and NVLink bandwidths. Furthermore, we meticulously optimize the memory footprint, making it possible to train DeepSeek-V3 without using costly tensor parallelism. Combining these efforts, we achieve high training efficiency." That's seriously deep work to squeeze the most out of the hardware they were limited to.
Additionally, the paper talks about something we find particularly interesting. "As for the training framework, we design the DualPipe algorithm for efficient pipeline parallelism, which has fewer pipeline bubbles and hides most of the communication during training through computation-communication overlap. This overlap ensures that, as the model further scales up, as long as we maintain a constant computation-to-communication ratio, we can still employ fine-grained experts across nodes while achieving a near-zero all-to-all communication overhead." The constant computation-to-communication ratio and the near-zero all-to-all communication overhead are striking compared to the "normal" ways of scaling distributed training, which usually just mean "throw more hardware at it."
This is a clear case where necessity is the mother of invention.
The impact of reinforcement learning during additional training on benchmark performance, DeepSeek applied reinforcement learning with GRPO (Group Relative Policy Optimization) in V2 and V3. But apparently, reinforcement learning had a major impact on the reasoning model, R1,its impact on benchmark performance is notable.
By using GRPO to apply reward to the model, DeepSeek avoids using a large "critic" model; this again saves memory. However, GRPO takes a rules-based approach that, while it will work better for problems with an objective answer, like coding and math, may struggle in domains where answers are subjective or variable. It will be interesting to track the trade-offs as more people use it in different contexts.
Multi-head latent attention and the distillation/RL trade-off
Multi-head latent attention (MLA), Multi-head latent attention is a variation of multi-head attention that DeepSeek introduced in their V2 paper. According to reports, while previous multi-head attention techniques were seen as a trade-off, in the sense that you traded model quality for better scaling in large-model training, DeepSeek says MLA doesn't just enable scaling, it also improves the model. We're keen to dig deeper into this.
Distillation vs. reinforcement learning, The R1 paper features an interesting discussion of distillation versus reinforcement learning. The DeepSeek team writes that their work makes it possible to "draw two conclusions: First, distilling more powerful models into smaller ones yields excellent results, whereas smaller models relying on the large-scale RL mentioned in this paper require enormous computational power and might not even achieve the performance of distillation. Second, while distillation strategies are both economical and effective, advancing beyond the boundaries of intelligence may still require more powerful base models and larger-scale reinforcement learning."
The first conclusion is interesting and genuinely intuitive. The second is reassuring, at least, they haven't completely upended our understanding of how deep learning works in terms of significant compute requirements.
What did we get to learn from what didn't work? This is always interesting to read. What did DeepSeek try that didn't work?
First, using a process reward model (PRM) to guide reinforcement learning wasn't viable at scale, though it could still be used for re-ranking top-N responses. Second, Monte Carlo tree search (MCTS), which was used by AlphaGo and AlphaZero, doesn't scale to general reasoning tasks because the problem space isn't as "constrained" as chess or even Go. Remember when, less than a decade ago, the space of Go was considered too complex to be computationally feasible? Now it's "constrained."
Other interesting things, There's obviously a huge amount of interesting material we could comment on. However, a few things are worth flagging:
A very impressive coding benchmark. Additional training plus inference-time scaling appears to be a viable strategy for building a highly efficient model.
What's next?
Breaking the circularity of benchmarks and models, After the release of every new, better model, we ask ourselves whether it was exposed to benchmark data at training time. "Did it simply study for the exam, or did it actually learn the subject?"
This is due to the perverse circularity of benchmark datasets; it's an endless spiral of misleading hype. You create a good benchmark dataset, the next model games it to win, gets hype, then you need to create another "fair" benchmark ... this adds value until the next model games it, and so on. Humanity's Last Exam will only be what it says until the next model is released.
In other words, when an LLM confidently generates correct answers on current benchmarks, that's great if its application was also on real-world data of similar complexity. Conversely, when the LLM fails on a new benchmark (or the domain it's being applied to), it's usually because it's confident in answers that are wrong. That's because the new benchmark data has a complexity it didn't encounter at training time.
This cycle needs to stop, and we need better, more generic evaluation mechanisms and more informative metrics that don't depend on new benchmarks every few weeks.
Replicating DeepSeek R1's results
What was left to do in early 2025 is now done: in early 2026, Hugging Face's openR1 project published a full reproduction of R1, training included, independently validating its findings. Here's what that reproduction required:
Three ingredients were necessary: GPUs (2,048, which isn't a huge number, for roughly $5.5 million per training run), training code, which DeepSeek hasn't open-sourced, and training data, arguably the most significant gap to bridge.
DeepSeek is unlikely to release its full training dataset, just as OpenAI or Anthropic won't release theirs. As far as we've been able to find, DeepSeek hasn't published samples of the data used to train the long chain of thought. So the open-source community has started creating its own datasets; OpenThoughts is one example.
Meanwhile, researchers at Berkeley claim to have reproduced R1-Zero for under $30...
Since then, DeepSeek has evolved its model family beyond V3 and R1, with a new generation that carries the same algorithm/hardware co-design principles to a larger scale, confirming that the architectural choices described here weren't a one-off but a repeatable method.
...I think I need to take a nap.
Thanks to our colleagues Shayan Mohanty, Ben O'Mahony, Chris Kramer, Sahger Lad, Prathamesh Kalamkar, Lauris Jullien, Emily Gorcenski, Karrtik Iyer, Runyan Tan, Parag Mahajani, and Andy Yates, who all contributed to this article.
STAY POSTED
Get our next analyses and field notes straight to your inbox.
