What DeepSeek-R1 changes in a model evaluation

DeepSeek published the R1 and R1-Zero weights in January, along with a set of smaller distilled models built on Qwen and Llama. The release repository reports results across maths, coding and reasoning benchmarks, and if you've got the infrastructure you can run the models on your own systems. So it's worth testing. I'd just add it to the evaluation you already have as one more candidate, because once you change the tasks, prompts or scoring rules for R1 you can't really compare it with anything.

That evaluation should come from the actual product. For a support workflow I'd want incomplete questions, conflicting source documents, requests that need a refusal and answers that need precise citations. For coding, I'd use repository tasks with a known starting commit, tests and an expected patch boundary. Keep the system instructions, retrieval results, tool definitions and output schema the same across models. If R1 needs a different prompt to do well, that's fine, but record it as its own configuration rather than quietly tuning one candidate until it wins.

The published numbers tell you how R1 did under DeepSeek's setup. The repository says its sampling evaluations used a temperature of 0.6, top-p of 0.95 and 64 responses per query to estimate pass@1. Your production request probably generates once, under a tighter time limit, with retrieved context and tools in the loop. That's why I'd keep vendor-reported results separate from anything you reproduced yourself, and write down the exact model identifier, serving provider, quantisation, prompt template and generation settings for every run. Otherwise a later claim that "R1 scored 90" doesn't point at any test you can check.

Reasoning models can spend a lot more tokens and time before they answer. Sometimes that's worth it, but it needs to show up in the results: end-to-end latency at realistic concurrency (percentiles, not one average), token counts and cost under the pricing you can actually get. If you're self-hosting, the size matters too. DeepSeek describes the main release as a mixture-of-experts model with 671 billion total parameters and 37 billion activated per token, and that doesn't give it the memory footprint of a 37 billion parameter dense model. The distilled versions, between 1.5 billion and 70 billion parameters, are more practical to run locally, but treat each one as its own model with its own failure patterns.

Before anything ships, check the licence and the security boundary for the exact thing you're deploying. DeepSeek applies the MIT licence to its code and weights and allows commercial use, while the distilled Qwen and Llama variants also carry conditions from their base models. Local weights don't remove the need to test prompt injection from retrieved text or malformed tool calls either. Then compare candidates task by task, since one model might be better at multi-step analysis while another is quick and good enough for classification, and that could justify routing or show the extra complexity isn't worth it.

So DeepSeek-R1 doesn't need an evaluation of its own. It goes into the one you already have as another candidate, on the same product tasks and settings, with DeepSeek's benchmark numbers kept to one side and its latency, token use and hosting cost recorded next to task success. What I'd be looking for in the task-by-task results is whether R1 improves enough of your real workload to pay for that latency and deployment work. If it does, route those tasks to it, and if it doesn't, the model you've already got stays in place.