When I would test an open-weight model

Meta released the first Llama 3 models on 18 April, in 8 billion and 70 billion parameter versions, so they can now go into a product evaluation alongside the hosted options. I'd still want a specific reason to self-host before I started, though. Maybe inference has to stay inside a controlled environment, or you need to tune behaviour for a narrow task, or you've got predictable high volume, or the product can't depend on one hosted endpoint. Each of those leads to a different test. If the reason is just "more control", I'd break that down into data location, access, uptime, output behaviour and release control, because having the weights doesn't sort out any of them on its own. If there's no real constraint, a hosted service might stay the simpler choice, and I'd let the evaluation land there.

Before touching infrastructure, I'd build a small evaluation set from the actual job. Normal inputs, hard ones, missing evidence, long material, odd formatting, and cases where the right answer is to ask for clarification. Strip out private data or run it somewhere approved for it, and write down what acceptable output looks like for each case. Then run the open model and a hosted baseline with equivalent instructions and the same retrieval material. Prompts might need adjusting because the formats differ, but don't quietly give one of them more context or more attempts, and keep the configuration with each result so you can rerun it later.

The Llama 3 release includes pretrained and instruction-tuned variants, and size and format change the hardware you need, the latency and the behaviour. Testing a remote demo tells you very little about the version you'd actually run. So I'd pick a realistic serving setup and record the weights, precision, quantisation if any, inference engine, hardware, context length and generation settings. Then test under concurrency, not one prompt at a time: time to first token, total completion time, queueing under expected load and memory use, with long and short inputs. What happens when the model process restarts, or a request goes over the context limit?

Running the weights is only one piece. You still need an API boundary, identity checks, resource limits, monitoring, a release process and somewhere to record where the weights came from, their licence and checksum. GPU memory and error rates tell you the service is up, but they won't tell you a prompt change made answers worse, so I'd keep the task evaluation around for every change.

On cost, comparing token prices with an hourly GPU price leaves out most of it. Self-hosting spreads cost across hardware, idle capacity, power, engineering time and incident response, and someone has to patch the host and manage capacity. Hosted options have their own risks too, like rate limits and provider changes, so both go in the same cost and risk record. It doesn't have to be one model for the whole product either. A local model could handle routine internal classification while a hosted one does occasional complex drafting, if the data rules allow it.

When would I actually test an open-weight model like Llama 3? I'd do it when there's a specific reason to self-host, and I'd judge it on the same cases as a hosted baseline, with the serving and running costs counted, rather than on a remote demo. I'd pick one task, gather ordinary and failure cases, and price one serving configuration you could actually deploy. If those numbers justify a local trial, then I'd package and run the weights, and if they don't, the hosted option stays.