What I am testing with o1-preview

OpenAI released o1-preview last week as an early preview of a model that spends more time working through a problem before it responds. The launch material points at science, coding and maths, which is fine, but I still need cases from my own kind of work to know whether that extra processing helps. I'd start small, with tasks where I can tell if the answer is right without relying on the model's explanation, and compare the final work, review effort, latency and cost against what I'm using now.

That means picking tasks with a check that sits outside the model. Code can run against tests, a schedule can be checked against its constraints, and a calculation can be recomputed by a separate program. Business work has plenty of these too: applying a documented rule set, finding an inconsistency between records, writing a query for a known schema. I'd avoid starting with something broad like "write a better strategy", because reviewers might prefer one answer without being able to say whether the extra reasoning had anything to do with it.

Each case needs the input, the permitted context, the expected properties, the right answer where there is one, and a way to verify it. A handful of polished prompts will mostly measure how well I wrote the prompts, so the set should include the routine cases plus the awkward ones where a plausible answer is still wrong. For a constraint problem I'd vary the number of constraints and include one impossible case and one with missing information. I'd also keep the cases fixed while comparing models. If o1-preview gets a rewritten prompt or extra context, that's a separate configuration, otherwise I can't tell the model's behaviour apart from changes I made around it.

Where I can, I'd score with deterministic checks: run generated code in an isolated environment, parse structured output against a schema, recalculate totals, confirm every cited source actually exists in the supplied material. For human review I'd hide which configuration produced each answer and ask reviewers to note the actual defect. "Missed the cancellation condition" tells me much more than "three out of five". A well-written explanation shouldn't rescue a wrong answer, and I want to see what happens when there's no valid answer at all. Does it name the missing information, or invent a way around it? Does it do the same thing on the next run?

Then there's time and money. A slower answer might be fine for an overnight analysis and useless inside an interactive form, so I'd capture response times at the percentiles users would actually hit, plus token usage, retries and rate limits (it's a preview, so throughput matters too). Review time goes in the same record. A more expensive call can still work out cheaper if it reliably cuts down manual repair, and a longer answer can take longer to review even when it sounds more thorough.

Before running anything, I'd write down the correctness threshold, acceptable latency, review budget and any failure that should stop the test. Then I'd run the same harness against o1-preview and the current baseline, pinning the tested model identifier where the API allows it. What I'm testing with o1-preview comes down to whether the extra reasoning produces answers that pass checks outside the model, at a latency and review cost my work can carry. So the defects and review time from that run are what I'd look at before changing anything.