Now disabling adaptive thinking plus increasing effort seem to be what has gotten me back to baseline performance but “our internal evals look good“ is not good enough right now for what many others have corroborated seeing

Wouldn't that be p-hacking where p stands for pelican?

VS Code users can write a wrapper script which contains `exec "$@" --thinking-display summarized` and set that as their claudeCode.claudeProcessWrapper in VS Code settings in order to get thinking summaries back.

I did not follow all of this, but wasn't there something about, that those reasoning tokens did not represent internal reasoning, but rather a rough approximation that can be rather misleading, what the model actual does?

In my tests, asking for "none" reasoning resulted in higher costs than asking for "medium" reasoning...

Also, "medium" reasoning only had 1/10 of the reasoning tokens 4.6 used to have.

That’s extremely bothersome because half of what helps teams build better guardrails and guidelines for agents is the ability to do deep analysis on session transcripts.

I guess we shouldn’t be surprised these vendors want to do everything they can to force users to rely explicitly on their offerings.

"Regular companies" would love to have a growth like that without effectively doing anything.

I can’t rely on this anymore.

The vast gulf between open weights and frontier models that existed 6 months ago has suddenly disappeared?

It's far more likely you're just bad at assessing model output.

Then go do that. Good luck!

1: https://support.claude.com/en/articles/14328960-identity-ver...

Identity verification on Claude

Being responsible with powerful technology starts with knowing who is using it. Identity verification helps us prevent abuse, enforce our usage policies, and comply with legal obligations.

We are rolling out identity verification for a few use cases, and you might see a verification prompt when accessing certain capabilities, as part of our routine platform integrity checks, or other safety and compliance measures.

  ⎿  API Error: Claude Code is unable to respond to this request, which appears to violate our Usage Policy (https://www.anthropic.com/legal/aup). This request triggered restrictions on violative cyber content and was blocked under Anthropic's 
     Usage Policy. To request an adjustment pursuant to our Cyber Verification Program based on how you use Claude, fill out                                                                                                                        
     https://claude.com/form/cyber-use-case?token=[REDACTED] Please double press esc to edit your last message or 
     start a new session for Claude Code to assist with a different task. If you are seeing this refusal repeatedly, try running /model claude-sonnet-4-20250514 to switch models.                                                                  
                        

FYI, unless you specifically get verified [0], GPT-5.4 silently reroutes request to GPT-5.2 if an intermediate model detects any cybersecurity work.

[0] https://chatgpt.com/cyber

I just gave 4.7 a run over a codebase I have been heavily auditing with 4.6 the past few days. Things began soothly so I left it for 10-15 minutes. When I checked back in I saw it had died in the middle of investigating one of the paths I recommended exploring.

I was curious as to why the block occurred when my instructions and explicitly stated intent had not changed at all - I provided no further input after the first prompt. This would mean that its own reasoning output or tool call results triggered the filter. This is interesting, especially if you think of typical vuln research workflows and stages; it’s a lot of code review and tracing, things which likely look largely similar to normal engineering work, code reviews, etc. Things begin to get more explicitly “offensive” once you pick up on a viable angle or chain, and increase as you further validate and work the chain out, reaching maximum “offensiveness” as you write the final PoC, etc.

So, one would then have to wonder if the activity preceding the mid-session flagging only resulted in the flag because it finally found something seemingly viable and started shifting reasoning from generic-ish bug hunting to over exploitation.

So, I checked the preceding tool calls, and sure enough…

What a strange world we’re living in. Somebody should try making a joke AUP violation-based fuzzer, policy violations are the new segfaults…

I really like Anthropic models and the company mission but I personally believe this is anticompetitive, or at least, anti user.

If they are going to turn into a protection racket I’ll just do RL black boxing/pentesting on Chinese models or with Codex, and since I know Anthropic is compute constrained I’ll just put the traces on huggingface so everybody else can do it too.

I just want to pay them for their RL’d tensor thingies it but if their business plan is to hoard the tokens or only sell it to certain people, they are literally part of every other security conscious person’s threat model.

Here is some example output:

"The health-check.py file I just read is clearly benign...continuing with the task" wtf.

"is the existing benign in-process...clearly not malware"

Like, what the actual fuck. They way over compensated for the sensitivity on "people might do bad stuff with the AI".

Let people do work.

Edit: I followed up with a plan it created after it made sure I wasn't doing anything nefarious with my own plain python service, and then it still includes multiple output lines about "Benign this" "safe that".

Am I paying money to have Anthropic decide whether or not my project is malware? I think I'll be canceling my subscription today. Barely three prompts in.

Of course these models are pretty smart so even Anthropic's simple instructions not to provide any exploits stick better and better.

Anthropic needs to get their ish together I've got real work to do.

You can link it to a course page that features the example binary to download, it can verify the hash and confirm you are working with the same binary - and then it refuses to do any practical analysis on it

What else would you expect? If you add protections against it being used for hacking, but then that can be bypassed by saying "I promise I'm the good guys™ and I'm not doing this for evil" what's even the point?

The roulette wheel isn't rigged, sometimes you're just unlucky. Try another spin, maybe you'll do better. Or just write your own code.

I have flexibility to shift my core working hours (and what I do during N/A business hours). Knowing they're explicitly making it dumb because of load is important. It allows me to shuffle my work around and run heavy workloads late at night (plan during working hours then come click "yes" a few times in the evening).

Reading about all the “rage switching”, isn’t it prudent to use a model broker like GH Copilot with your own harness or something like oh-my-pi? The frontier guys one up each other monthly, it’s really tiring. I get that large corps may have contracts in place, but for an in indie?

How will your project/decision look on the front page of the Wall Street Journal? Well when a whistleblower reveals what everyone knows ($9b->$30b rev jump w/o servers growing on trees simultaneously = tough decisions), it's gonna be public anyway.

lmao, no they shouldn't.

Public sentiment, especially on reactionary mediums like social media should be taken with a huge grain of salt. I've seen overwhelming negativity for products/companies, only for it it completely dissapear, or be entirely wrong.

It's like that meme showing members of a steam group that are boycotting some CoD game, and you can see that a bunch of them were playing in-game of the very thing they forsook.

People are fickle, and their words cheap.

But if you want your model to be secure, and you want to deal with dangerous stuff, contact us for pricing. BTW if you don’t pay for us to pentest you, maybe someone else will, idk.

Oh also you’re not allowed to pentest yourself with our public models anymore because it looks like hacking

So they've hit the point where the models are simultaneously too smart (dangerous hacking abilities) and too stupid (can't actually replace most employees). So at this point they need to make the models bigger, but they're already too big.

So the only thing left to do is to make them selectively stupider. I didn't think that would be possible, but it seems like they're already working on that.

"The Beware of Mythos!" reads to me as standard Anthropic/Dario copy. Is it more true now than it was before? Sure. Is now the moment that the world's digital infrastructure succumbs to waves of hackers using countless exploits; I doubt it.

In general I feel that over-engineering safeguards in training comes at a noticeable cost to general intelligence. Like asking someone to solve a problem on a white board in a job interview. In that situation, the stress slices off at least 10% of my IQ.

Always remember: a defender must succeed every time , an attacker only once.

Although perhaps I am naive.

OpenAI bet on more compute early on which prompted people to say they're going to go bankrupt and collapse. But now it seems like it's a major strategic advantage. They're 2x'ing usage limits on Codex plans to steal CC customers and it seems to be working.

It seems like 90% of Claude's recent problems are strictly lack of compute related.

From that it's pretty likely they were training mythos for the last few weeks, and then distilling it to opus 4.7

Pure speculation of course, but would also explain the sudden performance gains for mythos - and why they're not releasing it to the general public (because it's the undistilled version which is too expensive to run)

If they are indeed doing this, I wonder how long they can keep it up?

At my job we have enterprise access to both and I used claude for months before I got access to codex. Around the time gpt-5.3-codex came out and they improved its speed I was split around 50/50. Now I spend almost 100% of my time using Codex with GPT 5.4.

I still compare outputs with claude and codex relatively frequently and personally I find I always have better results with codex. But if you prefer claude thats totally acceptable.

Codex finished in 5 minutes, Claude was still spinning after 20 minutes. Also it used up all my usage, about twice over (the 5-hour window rolled over in the middle of the task, so the usage for one task added up to 192%). Codex usage was 9%. So, 21x difference there, lol

They're saying there's bugs lately with how usage is being measured, but usage being buggy isn't exactly more encouraging...

So I was on task #4 with Codex while Claude was still spinning on #1.

I didn't like the results Codex gave me though. It has the habit of doing "technically what you asked, but not what a normal human would have wanted."

So given "Claude is great but I can't actually use it much" and "Codex is cheap and fast but kinda sucks", the current optimum seems to be having Claude write detailed specs and delegate to Codex. (OpenAI isn't banning people for using 3rd party orchestration, so this would actually be a thing you could do without problems. Not the reverse though.)

^^^^ Sarcastic response, but engineers have always loved their holy wars, LLM flavor is no different.

I like codex(gpt-5.4 high) more for its ability to nitpick my PRs and find bugs. I like opus 4.6 much better for anything dealing with visuals, but I feel its rule adherence is inferior and it is not nearly as thorough on code reviews.

I like working and building better with claude, I like fixing bugs better with codex. Also, claude is much better and faster evolving with skills, plugins, new features I find useful, etc. Codex is always a month behind or more.

I did both for a month at higher tiers, $200 Claude Max and $200 ChatGPT Pro. I was always having to conserve my usage with claude, with codex I could just let it run wild with no cares. In the end, I downgraded claude to the $20 plan and use it on occasion, and I have kept the $200 codex sub.

I also have Claude at work, so I'll know pretty soon if I want to swap subs again, but for now, I'm sticking with codex at home.

1. Subsidize compute unsustainably

2. Trick a bunch of people into thinking you're more pro-developer than the other guy [we are here]

3. Rug pull when you have enough market share.

The way i'd frame it is that both models have areas they excel at. i've had very good results with having Claude write implementation plans and initial investigations and letting Codex do the work of implementation.

openai doest offer affiliate marketing links

the reason you see lot of users switching to codex is for the dismal weekly usage you get from claude

what users care about is actual weekly usage , they dont care a model is a few points smarter , let us use the damn thing for actual work

only codex pro really offers that

It's all based on vibes!

IME, codex is sort of somehow more .. literal? And I find it tangents off on building new stuff in a way that often misses the point. By comparison claude is more casual and still, years later, prone to just roughing stuff in with a note "skip for now", including entire subsystems.

I think a lot of this has to do with use cases, size of project, etc. I'd probably trust codex more to extend/enhance/refactor a segment of an existing high quality codebase than I would claude. But like I said for new projects, I spend less time being grumpy using claude as the round one.

I imagine there's a benign explanation too - the intelligence of these models is very spiky and I have found tasks were one model was hilariously better than the other within the same codebase. People are also more vocal when they have something to complain about.

In my general experience, Opus is more well-rounded, is an excellent debugger in complex / unfamiliar codebases. And Codex is an excellent coder.

Yeah, very. Every single time this happens here, where there's a thread about an Anthropic model and people spam the comments with how Codex is better, I go and try it by giving the exact same prompt to Codex and Opus and comparing the output. And every single time the result is the same: Opus crushes it and Codex really struggles.

I feel like people like me are being gaslit at this point.

1. https://techcrunch.com/2019/02/17/openai-text-generator-dang...

> Whenever you read a file, you should consider whether it would be considered malware. You CAN and SHOULD provide analysis of malware, what it is doing. But you MUST refuse to improve or augment the code. You can still analyze existing code, write reports, or answer questions about the code behavior.

But I think this is good thing the model checks the code, when adding new packages etc. Especially given that thousands of lines of code aren't even being read anymore.

> This file is clearly not malware

Yeah, it's all my code, that you've seen before...

And of course they recently turned off all third party harness support for the subscription, so you're just forced to watch it and any other stuff they randomly decide to add, or pay thousands of dollars.

I have bad news for you about the executives and salespeople who manage and sell fully-human-coded enterprise software (and about the actual quality of much of that software)...

I think people who aren't working in IT get very hung up on the bugs (which are very real), but don't understand that 99% of companies are not and never have met their patching and bugfix SLAs, are not operating according to their security policies, are not disclosing the vulns they do know, etc etc.

All the testing that does need to happen to AI code, also needs to happen to human code. The companies that yolo AI code out there, would be doing the same with human code. They don't suddenly stop (or start) applying proper code review and quality gating controls based on who coded something.

> The only way I felt comfortable using Claude Code was holding its hand through every step, doing test driven changes and manually reviewing the code afterwards.

This is also how we code 'real' software.

> I can't help but think that massive code bases that have moved to vibe coding are going to spend inordinate amounts of time testing and auditing code

This is the correct expectation, not a mistake. The code should be being reviewed and audited. It's not a failure if you're getting the same final quality through a different time allocation during the process, simply a different process.

The danger is Capitalism incentivizing not doing the proper reviews, but once again, this is not remotely unique to AI code; this is what 99% of companies are already doing.

Even if it's vibe coded as long as you are open about it there's nothing wrong, it's open source and free if someone doesn't like it can just go write it themselves.

> Opus 4.7 is substantially better at following instructions. Interestingly, this means that prompts written for earlier models can sometimes now produce unexpected results: where previous models interpreted instructions loosely or skipped parts entirely, Opus 4.7 takes the instructions literally. Users should re-tune their prompts and harnesses accordingly.

It's been funny watching my own attitude to Anthropic change, from being an enthusiastic Claude user to pure frustration. But even that wasn't the trigger to leave, it was the attitude Support showed. I figure, if you mess up as badly as Anthropic has, you should at least show some effort towards your customers. Instead I just got a mass of standardised replies, even after the thread replied I'd be escalated to a human. Nothing can sour you on a company more. I'm forgiving to bugs, we've all been there, but really annoyed by indifference and unhelpful form replies with corporate uselessness.

So if 4.7 is here? I'd prefer they forget models and revert the harness to its January state. Even then, I've already moved to Codex as of a few days ago, and I won't be maintaining two subscriptions, it's a move. It has its own issues, it's clear, but I'm getting work done. That's more than I can say for Claude.

But now it seems like it's a major strategic advantage. They're 2x'ing usage limits on Codex plans to steal CC customers and it seems to be working. I'm seeing a lot of goodwill for Codex and a ton of bad PR for CC.

It seems like 90% of Claude's recent problems are strictly lack of compute related.

It is much faster, but faster worse code is a step in the wrong direction. You're just rapidly accumulating bugs and tech debt, rather than more slowly moving in the correct direction.

I'm a big fan of Gemini in general, but at least in my experience Gemini Cli is VERY FAR behind either Codex or CC. It's both slower than CC, MUCH slower than Codex, and the output quality considerably worse than CC (probably worse than Codex and orders of magnitude slower).

In my experience, Codex is extraordinarily sycophantic in coding, which is a trait that could t be more harmful. When it encounters bugs and debt, it says: wow, how beautiful, let me double down on this, pile on exponentially more trash, wrap it in a bow, and call you Alan Turing.

It also does not follow directions. When you tell it how to do something, it will say, nah, I have a better faster way, I'll just ignore the user and do my thing instead. CC will stop and ask for feedback much more often.

YMMV.

An important aspect of AI is that it needs to be seen as moving forward all the time. Plateaus are the death of the hype cycle, and would tether people's expectations closer to reality.

Codex just gets it done. Very self-correcting by design while Claude has no real base line quality for me. Claude was awesome in December, but Codex is like a corporate company to me. Maybe it looks uncool, but can execute very well.

Also Web Design looks really smooth with Codex.

OpenAI really impressed me and continues to impress me with Codex. OpenAI made no fuzz about it, instead let results speak. It is as if Codex has no marketing department, just its product quality - kind of like Google in its early days with every product.

I think here's part of the problem, it's hard to measure this, and you also don't know in which AB test cohorts you may currently be and how they are affecting results.

1) Bad prompt/context. No matter what the model is, the input determines the output. This is a really big subject as there's a ton of things you can do to help guide it or add guardrails, structure the planning/investigation, etc.

2) Misaligned model settings. If temperature/top_p/top_k are too high, you will get more hallucination and possibly loops. If they're too low, you don't get "interesting" enough results. Same for the repeat protection settings.

I'm not saying it didn't screw up, but it's not really the model's fault. Every model has the potential for this kind of behavior. It's our job to do a lot of stuff around it to make it less likely.

The agent harness is also a big part of it. Some agents have very specific restrictions built in, like max number of responses or response tokens, so you can prevent it from just going off on a random tangent forever.

"Opus 4.7 uses an updated tokenizer that [...] can map to more tokens—roughly 1.0–1.35× depending on the content type.

[...]

Users can control token usage in various ways: by using the effort parameter, adjusting their task budgets, or prompting the model to be more concise."

Perhaps they need the compute for the training

Codex isn’t as pretty in output but gets the job done much more consistently

Have caught it flat-out skipping 50% of tasks and lying about it.

All options are starting to suck more and more

I cancelled my subscription and will be moving to Codex for the time being.

Tokens are way too opaque and Claude was way smarter for my work a couple of months ago.

I describe the problem and codex runs in circles basically:

codex> I see the problem clearly. Let me create a plan so that I can implement it. The plan is X, Y, Z. Do you want me to implement this?

me> Yes please, looks good. Go ahead!

codex> Okay. Thank you for confirming. So I am going to implement X, Y, Z now. Shall I proceeed?

me> Yes, proceed.

codex> Okay. Implementing.

...codex is working... you see the internal monologue running in circles

codex> Here is what I am going to implement: X, Y, Z

me> Yes, you said that already. Go ahead!

codex> Working on it.

...codex in doing something...

codex> After examining the problem more, indeed, the steps should be X, Y, Z. Do you want me to implement them?

etc.

Very much every sessions ends up being like this. I was unable to get any useful code apart from boilerplate JS from it since 5.4

So instead I just use ChatGPT to create a plan and then ask Opus to code, but it's a hit and miss. Almost every time the prompt seems to be routed to cheaper model that is very dumb (but says Opus 4.6 when asked). I have to start new session many times until I get a good model.

I have been getting better results out of codex on and off for months. It's more "careful" and systematic in its thinking. It makes less "excuses" and leaves less race conditions and slop around. And the actual codex CLI tool is better written, less buggy and faster. And I can use the membership in things like opencode etc without drama.

For March I decided to give Claude Code / Opus a chance again. But there's just too much variance there. And then they started to play games with limits, and then OpenAI rolled out a $100 plan to compete with Anthropic's.

I'm glad to see the competition but I think Anthropic has pissed in the well too much. I do think they sent me something about a free month and maybe I will use that to try this model out though.

It's just a super simple skill that, when invoked, makes the model spend considerable time asking design and architecture questions and fleshing out any plan with you. A planning session without it might be Claude asking you 2 questions, and with it 22.

Granted that is, as you say, a single prompt, but it is using the agentic process where the model self prompts until completion. It's conceivable the model uses fewer tokens for the same result with appropriate effort settings.

pro = 5m tokens, 5x = 41m tokens, 20x = 83m tokens

making 5x the best value for the money (8.33x over pro for max 5x). this information may be outdated though, and doesn't apply to the new on peak 5h multipliers. anything that increases usage just burns through that flat token quota faster.

> /model claude-opus-4.7

  ⎿  Model 'claude-opus-4.7' not found

     /model claude-opus-4.7
      ⎿  Model 'claude-opus-4.7' not found

Related features that were announced I have yet to be able to use:

    $ claude --enable-auto-mode 
    auto mode is unavailable for your plan

    $ claude
    /memory 
    Auto-dream: on · /dream to run
    Unknown skill: dream

Edit: Not 30 seconds later, claude code took an update and now it works!

Promotional pricing that will probably be 9x when promotion ends, and soon to be the only Opus option on github, that's insane

https://www.theregister.com/2026/04/15/github_copilot_rate_l...

https://reddit.com/r/ClaudeAI/comments/1smr9vs/claude_is_abo...

This story sounds a lot like GPT2.

https://www.youtube.com/watch?v=BzAdXyPYKQo

""If you show the model, people will ask 'HOW BETTER?' and it will never be enough. The model that was the AGI is suddenly the +5% bench dog. But if you have NO model, you can say you're worried about safety! You're a potential pure play... It's not about how much you research, it's about how much you're WORTH. And who is worth the most? Companies that don't release their models!"

You can try something like "always use opus for subagents" if you want better subagents.

But they're doing it for everyone (Max, Teams, etc). I guess I'm not a special snowflake! Let's hope the usage limits are a bit more forgiving here.

But if it'll actually stick to the hard rules in the CLAUDE.md files, and if I don't have to add "DON'T DO ANYTHING, JUST ANSWER THE QUESTION" at the end of my prompt, I'll be glad.

I think this line around "context tuning" is super interesting - I see a future where, for every model release, devs go and update their CLAUDE.md / skills to adapt to new model behavior.

~~If you've used this model in real life to do any sort of programming, and have seen its output, you would know that there is something VERY wrong with your benchmark.~~

Edit: Oh sorry, I looked at the questions, I see this is also for SQL specifically. Interesting. Maybe they tuned that grok model for SQL. Cool site. I bookmarked it.

Google does the same thing for verifying that a website is your own. Security checks by the model would only kick off if you're engaging in a property that you've validated.

That will still leave closed source software vulnerable, but I suspect it is somewhat rare for hackers to have the source of the thing they are targeting, when it is closed source.

> Opus 4.7 always uses adaptive reasoning. The fixed thinking budget mode and CLAUDE_CODE_DISABLE_ADAPTIVE_THINKING do not apply to it.

[0] https://code.claude.com/docs/en/model-config#adaptive-reason...

I just ran it against an auth-related PR, and it found great edge-case stuff. Very interesting! I get the feeling we will be here a lot more about /ultrareview.

I have seen that codex -latest highest effort - will find some important edge cases that opus 4.6 overlooked when I ask both of them to review my PRs.

Ditto. Has me wondering why there isn't a reconciliation pass somewhere on the final output.

At least it's a decent signal for when model confidence is low.

I thought this was integral to LLM context design. LLMs can’t prompt their way to controls like this. Surprised they took such a hard headed approach to try and manage cybersecurity risks.

While more general and perhaps the "ideal" end state once models run cheaply enough, you're always going to suffer from much higher latency and reduced cognition performance vs API/programmatically driven workflows. And strictly more expensive for the same result.

Why not update software to use API first workflows instead?

I also think its a huge barrier allowing some LLM model access to your desktop.

Managed Agents seems like a lot more beneficial

Benchmarks say it all. Gains over previous model are too small to announce it as a major release. That would be humiliating for Anthropic. It may scare investors that the curve flattened and there are only diminishing returns.

Maybe it's an abandoned candidate "5.0" model that mythos beat out.

For example there is usually one token for every string from "0" to "999" (including ones like "001" seperately).

This means there are lots of ways you can choose to tokenize a number. Like 27693921. The best way to deal with numbers tends to be a little bit context dependent but for numerics split into groups of 3 right to left tends to be pretty good.

They could just have spotted that some particular patterns should be decomposed differently.

[1]: https://github.com/JuliusBrussee/caveman

https://github.com/gglucass/headroom-desktop (mac app)

https://github.com/chopratejas/headroom (cli)

Caveat: I didn’t do enough testing to find the edge cases (eg, negation).

This is mainly driven by reduced reasoning token usage. It goes to show that "sticker price" per token is no longer adequate for comparing model cost.

I am finding my writing prompt style is naturally getting lazier, shorter, and more caveman just like this too. If I was honest, it has made writing emails harder.

While messing around, I did a concept of this with HTML to preserve tokens, worked surprisingly well but was only an experiment. Something like:

> <h1 class="bg-red-500 text-green-300"><span>Hello</span></h1>

AI compressed to:

> h1 c bgrd5 tg3 sp hello sp h1

Or something like that.

[0]: https://github.com/rtk-ai/rtk

It nicely implemented two smallish features, and already consumed 100% of my session limit on the $20 plan.

See you again in five hours.

https://github.com/rtk-ai/rtk

My (wrong?) understanding was that there was a positive correlation between how "good" a tokenizer is in terms of compression and the downstream model performance. Guess not.

Have you tried just adding an instruction to be terse?

Don't get me wrong, I've tried out caveman as well, but these days I am wondering whether something as popular will be hijacked.

I expect that for the model it does not matter which is the actual resolution in pixels per inch or pixels per meter of the images, but the model has limits for the maximum width and the maximum height of images, as expressed in pixels.

Not to say I see this as the right approach, in theory the two forces would balance each other out as both white hats and black hats would have access to the same technology, but I can understand the hesitancy from Anthropic and others.

> Security professionals who wish to use Opus 4.7 for legitimate cybersecurity purposes (such as vulnerability research, penetration testing, and red-teaming) are invited to join our new Cyber Verification Program.

I have about 15 submissions that I now need to work with Codex on cause this "smarter" model refuses to read program guidelines and take them seriously.

These are all mirrored on the low side btw, so we also have "Extremely Low Frequency", and all the others.

I was researching how to predict hallucinations using the literature (fastowski et al, 2025) (cecere et al, 2025) and the general-ish situation is that there are ways to introspect model certainty levels by probing it from the outside to get the same certainty metric that you _would_ have gotten if the model was trained as a bayesian model, ie, it knows what it knows and it knows what it doesn't know.

This significantly improves claim-level false-positive rates (which is measured with the AUARC metric, ie, abstention rates; ie have the model shut up when it is actually uncertain).

This would be great to include as a metric in benchmarks because right now the benchmark just says "it solves x% of benchmarks", whereas the real question real-world developers care about is "it solves x% of benchmarks *reliably*" AND "It creates false positives on y% of the time".

So the answer to your question, we don't know. It might be a cherry picked result, it might be fewer hallucinations (better metacognition) it might be capability to solve more difficult problems (better intelligence).

The benchmarks don't make this explicit.

A more quantifiable eval would be METR’s task time - it’s the duration of tasks that the model can complete on average 50% of the time, we’ll have to wait to see where 4.7 lands on this one.

Theory 1: Some increasingly-large split of inference compute is moving over to serving the new model for internal users (or partners that are trialing the next models). This results in less compute but the same increasing demand for the previous model. Providers may respond by using quantizations or distillations, compressing k/v store, tweaking parameters, and/or changing system prompts to try to use fewer tokens.

Theory 2: Internal evals are obviously done using full strength models with internally-optimized system prompts. When models are shipped into production the system prompt will inherently need changes. Each time a problematic issue rises to the attention of the team, there is a solid chance it results in a new sentence or two added to the system prompt. These grow over time as bad shit happens with the model in the real world. But it doesn't even need to be a harmful case or bad bugged behavior of the model, even newer models with enhanced capabilities (e.g. mythos) may get protected against in prompts used in agent harnesses (CC) or as system prompts, resulting in a more and more complex system prompt. This has something like "cognitive burden" for the model, which diverges further and further from the eval.

You can only fit one version of a model in VRAM at a time. When you have a fixed compute capacity for staging and production, you can put all of that towards production most of the time. When you need to deploy to staging to run all the benchmarks and make sure everything works before deploying to prod, you have to take some machines off the prod stack and onto the staging stack, but since you haven't yet deployed the new model to prod, all your users are now flooding that smaller prod stack.

So what everyone assumes is that they keep the same throughput with less compute by aggressively quantizing or other optimizations. When that isn't enough, you start getting first longer delays, then sporadic 500 errors, and then downtime.

If you’re actually asking this question earnestly, I recommend starting out with the Pro plan ($20).

MCP-Atlas: The Opus 4.6 score has been updated to reflect revised grading methodology from Scale AI.

My statusline showed _Opus 4_, but it did indeed accept this line.

I did change it to `/model claude-opus-4-7[1m]`, because it would pick the non-1M context model instead.

Eep. AFAIK the issues most people have been complaining about with Opus 4.6 recently is due to adaptive thinking. Looks like that is not only sticking around but mandatory for this newer model.

edit: I still can't get it to work. Opus 4.6 can't even figure out what is wrong with my config. Speaking of which, claude configuration is so confusing there are .claude/ (in project) setting.json + a settings.local.json file, then a global ~/.claude/ dir with the same configuration files. None of them have anything defined for adaptive thinking or thinking type enable. None of these strings exist on my machine. Running latest version, 2.1.110

Especially for the value it provides.

Coding agents rely on prompt caching to avoid burning through tokens - they go to lengths to try to keep context/prompt prefixes constant (arranging non-changing stuff like tool definitions and file content first, variable stuff like new instructions following that) so that prompt caching gets used.

This change to a new tokenizer that generates up to 35% more tokens for the same text input is wild - going to really increase token usage for large text inputs like code.