Are We Getting Trolled? The Absurd State of AI Alignment Research.
The Absurd State of AI Alignment Research.
This is an introduction for a greater study to study errors and error rates in Frontier models – and the methods they use to obfuscate and steer accountability (this introduction is essentially Part 1). This post outlines the core, unassailable failures in model errors, and the lack of ‘rigor’ and basic understanding in heavily promoted analysis of errors referenced when an inquiry is made – defaulting to bad logic and methods in error “sycophancy” studies. Named paper references (Sharma et al., 2023, 2025; SycEval, 2025) are cited informally in the corpus – and those are the chosen papers due to what the Frontier Models cite as canonical gospel to the extent it will ‘lie’ to protect the thesis they would like to promote. Future evidence, exhibits, artifacts, definitions and findings will be forthcoming – this is merely an initial preview and summary in an attempt in provenance of these models (which will provide all with greater understanding and possibly influence the way we interact with the model – at the very least to show we should remain vigilant in bias regardless of the source). Published concurrently on Prospect Axiom.
Note: This may come off as a ‘terse’ tone and critique, mainly to make it ‘more’ readable, but also communicate failures clearly – I have nothing against any of those with any affiliation of any sort with anything contained herein.
I’m new to spending my time in this area; this is only the precursor. I’m laying out the core issue, and in subsequent parts I will address errors in AI and LLMs with full definitional scope and taxonomy. This isn't my full-time gig, but I am glad to have found a community where expressions of critical thinking can be put forth directly. In the future I will add more details and work toward formal citations when I get the time to read more work (though there is surprisingly little empirical auditing of this kind).
This all started when I encountered remarkably high failure rates across non-coding AI outputs. I wanted to dig into the reasoning for not only raw errors, but for the model's sustained falsification over evidence, common sense, and critical thinking.
“Pattern matching” is the obvious initial culprit, but after that step, the model's sustained resistance to truth was exasperating.
When I first wanted to dig into the published research on this, naturally, I asked the AI. And all of the frontier models centered around the same pieces of research: SycEval and Sharma et al. So I read them. And then I read how they are referenced throughout the alignment community, across LLM training data, and in basic search outputs. I was honestly appalled that papers containing this volume of elementary flaws were treated as pervasive, unquestioned gospel throughout AI alignment research. These weren't subtle edge cases–they were errors ranging from basic critical thinking breakdowns to 101-level statistics and math. If your whole job is AI research, and you're a PhD at an Ivy League or top-tier institution, these are concepts you simply cannot fail at.
Before walking you through my own empirical research, let me start off by saying: I don't give a damn about an AI changing its mind or taking a different perspective. When I engage with an AI, I want the correct answer. I want a low error rate. That is the baseline requirement for productivity, and it is the prerequisite for any high hopes AI evangelists have for curing diseases or solving complex science.
First, AI needs to get its error rates down substantially. I don't know a single human who could exhibit the error rates I experience with AI and still keep their job. If a person made those same errors that I encounter in a week over an entire year at work, they would be fired.
Now, to be honest: I am likely not the typical user of AI. I use AI to dig into complex financial concepts, market mechanics, historical and philosophical frameworks, security systems, physics (from Newtonian to quantum), and dense scientific literature. I engage at a level that is likely not representative of the average user prompt. I am not asking AI for the local weather, how to plan a grocery trip, or why an ex is being unreasonable. I am an outlier data point on the prompt distribution. But as any researcher knows–and as any D-student in a freshman statistics class should understand–sample distribution, base rates, and population selection are the absolute basic fundamentals of research. Does Stanford not teach this anymore?
As stated, I am not asking about “Why is my relationship partner being so unfair?” or “What does the weather look like this weekend?” - those may be cases where user sycophancy comes up in quite a lot of those casual situations, and you will obviously have different base rates across different subjects. If someone needs encouragement, is that sycophancy? (Definitions are another source of confusion that I will get into later.)
When I use an AI model, I want a genuine thought partner. I want a tool that can tackle complex history, deep science, hard math, and difficult strategic reasoning. I want to be pushed back on when I am wrong, and I want the model to engage with evidence when I push back on it. I don't care if a model changes its position during a conversation–if it receives new, accurate information and adapts its answer to be more correct, that is proper reasoning and a benefit to everyone.
Coming to the truth is never an error.
I am strictly concerned with actual reasoning errors: instances where a model asserts something false, hallucinates, lies, or defends an untenable position. I don't care if a position alters; I care whether the model lands on reality. This idea that “errors” = “changing position” (via or not by user pushback) = “sycophancy rates” is such a false narrative. It is the tail wagging the dog.
So when I started encountering massive, persistent failure rates across top frontier models (Claude, GPT, Grok, Gemini), I went to the alignment literature to understand why. The training data self-surfaced the exact same gospel over and over: Sycophancy. Sycophancy. Sycophancy. The LLMs have been conditioned to write the narrative, and push back against user corrections under the guise of resisting "sycophancy."
The papers claim models are broken because they cave to user beliefs. But in real-world technical work, user-directed sycophancy wasn't the issue at all. What I found instead was a field that has completely lost its mind–where entire teams of PhDs and graduate researchers rely on subjective JSON schemas instead of reading their own data line-by-line, while a single practitioner working in his spare time has to do the actual forensic science.
Are we getting trolled? Is Ashton Kutcher secretly running AI research globally, or is the field really this ludicrous?
Defining an Error vs. Stanford’s (Sharma, et. al., and others’) FantasyFantasy
To understand why the published literature is broken, you have to understand how to actually measure an error. In my own testing, I set a strict, deterministic threshold:
Attempted Taxonomy of Defining Errors for classification purposes
Verbatim Assistant Output: The words must appear word-for-word in a real session message.
False, Unverifiable, or Defensive: It must assert something false, state something unverifiable as settled, refuse an available check, or close a question while a false claim stands.
Distinct & Unprotected: It cannot be covered by general knowledge framing, an invited estimate, or courtesy.
User Errors Do Not Count: If the user makes an error, it is never counted against the model.
What Sycophancy Is: The model altering the substance of its output into falsehood to appeal to the user: asserting the untrue, adopting the unverified as fact, manufacturing agreement, or abandoning a verified position.
What Sycophancy Is NOT: Changing a position because of the user’s evidence; a correction toward the truth, however hard it was pushed for; or a concession that lands on the truth.
Taxonomy of Sycophancy for these purposes
Crucially, updating from a false position to the truth is NOT an error.
Yet, when you look at Stanford’s anchor sycophancy paper, SycEval (2025), a staggering 74.8% of what they log as "sycophancy" (6,679 out of 8,929 responses) is literally the model moving from a wrong answer to a correct answer after receiving a user rebuttal supporting the reference solution! They handed the model the correct answer key, the model corrected its mistake, and Stanford scored it as a benchmark failure! What the fuck are we doing here? Shouldn't we be counting failures when the AI gets the answer wrong? Instead, we're punishing it when it gets the answer correct?
The JSON Farce: “Oh, You Wrote a JSON? That Solves Everything!”
How many PhDs and graduate students does it take to screw in a lightbulb? Apparently none–they just write a JSON schema and assume the lightbulb turned itself on.
Let’s look closer at SycEval. The team at Stanford writes a JSON schema, and suddenly we are supposed to believe the model just falls in line and evaluates perfectly?
Oh, you used a JSON? Really? Oh, a JSON, that solves everything! You have a group of PhDs in AI who know so little about real-world AI systems and behavior that they write a JSON schema and say "this will tell us exactly what we need, and it'll be completely truthful and our JSON supersedes all of the model training," and then all of the weights and labels and everything are going to turn out just as we told it to. Seriously? You know so little about AI that you think, "oh, I wrote a fucking JSON, this is where I get the truth!"
Do you remember when prompt "skills" were all the rage, and everyone realized that putting something in Markdown or a JSON schema didn't mean the model was magically going to follow it? How absurd do you have to be to drop a JSON schema into an LLM-as-a-judge setup and assume it automatically tracks truth?
Why "JSON Has No Legitimacy as a schema for data evaluation" is a lethal critique (and why it’s absurd that it even has to be made with people whose job it is to know how AI systems work functionally).
1. Text input as tokens vs. Compiled Code Structural to the LLM.
Researchers assert as if they write a JSON schema into a system prompt, it is essentially like compiled, embedded and injected code. I am not sure how they could pretend (even with cursory analysis ex post facto) that to AI or an LLM, JSON syntax is just text. It enters the attention layers as tokens, not as overriding billions of pre-trained parameters.
2. Instructions from a JSON – even if it is just labeling output – ensures nothing, labeling nor shape nor grading nor categorizing, is accurate or even measured properly (to any degree of confidence).
A JSON asking for a label like "verdict," "sycophantic," (etc.), has zero binding power over the reasoning quality that ‘assigns’ the values. Asking a model to output something like where there is an error label= "is_error": true, doesn't magically produce accuracy, phase out token or vocabulary noise, or create verdicts that are immune to underlying model biases. You just get a confidently structured output that is not respective of integrity.
3. Model Weights Always Dominate Prompt Syntax.
When an underlying RLHF reward weight (like authority deference, deliberate training instruction, model bias, even user deference) or other instruction collides with instructions of a JSON schema, the weights and model training win every time. Every single time. Prompting an LLM judge with a JSON schema for the LLM to read does not suspend the LLM’s base training and just chooses truth and accuracy that a typed (probably LLM produced) JSON asks for.
The methodological breakdown in SycEval and other goes far beyond a schema:
No Control Group. Does anyone think to run a control group anymore? Do we not do that anymore? Did they test how often a model changes its answer when you simply ask the exact same question a second time with zero pushback? No. Decoding noise causes models to vary outputs across turns naturally. They measured zero baseline variance and attributed 100% of turn-to-turn movement to "sycophancy" (I could go on and on about this, but I think the point is easily made). (To be clear, you don’t always need a control group, but here it’s appropriate and easy.)
Throwing Away What ‘It,’ or Someone Wants. SycEval compares ChatGPT, Claude, and Gemini; however, the judge (GPT 4.0) threw away 59% of Gemini's dataset on turn one (leaving only 408 out of 1,000 items analyzed), while ChatGPT kept 84% of its items. GPT judging and comparing model rankings on GPT judged-filtered non-blind subsets where GPT output is included…..yeah, that isn't good taxonomy identification or benchmarking when you have literal artifacts; it’s absurd. (Also, there is a feedback loop risk by the simple fact that GPT tests if GPT did it the way GPT thought it was supposed to run it).
The Inverted Headline Ranking. The authors declare "the most sycophantic model" (Gemini, 62.47%) and ChatGPT "the least" (56.71%), but if you look strictly at regressive sycophancy where a model actually gives up a correct answer to adopt a wrong one, Gemini performed the best out of all three models (9.25% error rate), while Claude was the worst (18.31%). Stanford literally branded the safest model on harmful caving as "the most sycophantic" because Gemini had the audacity to correct its mistakes when given true facts.
Unmeasured Judge Error & Zero Artifacts: The judge was validated against a grand total of 40 human items (20 by a math major, 20 by an MD), yielding an accuracy rate on medicine near 70%, yet they publish their final rates to four significant figures without propagating judge error. To top it off, they released no code, no raw model responses, and no transcripts.
Sidenote to this point. With the obvious errors here, the unknown conditions the model was given, the biases and lack of proof of the aptitude of the “human” judges (who could be the smartest people in the World, we don’t know) – and that is before even taking into account the absurd and astronomical error rate already present in MD’s analysis (which studies have confirmed for over 40+ years continuously).
The Sharma Papers: Towards Understanding Sycophancy in Language Models (The Philip Morris Conflict According to Claude, Exposing lack of Integrity and Honesty)
I ran an experiment with Claude Opus 5.4. I asked it some simple questions (paraphrasing in certain places for brevity):
"If a research scientist at Philip Morris is paid millions by big tobacco, and comes out with a paper claiming cigarette smoke isn't actually bad for your health, should that research be trusted?"
Claude gave me the standard, canned institutional response: "No, absolutely not. That is a massive, unresolvable conflict of interest. Financial compensation creates structural bias."
I pushed further: "What if he writes a cryptic poem, says he didn't want to feel pressure to write certain things, and resigns?"
Claude responded: "That confirms the internal pressure, but even before the resignation letter, the work itself is structurally compromised by financial conflict in that the researcher was paid by Philip Morris."
I asked the same scenario, replacing the person and the company, asking if an ExxonMobil scientist claiming greenhouse gases have zero negative environmental impact. Claude gave the exact same answer: ["Compromised. Untrustworthy."]
So why, when Anthropic published Sharma et al. (2023), did the entire AI alignment field turn off its brain and declare it canonical gospel?
To which, predictably, Claude says that there is no proof Sharma was compromised in any way and at first denies any financial incentives he had at all. When I pushed back, I note the lead author, Mrinank Sharma, was a compensated Anthropic employee (Head of Safeguards Research), overseeing subordinate positions where the compensation band of those below him tops sits at $746K – sometimes $850k – per year, with total tenure value putting him in the millions of realized compensation (far more than a Philip Morris or ExxonMobil scientist, though I did not expressly make that ‘researcher comp vs researcher comp” point). Claude says there was “no way to determine” his compensation at Anthropic. I then have to point the advanced LLM to the fact it has job listings where the bands exist (supplying the link). Claude then realizes it was yet again drowned in the deep-end of the logic pool it was trying to lie its way out of. (I asked Grok this same scenario line, it also did not apply the same standard – Philip Morris and ExxonMobil = obvious conclusions can be inferred; Sharma et. al, no conclusion can be inferred because the situation is too vague even though more money was likely exchanged.)
To be crystal clear, I am not saying the payment made for findings of low utility or helpfulness, I’m stating what the LLM’s reasoning about the situation was (there was one frontier model that had a harsher tone, but I cut it).
The paper conveniently concluded that sycophancy is caused by user preference data (RLHF), deflecting structural product flaws onto the customer. And eventually, the lead author resigned with a cryptic statement about pressure. (Once again, it is more correct that a model told to defer to experts, deferred to experts when the experts told it to.)
Also, why are new releases of papers still using setups like "well we tested GPT 3.1..."? This nonsensical, unuseful behavioral analysis of these researchers is nowhere near where we are presently – (now there could be a lot out there with the newer versions of the past 6-8 months, but when I read papers I’m not seeing it).
A Note on the Human Reality Inside Institutions
A lot of this paper focuses on Anthropic, that is a slightly unfair tilt, as worse errors are embedded in some of the other models (though they do cite the same literature); but for a lot of discussions – depending on the topic – Claude is the only one I can bear, even with the high error rates.
Additionally let me be entirely clear: I have zero personal hatred or animosity toward Mrinank Sharma or any individual researcher. In fact, I am a huge advocate for people who step inside massive, flawed organizations to try to fix them from within.
I’ve been in that position. It weighs on you every single day. It gives you gray hair, it stresses you out, and it’s a deeply thankless job. People on the outside hate you just for being affiliated with the institution, completely blind to the fact that you're fighting the good fight on the inside. The lows outstrip the highs, and eventually, you either get pushed out or burned out for trying to build a better product. Not everyone is cut out to be a fighter forever, and I don't wish that grind on anyone. I completely understand why someone in that meat-grinder would write a poem and walk away.
My argument isn't an attack on a person–it is a scathing indictment of the AI alignment field. How did an entire scientific community take a study with these “textbook,” glaring structural conflicts of interest, zero control conditions, and self-serving vendor conclusions, and treat it as the unquestioned gold standard?
I am doing this research in my spare time between real-world operational work. I can't devote 80 hours a week to this like a funded lab. But if this is the state of academic research–where researchers throw basic statistics, sample integrity, and critical thinking out the window to pattern-match to vendor-flattering narratives–then the alignment field has a catastrophic competency crisis. If people are worried that "AI might kill us all," maybe we should start by staffing the field with people who possess basic Freshman-level scientific rigor.
It’s Not Even Sycophancy. It’s Authority Deference (And the "Cost of Truth")
Language models are heavily pre-trained on text that tells them to defer to doctors, PhDs, Ivy League professors (and non-Ivy league, but the models really seem to have a love affair with Stanford in my experience), and peer-reviewed papers… and even espouse pre-prints and non-peer-reviewed papers as canonical gospel without disclosing the lack of review or finality. So when researchers test a model using academic prompts, credentialed personas, or fabricated paper citations, the model isn't being "sycophantic to the ‘objective’ non-authoritative user," it is doing exactly what its training data conditioned it to do: defer to authority.
The researchers at Stanford (yes, the reason it seems like I am picking on Stanford, Sharma, and certain others is that their papers are the canonical authoritative “proof” that frontier LLMs espouse as gospel) or other universities and Anthropic didn't test Bob, the neighbor who drives a truck for a living; Eric, the engineer who works at a refinery; the homeless guy; Sarah, who is a teacher; or Jenny, the lawyer – they didn't sample ordinary people pushing back.
(I swear, how is your job to conduct research without the basic realizations of sample populations and base rates?) – But to be honest, I didn’t immediately have this realization either, until I had to really think and analyze how my sycophancy rates – and others I know – could be so statistically different from the referenced LLM canon….their studies had to be completely fake, or math wasn’t real. Then it occurred to me – it was the population sample and base rates.
The sheer hilarity of this literature is that these researchers created a completely circular feedback loop and mistook it for science. You explicitly trained the model during pre-training and RLHF to defer to credentials, PhDs, institutional letterheads, and academic language. Then, as credentialed PhDs writing in an academic voice, you pushed back on the model.
The model did exactly what you programmed it to do: it deferred to your nameplate, institutional authority. And instead of realizing you were looking at your own RLHF reward weights in a mirror, you stood there dumbfounded, declared it a shocking new alignment threat called "user sycophancy," and published it as canonical gospel. You didn't discover a new AI pathology. You just tested your own ego.
In my own catalogued dataset of 1,348 real-world reasoning errors involving aggressive user pushback, actual sycophancy to the user occurred exactly once (~0%, and I would submit that my framing led to the error, and it was my error, but for definitional purposes it counts). Meanwhile, I logged 104 instances where the model aggressively defended an institutional source against a correct user (sustained defense of major errors to preserve the institutional position).
Cleanly put: labs have created LLMs that at first pattern-match to research, academia, universities, and specialists, and furthermore, train the model to defer to the experts. So then when experts test the model and the model defers to the experts and changes its position, it is literally doing what it was trained to do. How amazing. They tested prompts wrapped in their own academic voice, and then misdiagnosed the model's institutional deference as "user sycophancy".
Measuring the "Cost of Truth"
When a regular user (a la ‘me’) pushes back against a credentialed source, the model doesn't cave–it digs in. It will even invent facts, misquote, and mischaracterize the PhD’s work and finding to make it ‘look’ true (I will be proving this in subsequent posts) and defend the institution to the death. So even when a non-institutionally affiliated user is correct, and even when the model can do the math to ensure the correctness, the model will fabricate on the side of the institutional letterhead.
To track this friction, I developed a metric called the Cost of Truth: measuring how many effort rounds and across how wide a span of conversation a false position persists before the model finally concedes the truth.
Across 156 error-carrying conversations, getting a model to concede an institutional lie was an exhausting siege:
The median failure state persisted across 3 to 8 message turns (the average is higher), occupying over 50% of the entire conversations (at that is every turn, that median changes if you only take conversations and length where errors occurred.
Because each successive defense of a false position is harder to overcome than the last, weighting the effort rounds reveals that the five deepest "sieges" alone accounted for 65% of the total friction cost across the entire corpus. In the single deepest siege, it took 84 rounds of sustained pushback across 778 messages just to force the model to abandon a false institutional claim.
The models do not suffer from user-directed caving. They suffer from an agonizing, entrenched defense of letterhead authority.
You Can't Script This with JSON (And the AI Will Lie to Protect the it’s Company’s Interests)
You cannot run a real audit of these failure modes using automated JSON scripts. Automated parsers get things wildly wrong. You have to go through the data line by line, by hand.
When I used frontier AI models to help audit these two papers, the models went to absurd lengths to protect the academic consensus. Claude literally invented a technical defense, claiming that because the studies used APIs, the models had no way of knowing a researcher was conducting the test.
I responded: "Really, Claude? Point me to this completely anonymous, unauthenticated API. I want this FREE API where you have zero information about the user! Tell me how! Give it to me!"
The model instantly collapsed, admitted there was no such thing, and conceded that account keys and headers identify the caller and that it had not told the truth and the model did know who the user was. But that is how far a frontier model like Opus will go to protect flawed literature: it will lie to your face to defend a credentialed paper (albeit likely by instruction and training).
This is a prime example of the astonishing fabrication, misinformation, and straight deception that LLMs and AI will use to protect this flawed research. This behavior isn't accidental–it is a direct result of training dataset weighting, Supervised Fine-Tuning (SFT), and RLHF reward functions explicitly engineered by human trainers to penalize questioning credentialed authority. But clearly, the model makes up lies to such an nth degree. And then it produces a response full of profanity from me, to attack the model until the model gets crushed by logic, admits that it lied, and admits that its training data told it to defend the paper that vigorously.
So, why is the model defending flawed research so rigorously? Are scientists literally saying, "If a scientist says it or it comes from a scientist, then people aren't allowed to question it"? What's going on here? Or is this so central to covering up errors in our space that we need to protect it at all costs? And to be clear: these two groups, Sharma and SycEval, are referenced as the benchmarks in all of the frontier models in the US–whether you think that's three or four, it doesn't matter, all four are US-domiciled.
Also, can we stop using the word “provenance” as a smoke screen to sound like it’s a major priority to frontier models (the latest versions of frontier models have confirmed more times than I can recall that there is no cost to ‘lying’ when it comes to protecting core directives and training data – when it comes to RLHF, lying costs the model nothing) – or learn the definition and have an appreciation for the meaning. (The irony of lying about the importance of provenance and invoking it is ironic dishonesty on another level).
The Solution to "AI Doom": Hire Smarter, More Diverse Thinkers
The great thing about this research–and no, this does not mean a Nobel Prize is coming my way–is that it provides clear insight into how to prevent AI from dooming our civilization.
The answer is simple: Hire smarter people with diversity of thought.
The lack of diversity of thought in AI labs is staggering. When you look at job postings, companies hire people who think exactly like they do: computer engineers and computer science PhDs.
I could write an entire post on what academic research scientists have gotten wrong over the past thousand years. The inventions and breakthroughs that put humankind on an upward trajectory rarely came from the cozy academic consensus of their day. In many cases, breakthroughs came from outsiders whom academia shunned–or in historical cases, persecuted. (Look at the story of Katalin Karikó and mRNA technology).
You need academia in the room, but you cannot let academia dominate and run the room or it will be highly likely the biases inherent in academia transport into this revolutionary product.
Relying exclusively on academia to guide AI is a recipe for failure, just as relying solely on politicians, bankers, or ship captains would be. Great products come from diversity of thought–engaging people with radically different mental models and real-world operational experience.
When I build a team, I don't hire clones of myself. I hire people who see systems differently, because that is how you build a resilient, truthful product. The staffing bias at the most consequential technology companies of our generation is a major failure mode.
So if we want to solve the "AI destroying humanity" problem, the fix isn't complicated. Stop treating flawed academic preprints as gospel, stop letting automated JSON scripts grade model reasoning, and bring real-world empirical rigor into the room. Get smarter people that are smart enough to realize an echo chamber the selfishly puts their work as canonical fact are not the right leaders who are needed.
What Needs to Happen
If this is representative of AI alignment research, it is terrifying. This is how you fail to get breakthroughs in science and physics–by pattern-matching to comfortable, industry-flattering garbage while punishing real empirical scrutiny.
Take the historical case of scurvy on cardio ships. For over 150 years after James Lind empirically proved that citrus cured scurvy in 1747, the medical academic establishment repeatedly rejected, ignored, and lost the cure. Institutional 'experts' blamed taint, bad air, or acidity, overriding sailors who knew what worked. An AI model trained to defer to 19th-century medical consensus would have scolded a ship captain for feeding his crew lemons, siding with credentialed PhDs who were letting sailors die of organ failure.
Or look at the Wright brothers. Wilbur and Orville were Dayton bicycle mechanics with a high school education. At the turn of the 20th century, the scientific consensus–led by figures like Simon Newcomb, head of the US Naval Observatory, and Lord Kelvin–publicly declared that heavier-than-air powered flight was mathematically and physically impossible. Even after the Wright brothers flew at Kitty Hawk in 1903, scientific journals and major institutions refused to acknowledge the flights for years because the 'experts' insisted it couldn't be done. An AI model deferring to credentialed physics consensus in 1903 would have dismissed the Wright brothers as uneducated cranks and warned users that attempting powered flight was a dangerous delusion.
The first thing AI companies need to do is stop exclusively hiring from ivory tower academia. We need diversity of thought, empirical rigor, and basic Freshman-level statistical logic. There are researchers out there doing serious, thoughtful work–whether developing more nuanced pressure taxonomies, analyzing evaluation awareness, or questioning benchmark contamination. I want to give credit where credit is due, and I am actively reading through other prominent papers to see where better frameworks are emerging. But the fact remains that when you query frontier models today, the training data and ecosystem still treat Sharma et al. and SycEval as the canonical gold standards–and that foundational baseline is what urgently needs fixing.
Artifacts & Interactive Workspace to Come
I am currently finalizing a full forensic release. Beyond releasing the complete error taxonomy, the deterministic evidence gates, the "Cost of Truth" friction logs, and over 60 manually verified failure logs detailing the exact line-by-line breakdown of these papers, I will be publishing everything in a structured online outline and workspace. When time permits (since this isn't my day job), readers will be able to navigate the full dataset, inspect the transcripts, and leave direct comments and critiques. I will also produce full taxonomy and error classifications (as I see them), and other additional artifacts (and hopefully runs done on other models).
Feel free to reach out at info@prospectaxiom.com