Update on that thing

Friday I posted that the latest Mistral models might be getting close to checking all the boxes when it comes to balancing quality and ethics. They have the documents for EU compliance, you can run them in regulated datacenters, their commercial terms let you block out training using your data. The box to check was quality. How do they stand up to the big guys with less regulation?

I have some spot checks I use that are clear indicators for me, but to explain them I need to get a little into how this thing works:

What I built is basically a workflow engine. The workflow is getting editorial notes on a book, and the challenge is straightforward. Book goes in one end, formatted notes for your consideration come out the other. No editing happens, all authorship is done by the author, no training, just a tool in the toolbox of the author.

What happens in the middle is a little more nuanced though.

1. First thing we need to do is ingest and understand what the book looks like. Is it fiction, memoir, self-help, business non-fic? How many words are we talking here? What’s your (intended) genre? Do you have chapters, scenes, acts, a prologue, epilogue, front matter, back matter, chapter front matter, maps, endnotes, a bibliography? And importantly, are there any narrative choices we should know about? Did you write an epistolary? An unreliable narrator? A 4-act Kishōtenketsu? I’ve built anti-patterns to prompt the LLMs for all these scenarios. In all, there are over 200 and counting prompts that can inject at different times to offset the built-in western bias in all the models. It’s a lot…
   
   But then there’s a last thing in this phase, which is the most important piece – What is your question? What are you looking for from an editor right now?

2. With all this collected, and your book sliced and diced into its parts, with all your burning questions recorded, and everything confirmed, we can finally have a read through of the book.
   
   The first read isn’t an edit, it’s understanding. On the first pass I have the models categorize and tag everything it passes through:
   • Is this a character?
   • Does this character have a manner of speech?
   • Is this a worldbuilding element?
   • What description did you give to that (person, location, culture, etc.)
   • Did you just mention a date, time, deadline, or a backstory?
   • Is this an unusual way of writing? (subjective, I know, but the reason will be obvious in a sec)
   • And lots more…
3. With all those tags and points, now I can construct the first real output of the editor—The Story Bibles
   
   The story bibles are a condensed read of all the things I listed above.
   
   • The Character Bible – Lists all the characters, their descriptions, relationships, arc state at the beginning, middle, and end, their key desires—basically everything we need to know about everyone in the book. There’s even a minor walk-on section for people you just mention or have a one liner in chapter 27. I don’t neglect other types of personified entities too. Pets, aliens, things, collectives…if you can anthropomorphize it, we’ll give it a mention.
   • The Worldbuilding Bible – Catalogs all the elements in the world of your book. Geography, politics, factions, technology, magic systems, corporations, biology and physical limitations. Whatever makes the world of your story tick.
   • The Timeline Bible – the complete history of the book, from the backstory elements, through the events of the book. We mark down every tick of the clock mentioned in the book.
   • The Voice Bible – This is where we want to capture the voice of you, the author and the characters you’ve created. This is how we’ll prevent the copy editor later from flattening and correcting all the quirks that make your work yours.
   • The Continuity Watch List – Even on the first read we can likely already see if things change halfway through or if you put down the book for a few months and didn’t remember how you described that character in chapter 2. These are things to investigate, not necessarily bad things, just what we notice on the first pass.
   • The Questions Routing – Finally, based on everything we’ve seen, we take a first glance at your questions, and recommend what edits will answer them. Are you asking about the main character’s arc? A targeted developmental edit, centered on that character’s journey will give you the answer you’re looking for.

4. At this point in your “Author’s Journey” you can decide to take up the quest or you can read through the bibles to make sure everything we caught was correct and clear. This is an important stop, because if things are wrong in the bibles, they’ll be used to inform every edit that comes after. Big cascading failure energy…
5. And last but not least, the edits themselves. Developmental, Voice, Continuity, Copy, and Line edits are all possible, either full book or targetted by chapter ranges, character arcs, plot elements. Whatever you need. The routed questions are lined up to fire off the needed edits at the push of a button.

One thing you should notice about all these steps is that none of them really requires an LLM. Nowhere up there did I say an AI is doing all this for you. If I did, you’d be 100% correct in calling me for bullshit and closing this window right now. Most of the splitting, routing, searching, indexing, and formatting is all old school computer work. LLMs are just one tool among dozens that make this system all work. (Which is part of my point)

So back to those quality checks I mentioned at the top.

When I test out a LLM for editor duty, there are a few good indicators I use from my own books. One I call the “Nance Test” — Early in Bluest Moon, there’s a scene where the Admiral, the main character Lync’s adoptive father, walks into a room occupied by Lync, another admiral, and Lync’s adoptive mother. Upon entering the room, the admiral calls his equal in rank by the diminutive “Nance” and kicks off a verbal pissing match among the military brass. It’s pretty clear for any human to visualize who he’s talking to. The main character’s inner dialog comments on the goings on a few sentences later. Before I started testing LLMs I would never have thought this was in the least bit unclear.

But LLMs get it wrong a lot.

The most common failure I see in this scene is a misattribution. It assigns the diminutive “Nance” to the wife standing next to her. Whether this is a baked in misogynist bias in the models, assuming the diminutive must belong to the man’s wife. Or a lack of context grasping who the name is directed at, with 2 characters standing right next to each other. OR a cultural context bias that refuses to assume an admiral is disrespecting another.

Whatever it is, it’s baked into a lot of models, and smaller ones fare a lot worse.

But you know who gets this right nearly every time? Freakin Anthropic. Only their Haiku small model gets it wrong. Sonnet got it wrong half the time up through 5.0. Opus never got it wrong.

SO after my last post, I tried it on the newest Mistral Large 4. I threw $10 at an API account and fired off a dozen test runs. For comparison, I ran it on a few runs of Mistral Small 4 too. I can run that one locally, but it dates from back in March, so expectations aren’t super high.

The Result? Big ol’FAIL. Large 4 got the Nance test wrong every gods damned time. Apparently French models have just as much misogynist bias as the Chinese ones. For comparison, Qwen 3.8 Fast Next gets it right only 1 in 3 times. That’s about the biggest thing I can run locally on my little GB10 Spark box, and I’m not about to send any money their way to test bigger versions.

There is a glimmer of hope though. A feature of all the above steps I didn’t talk about is The Verifier… More on that later.

Yeah, like this…