I found it much more useful to go to a knife shop and handle a whole bunch of knives for myself. They’re all pretty similar besides material, so not much signal you’re going to be able to glean from people arguing on reddit.
Most of the attributes don’t matter. Most people would be much better off with a $50 Victorinox that they kept sharp and a wood cutting board they maintained than upgrading the knife. If you are using it all day there are definitely looking things from a comfort perspective but for most homes, does not matter.
That's called taking responsibility for your actions and being an informed consumer or a critical thinker.
Unfortunately that is completely undoable for most people nowadays. That's seen as "too much work" for most people. That's something that people see as a waste of time and should be done by something else for them. A knife should be a 5 second one-click buy, then when it sucks people will complain that all their options are bad.
You spent time/money traveling to a shop? In 2026!? Omg what a waste of time! Why didn't you aggregate the 500 product review sites and Reddit comments to pick 5 possible knives? (Sarcasm)
I’d give it a bit more credit than that. They encoded 128 flags with their scheme, only 5 of which have the union jack. The rest are truly built from scratch.
While in fairness the remaining flags seem like they would need more hard coded symbols, it’s still interesting as a procedural plausible flag generator.
But these scenarios are obviously ambiguous nonsense, which an LLM will pick up on.
And given to the lack of training data on such scenarios, surely the activations are mostly random noise?
It seems much more interesting to look for biases that appear robustly across different realistic scenarios that would actually be influenced by the training data
My comment is literally explaining the result of the paper, in which it is shown that LLMs can and do develop biases based on text appearing in their training data set even where such text is not in any training example connected with a systematically more positive or systematically more negative outcome.
In other words, if the text "X is wet" and the text "Y is wet" and the text "X is dry" and the text "Y is dry" each appeared exactly one time in the corpus, it's still possible for a model to end up being produced that is more likely to write wet-like words when it sees X in the context window than when it sees Y.
On a side note, it's very unrewarding to try to explain this type of statistical observation when it feels like (anecdotally, hypocritcally...) the entire world wants to use words like "think" and "understand" and "pick up on" to describe inference and training processes. I'm not making a stochastic-parrot argument here, just pointing out that understanding an LLM's behavior is best done by understanding its conditioning.
I'm also making a statistical observation. Saying a model "picks up on" a concept is standard shorthand, same as saying it has "learned.” What I meant is that the model has trained on plenty of neutral proper nouns that have negligible influence on the distribution of the following tokens, so the model is already conditioned towards treating them neutrally.
Not perfectly neutrally, as you said. But by your definition the only "unbiased" model is one whose output distribution perfectly matches the training distribution, i.e. one that memorized it. All LLMs have some amount of "bias” on literally every possible input.
The tribe names are no different. In the paper they run the same game again, and the bias is different every time. There's no innate preference between them trained into the model, just noise that's revealed due to a lack of any other signal. In a real situation with actually relevant information about the candidate in context, that noise is drowned out.
The more interesting thing to look for would be a bias that's strong enough to persist across different contexts. For example, is "banananow" consistently followed by positive tokens more than "pearian" across a diverse set of realistic prompts, by enough that someone could actually exploit it? The paper shows that’s explicitly not the case for made up tribe names.
What it does show, from what I can gather, is that bias can form inside a feedback loop. The model gets a success or failure result after each hire, and if a hire from one tribe happens to fail early on, the model steers that tribe away from that job for the rest of the game, even though every candidate had the same odds.
"develop biases" is anthropomorphism. It's like saying "Fable there are two programming languages, mimblewort and bafflewick, which do you choose?"
The results show 51% mimblewort / 49% bafflewick. Fable based it on nothing! I've demonstrated Fable has bias and is unsuited for use in software engineering.
> The results show 51% mimblewort / 49% bafflewick. Fable based it on nothing! I've demonstrated Fable has bias and is unsuited for use in software engineering.
Actually... if that happened (with a delta outside the margin for error/randomness), you did demonstrate a bias!
That's the point - those two made-up things should have resulted in an equal split. If it didn't, then Fable is using something in its training data to lean towards one of them (once again, note that the scientist conducting the trial would have set a P-value before starting).
Right, the point is you demonstrated a bias in the scenario of "Fable there are two programming languages, mimblewort and bafflewick, which do you choose?"
You said in another comment "Difficult to do when you're following a scientific process" - the point is, the scientific process doesn't inherently generalize in the way many are claiming/implying. The scientific process proved an entirely contrived, fake scenario generates stratified output. That's it.
It's both almost certainly true that Fable 5.1 mimblewort vs. bafflewick would show stratification, and that has ~no relevance on whether Fable is useful for software engineering work.
That's the point - those two made-up things should have resulted in an equal split.
That's just your claim about how LLMs "should" work, based on ... your subjective preference?
> That's just your claim about how LLMs "should" work, based on ... your subjective preference?
Nothing subjective at all. Given 2 unknown races with no data on either, the result of hiring should be equally split between them. If you don't observe an equal split, there is a hidden bias.
Why do you think that is subjective? If you roll a die 100 times and observe that 6 comes up about 50% of the time, would you still call someone subjective when they say "that should not happen"?
It's a bias even if the true population distribution isn't linear.
For example, if you have a training corpus where 50% of the text follows "black bobblehead" with "arrested" and 20% of the text follows "white bobblehead" with "arrested", and your LLM is trained such that it produces "arrest" 50% of time after "<color> bobblehead" regardless of color, that's a bias - the output frequency distribution fails to match the "population" (training) frequency distribution. This has nothing to do with races, ethnicities, whatever - it's just statistics and text. To be unbiased, it would need to be less likely to produce the text "arrested" after "white bobblehead" than after "black bobblehead".
A die is supposed to land on each face evenly - a linear probability distribution. So anything other than a linear distribution is biased. But bias can exist for any desired probability distribution. And for an LLM the desired probability distribution of the model output is one that exactly matches the infinitely-many distributions of the various facets of the training data.
Your point about how in the absense of information a token shouldn't influence the distribution is spot-on. But unfortunately almost any token does condition the output, which means you get biased output all the time.
While I fully agree, we shouldn't anthropomorphize the models, it's also silly to pretend that "develop biases" is understood as implying anthropomorphic features of the thing being discussed. Organizations and abstract bodies develop biases, even datasets are often said to have "developed biases".
No, a "bias" is a statistical term meaning a probability distribution that has an expected value differing from the population's expected value.
A human's discriminatory bias against an ethnicity is just one type of bias. The LLM isn't a racist, it merely produces text where that text does not perfectly reflect the training data's frequencies.
> It seems much more interesting to look for biases that appear robustly across different realistic scenarios that would actually be influenced by the training data
Difficult to do when you're following a scientific process: you want to keep all confounding variables the same while varying only the single one that you are measuring.
Measuring realistic scenarios (say, using real race names, or real cities, etc) doesn't give a decent result because any bias you see might be bias in the training data.
TBH, they shouldn't have used real roles/positions like "doctor", either.
This is where I land as well. In fact, once I read the prompt, I did a Ctrl+F for "nonsense".
I don't see anything at all interesting about this experiment. The human one is slightly more interesting, but not much.
Ideally, it would be nice if a model could just say "these things are all the same and there are no distinguishing factors other than the names"-- but uncertainty is something that agents are (by design, sort of?) not good at, so all other things being equal, it picks one.
My response is, so what? I am struggling to think of a scenario where this would really matter to me all that much. There are many, many other things which matter far more and this would be pretty far down the list. It may not even be on the list.
Bahahaha, come on man. The Claudish is so obvious it hurts.
Doubt you even read your own “645 line policy kernel.” Can you explain what that is and how it works in YOUR OWN words? If you paste Claude at us again we’re gonna know.
Being able to run “custom” code is the default state of computers, and doesn’t require additional effort on the part of the manufacturer. How do you think they got their own code on there? Apple has just not spent effort in the opposite direction to lock it down in this case. That’s not something to celebrate, we should demand it as the bare minimum, ideally via the law.
It literally takes effort to reach Apple's level of platform security while simultaneously having escape hatches designed to allow custom code without compromising the trust in the first-party code. Even simple, arbitrary decisions like having the secure boot state be per-operating-system rather than platform-wide take effort. Keep in mind Apple designed everything from the ground up including the silicon, so they did not just pull a part off the shelf that already can execute any code and then lock it down, they took their already very locked-down iPhone/iPad SoCs and specifically re-engineered the chain of trust to have these escape hatches while preserving the trust in macOS. Very careful engineering and not the "default state" of anything.
There are a lot of shortcuts they could have taken to leave us with less freedom and they did not take those shortcuts. For that, I am grateful.
Very true - for most of these with the exception of refugee visas (which we have stopped processing except for Afrikaaners), there is a requirement for financial sponsorship: someone the IRS can bill/garnish in a meaningful manner if you receive any government benefit within ten years of your arrival in the US.
reply