I guess there is a bit of a realization from the laymen (and I'm speaking as a member of that group) that a lot of these problems are toy problems.
It's clear of Fermat's last theorem. But I figured that the Millennium Prize Problems (of which there are only 10) would have some real spillover into either physics or chemistry.
Yet when one of them gets solved, it seems to be that the argument is, "yeah but not like that".
Imagine if an AI solved cancer, or a vastly cheaper way to get into space, we wouldn't react like that.
As per the post, this mathematician has been working on this problem for 20 years. So either he was "just" about to breakthrough and this is a big coincidence, or Astra was able to push through the remaining block of 5-10-20-never years it might have taken.
That's still a pretty big marker of competence in my eyes.
The point of controversy seems to be who gets credit
To me that is not a credit thing because this removes a piece evidence for the ability of AI to come up with novel ideas while still making it a useful tool.
The question's not new. In the early 1900s, women could not become PhD astronomers. Yet two women (Payne with stellar composition and Leavitt with cosmic distances) made fundamental, essential contributions to the science. Credit mostly went to male astronomers. The same might be said of Franklin and DNA.
It was nearly a century before the stories of all of them were revealed to public history. That the discoverers were not all equally rewarded is unjustifiable.
With regards to Franklin and DNA: the credit went to Watson and Crick because they had the fundamental insight: that DNA is an antiparalleldouble helix (Franklin knew it was a helix, but not an antiparallel double helix, which is key to the function of DNA). That data was shared in a departmental seminar. Further, she is explicitly acknowledged in W&C '53, and further, is the author of the paper immediately following W&C. She was never qualified to win the prize.
The big LLM providers, desperate for good PR before their IPOs, are all actively looking for 'almost finished' hard problems, e.g. where the conceptual / creative parts are almost done and they only need to throw their VC-backed resources at to brute-force through the remaining computationally expensive problem (lean, etc) and claim 'they have solved it'.
It's an utterly disrespectful, exploitive process, but all in line with exploitative predator capitalism of the stock market and big companies, now exploiting the knowledge / academia domain for scraps with a thin veneer of 'for science' PR.
Can someone ELI5 how LLMs are manipulating environments like this? I have a similar question for when you see the blender examples online.
AFAIU the models are multimodal, so you could screenshot factorio and let it come back with some manipulation like move west 5 tiles, do that and then take another screenshot, but that seems wildly inefficient so I'm sure i'm missing something
Simplest general way is literally asking the model for the X and Y coordinates to click, given an input image. Tends to work with most multi-modal models, although some have better accuracy than others, obviously.
More involved integration, would be to expose reading and writing to the game state or character controls via tools that the LLM can use (either in your own harness however you want, or via MCP). This is basically what most "AI tooling" connected to editors do today, one way or another, with fallback to "click on this UI element after taking a screenshot" often, or "simulate click on this widget with #id".
So given a goal, the LLM would first call check_inventory(), then craft_building(:building), then walk_to_coords(x,y), then build_building(:building) and so on, which tends to indeed be a lot faster than doing it by screenshots and inferring coordinates.
Ultimately I think the harness and prompt matters more than the model here, but I guess what they're exploring, is the shortest possible prompt to have it do it correctly. Otherwise, setting your harness to "Continue until you've launched a rocket in the game" together with a bunch of added rules to the prompts as you discover where it gets stuck, is an easy way to probably get most LLMs to be able to do this correctly and relatively fast. Some of them requires more handholding than others, which I guess might be another thing they're testing for here, it isn't 100% clear what the intention/motivation nor exact environment was/is.
Edit: As fitting as it is, I was just trying out the Unreal Engine Editor MCP/tools for the first time with Codex, which ended up also being a good example for how a "real turn" would look like, with a LLM connected to some other editor: https://gist.github.com/embedding-shapes/b16c50ed591569d3c70...
In this case yes, it's a custom harness that is able to read the game state via LUA modding API. But control wise it sends standard InputActions - in Factorio those are internal representation of the keyboard and mouse inputs, so it does not have any ability to programmatically create things in game.
There are people much smarter than me to answer this, but the basic idea are that models have a harness around them that include something called "tools." Each tool is basically a text description of its functionality. So for something like this, for example, the model might have a "move" tool. All of the tools are added to the model's context. Then the basic idea is that when the model detects it needs to "move west 5 tiles" it will spit out a bunch of JSON to "call" the tool, and then at that point a normal program picks up that tool call output and actually performs the action. So in other words, the LLM isn't directly manipulating the environment, it's spitting out JSON that tells the harness it is in what to do.
Factorio has an extremely advanced modding API, so the codex harness has access to the game data via the client side modding LUA api to read things, it also uses the same api that is exposed as MCP tool to send Input Actions - input actions are internal Factorio representation of mouse and keyboard.
It also does use screenshots sometimes to understand what's goiung on, so it's not just the modding api proxy.
Inputs wise it plays exactly as a human does, there is no difference.
I agree, but I also long considered llm's stochastic parrots. Then this year happened.
Opus/Sol are easily far smarter programmers than I, and this thing supposedly blows them out of the water. Once an LLM is a better doctor, researcher, biologist, chemist, mathematician, physicist than any human is that not AGI?
It didn't arrive in the form I would have ever imagined, but it's hard to say its not (imo).
I think you misunderstood the emphasis there - yes Europeans love football.
The point is that it's so blatant for such little gain, it's still a game. I.e. if he is willing to do that so openly for such low stakes, imagine what he does behind closed doors when it matters.
I'm so tired of the US trying to fight a proxy political turf war by fitting their simplistic maga/democrat viewpoints over everything.
The truth, it's pretty complicated. But since all things US blow over, the biggest help would be fix your own backyard, and then perhaps moderate discussions can again prevail.
As a European and Londoner we absolutely have to speak up and be upset if companies support someone who’s said a whole load of awful shit and got away with it.
reply