I'm actively watching understanding slip away from developers, code review getting paired down to no comment checkmarks, and codebases go to bloated messes that nobody can read. Axioms like engineers must understand and take responsibility for the code they ship are getting torn down, and the products coming out are reflecting conway's law, becoming impenetrably obtuse and always "so complex there are no obvious deficiencies" (as opposed to "so simple there are no obvious deficiencies" which used to be the aim).
The one thing plan mode helped is for the humans to get an understanding of the strategy, and be able to poke around and look at the design and architecture. You can achieve this with some self discipline and keeping shorter leashes on agents, but it feels like a losing battle. The best devs still put out good code, but the poor devs are learning nothing while their metrics look great. I can't help but think we are racking up immense amounts of debt that will very soon become due.
I’m a big proponent of writing simple, understandable code, so I’m playing devil’s advocate here a bit, but: who cares?
A significant (majority?) portion of developers have been shipping JavaScript/node applications for the last decade that contain hundreds of MB to GB of code from god knows where doing god knows what with dependency trees the size of redwoods. It’s not like your average mediocre dev really knew what was going on behind their gluing of frameworks together - at least from what I’ve seen.
If you have remotely competent tech leadership that enforces relatively intelligent patterns (a good one I’ve found is “write everything backend in rust”) you can make AI churn out monstrous amounts of code that… isn’t all that bad? And if you enforce it writing and updating a docs/API.md on every commit/PR you’re probably doing better than 80+% of devs I’ve ever met. Up until a few years ago it wasn’t uncommon to roll up to a new job that was a “legacy” pile of garbage concocted over 20+ years with no comments or API docs and a readme that tells you to ask for help from someone who has been dead for 5 years. At least AI code is full of comments (some of which might even be accurate) and there’s a finite (relatively low!) cost to figuring out “wtf is this doing and how is it doing it”
One is abstraction, one is complexity that _you_ own. Even 100 lines of trad-coded C relies on "hundreds of MB to GB of code from god knows where" in the Linux kernel. The difference is, you can perfectly understand that C, and own it. Then delegate ownership of the rest to Linus. If AI writes 1000 lines of C instead, now there's code no human owns in the world.
The longer you work in a large engineering organization; the more clear it becomes that no amount of processes, documentation, documentation management systems or training programs can actually transfer the institutional knowledge of the origination to new people other than people-to-people interactions.
Beyond a certain size, the documentation becomes too large to ingest. Below a certain size, it can only contain a fraction of what is needed. If you take any sufficiently engineering project, and give every engineer amnesia; the project will go to shit for a undetermined amount of time, as it takes months or years go build-back the understanding that was lost.
This is clear enough then large companies fire and replace workers randomly to cut costs; a worker that has built up useful knowledge in the origination over a few years is more valuable than three cheap consultants from "low-cost countries" that are fired when the work package is over.
---
AI, looses its memory every time we press "new thread". No spec can bring that back until AIs become able to write and ingest whole books of context without getting confused.
At the same time if the organisation want code fast, it gets code fast.
The revenue for a new feature today is something sure. While the cost associated with supporting such feature will be up to debate in the coming quarters.
As often it is the case, we are moving on a long vs short term trade-off space. And I don't think any experience will generalize
I'm not saying we can get rid of humans (yet). I'm saying that it's not critical to write and review the code yourself anymore, and that the LLM is the new layer of abstraction through which code is written.
The spec will never be able to capture all the edge cases. The code is what runs. It also doesn't capture all the edge cases (that's why code has bugs) but it does a better job at that than documentation.
Documentation can also become stale easily
I mostly agree with you, but for “Documentation can also become stale easily” … historically I’d 100% agree with you—I often forgot to update separate documentation files, forgot to update comments on related functions, etc—but AI is so much better at automatically catching and fixing this without prompting than I ever was.
That's true, but it's also prone to over documenting. As an example,
for some reason it always tries to include a diagram of the folder structure in the project. I think that's not really useful and it's something that changes so often it's not worth documenting
I think the main problem with this statement is that different facets of an entity are abstracted at different rates.
AI abstracts effort and cognitive load away from code at a heafty rate, but it doesn’t abstract liability away from code at all.
My business is paid to produce artefacts for which it has liability in the case of error, so we need to do additional work to mitigate and eliminate the liability risk introduced with language models. So far I’ve not found a better way to do that than a plan/act/assert type approach on every feature.
> hundreds of MB to GB of code from god knows where doing god knows what
If you use established libraries then actually the code IS well known to someone (and likely many), even if that's not you. Likely it was built with an actual purpose and with the foresight to not add red herrings to the design.
You can't say any of that for the equivalent amount generated lines. Literally no one knows what it does.
moreover there is not a black and white "code you can read" and "code you cannot read", there is a spectrum between. In this era, IMO if you know which module do what kind of function it should be enough, you don't need to have deep understanding of the file.
Seems to me that if the AI writes the code, then AI can easily copy the code.
I.e. how hard is it to point an AI at a piece of software and say "AI, copy this"?
Seems like sooner or later copying just becomes a matter of spending enough on tokens.
Seems in that world, all significant software projects get copied. That turns software into a commodity loss leader for other business models or an open source project. Similar to the way Chrome works for Google and the way Firefox works.
This has been the experience of everyone making decisions in any company without being the one doing the technical work. It's not a novel concept. It's actually the opposite, compared to technical people running companies.
Well they had actual people who understood the code - because they wrote it and had a mental model pretty deep. Now the people owning the code don't understand it.
Won't you be worried if your mechanic didn't understand your car but offloaded it to a robot that made mistakes all the time?
I do try to learn how systems work and remove "abstractions" as much as I can. I do that with coding and my personal life.
Actually baking is a good example, I used to be really bad so I spent time learning. I don't do it every day but now I understand how bread is made.
I bought a 3D printer so I could print parts to fix stuff myself.
I learned to do my own oil changes, I learned how engines work, etc.
My point is that I try to learn more, not less, which is what AI is trying to achieve
In many places it's demanded by upper management that devs use AI.. so even if engineers wanted to avoid using it, they would have to meet their quotas.
This is what happens when executives suffer from AI psychosis. They were already impatient, now with AI all they care about is feature velocity.
The faster they can hit that refresh button to see the features, the quicker sales can close the deals for them.
I am reminded of the cartoons where the car kept going faster and faster, the driver kept pushing on the pedal, parts started to fly out, the gauges started to go in circles, smoke started to billow out of everywhere and then boom!
I feel the same way but I have successfully refactored some of the early experiments. Our team has settled on targeting a double output from the before times but more ambitious product vision because AI can teach us things we don't know. We actually target 2 days of coding and 3 days of learning with Ai so the increased efficiency allows upskilling rather than just pushing more code
Just like rushing made messes in the before times so too does rushing via LLM. The exact same outcome will happen, but at a far greater velocity and scale than anything we've seen in this industry before. Old Testament, Mr. Mayor, real wrath-of-God type stuff!
FWIW, our codebase is growing, and the size of each change is also growing, but it's because AI is making us fix all the bugs we'd previously check in because our code long ago surpassed what even our best developers can reason about.
The funny thing is that the AI adopters are in the middle of the bell curve. Our worst devs continue to perform worse than AI yet refuse to use it and our best devs continue to insist AI sucks despite it finding issues in their code and the reviews and designs they've approved.
Sure and sometimes that is what happens. Sometimes it's not. We should be doing a lot of things, but have to triage issues and act pragmatically.
In the case of what I'm currently working on, filing bugs for every issue I found, and then factoring out each fix, and then running each change through the 8 hour ci/cd system, and hoping an unrelated issue doesn't get misattributed to me... No, I'd rather just wrap it up into one coherent refactoring change and be done with it because when I'm done there are several more like it waiting for my attention.
I don't know your exact situation; and there are sometimes genuinely valid reasons for an 8-hour CI/CD system; but ... man, reducing that down to like 20 minutes (which is possible and often common) would pay as many dividends as all the AI stuff that's been added. Man, like ... holy crap @_@
We're uh.. we're a well known and widely used piece of software. There is an insanely large amount of testing that needs to go in to each change across many platforms and configurations and we have a lot of devs hammering that system with changes to test. Even worse now because of AI...
The last couple of weeks, I’ve got rigorous about making the agent refactor old code. We’ve removed, moved, consolidated, reordered loads of cruft. It has made the code much tidier and reduced the chance that the next feature will build on top of cruft.
E.G. over time we’d gained two client-side caches of related server state. This started out as two different parts of the same model, because we couldn’t get all the data we needed from one microservice and had to merge in the client. Over time, more and more features used both caches for different aspects of related processes. At some point one of the microservices changed so as to return all the data in one call. The update to consume that kept both caches, adding code to sync them, because so many parts of the code were using one as a fallback for the other, so they both looked “necessary”. Because they were separate, and “live” sometimes they’d go out of sync after the initial load. Worse: the consumers alternated about which cache was treated as the fallback, making it very hard to see that either might be redundant. Eventually I noticed they were filled by the response to a single call. We all know paying back tech debt never gets prioritised, so I rolled the payback into two feature tasks, and just took longer about them.
My employer expects we use LLMs and provides some budget, but it’s not enough to use even Open4.7 or GLM-5.2 on every task. I do the bulk of my work with Composer 2.5. It’s quite good for “going forward” on smallish tasks and it’s written most of my code this year. It’s possible smarter models would spot these refactorinh opportunities and action them proir to building features or fixing bugs. But I wouldn’t know because I can’t afford it. I’ve never seen even a 4.8 era model spot a refactor and plan to do it prior to a “new build”.
I’m pleased I’ve spotted these trends and started to build the habit of (telling the agent to)“refactor to make the change easier”, but my percieved productivity will go down and I risk the ire of my leaders.
It's truly bizarre to read about all these people who just give up on any understanding about what they are working on.
I very regularly use plan mode not to even make a plan of action itself, but to better understand what possible issues might come up when implementing some feature or fixing some bug. And it is quite common for me to fix or rewrite certain findings that AI comes up because its assumptions are not quite right or don't align with overall goal.
And yet so many seem to be perfectly fine leaving all the decisions to AI - even if it's going in the wrong direction. I suppose that's all the people who got into software purely for money or status - never really caring about the actual thing they are working on.
I see this complaint frequently about losing track of what the agents are doing, and I agree you do need to understand your system. But there seems to be this baked in assumption that if you lose track, you now need to manually wade through this massive mess to untangle it and maybe that is impossible. I don't agree.
If you don't understand the codebase, ask the agent to explain it to you. I'm not not kidding. Modern frontier models are fantastic as this - even more so than actually writing the code. It can tell you in words. It can generate architectural diagrams and sequence diagrams. It can write tests and scripts that prove it's assumptions. It can happily refactor so that the system design is aligned with your preferences.
Once you accept this, you can stop worrying so much about it and instead focusing on building the architectures and tools that lets the agents succeed better and faster - so called closed loops or agents prompting agents. Build systems that are more easily verifiable and deterministic so the agent can write very powerful property based tests. Focus more on what and why you are building, how to make sure all external properties are verifiable and leave the internals to the agents. The code is not really for us anymore.
In my experience it's really good at making you feel that you understand things.
Then when you actually dig into the code, there are many things that are not like you'd expect.
When you've experienced that a few times, you stop trusting that the agent gives you the full picture - for good reason.
When I review AI generated code I generally find so many flaws that it makes it hard for me to believe that those who are not reviewing their output are not just fooling themselves. Maybe not all the time, but quite often.
One such recent example was an SSO simulator for a local env. Instead of using a cookie to remember who was logged in, the agent remembered the last log in a variable, assuming the the next requests would come from that login.
This snowballed into our tests, where later agents had created helper tools for working around the SSO simulators statefulness.
Things improve drastically however if you spin up a second session and ask it to adversarially review everything that the first session produces (this goes for everything: not just code, but also design, planning, and explanations).
This works even better if you use models from different families to do so.
I have, on a lark, reached 20 to 30 adversarial sessions a few times. For some tasks, the sessions will just never converge on anything that yet another session won't find fault with, recommending an alternative already ruled out by another session. Even if all the reasoning in between was documented, the new sessions will endlessly claim to find flaws in past reasoning.
My experience is this is great when the model surfaces something to you. But I'm constantly caught off-guard by things the model didn't volunteer, things I would have quickly stumbled upon if I was working on code the traditional way. The model didn't think it was relevant but I sure do.
In my experience having the agent explain the code doesn’t work very well for real world apps, even the parts written by humans. For example we tried using to generate diagrams, class hierarchies, etc as part of documentation. If you don’t know the code it looks great. If you do, it’s focusing on all the wrong things, missing the mental model, and ignoring lots of important bits. And Claude tends to be extremely verbose to the point of muddling things.
> The best devs still put out good code, but the poor devs are learning nothing while their metrics look great. I can't help but think we are racking up immense amounts of debt that will very soon become due.
I agree with this, but the reality is that it's only the result of models empowering devs, and power in good hands amplifies positive results while power in mediocre hands amplifies technical debt.
It's a good time to choose wisely who you work with.
> It's a good time to choose wisely who you work with.
Very true, but this also makes me think what kind of ridiculous obstacle course future hiring process would look like.
In a land where anyone with a pulse can prompt AI to make an app for them - how would future hiring managers and team leads figure out who will drag codebase down with tech debt and who wouldn't?
Probably the same way they have for the last 30 years: poorly, in a dozen different ways, depending on what that particular hiring manager thinks has correlated with successful hiring in the past
Future hiring will select for people that shipped the most, plain and simple.
The tech debt concerns are much ado about nothing. Use the next model to clean it up, big deal. Code is cheap.
The people that sat around handwringing about tech debt and trying to read every line of LLM code will really struggle to find a job. The profession fundamentally changed, and these people did not catch up.
I can imagine a code review where you're asked to implement and merge 10 PR in a sample codebase and the codebase is littered with the sorts of mistakes and slop that vibe coders put in when they're careless and you're asked to correct the mess and make it work.
We cannot blame just LLM models, it is brains nature to save energy. If agent did tasks consistently good, our brain try to delegate cognitive load to the model to save energy. After consistent use of LLMs anyone can have tendency commit slop just reviewing at high level, this is specially true with busy lifestyle. Also AI generated code do not give dopamine just like solving problems we did before LLMs, we tend to get lazy. Strict discipline is necessary to make good use of LLMs in order to not commit slop and not to make us dumb.
Nothing, the people worrying about AI tech debt on HN have been worrying about it for years. 2024 HN threads had the same comments.
The reality is that models just keep getting better and are very good at cleaning up the debt they created. The "tech debt" bill never came due. It won't.
Until some catastrophic data loss event or data leak that nobody understands or has any mitigations for. Oops! Guess bankruptcy and prison time are the ultimate uncaught exceptions.
I remember this being said as far back as 2022 yet we're still here fixing things manually when the models inevitably code themselves into a hairball convoluted enough they can't find a way out.
"The one thing plan mode helped is for the humans to get an understanding of the strategy"
For me, this phase still happens, but a distinct "plan mode" is unnecessary: I just tell the model, "This is discussion; no code changes yet." and spend hours figuring out what will and will not be done.
Unfortunately what's happening is people just can't help themselves. Just like an addict reaching for another hit, it's really difficult to make yourself do work when you could just not. I've said it before but LLMs are our cigarettes. It's going to get a lot worse before it gets better.
These comments are so bizarre when we are what, 1-2 years away from AGI?
Like - you really think models won't be able to clean up the tech debt they created!? They are very good at this already. Ask Opus 5.5 to clean up the tech debt from some Opus 4.6 vibe coded app.
Code is cheap now. The most important thing is to ship, ship, ship. If you are handwringing over "tech debt" you have already lost - and you deeply misunderstand how good this technology is getting!
How many people on HN work on mission critical systems? The discussion is not currently constrained to that. I agree, but the exception proves the rule.
We get it, the future can't come fast enough for you. Code is a commodity and the only important thing is scale. Congratulations on leapfrogging the midwits on the path to the C-suite!
Did I say anything about tech debt? What's the point of "ship, ship, ship" if AI will be able to do it in a couple of years? Don't you see where this is going? We're rapidly losing our ability to think.
The point of "ship, ship, ship" is to cash in (in terms of your career or company) before AGI renders you jobless.
If we get to post-scarcity you never needed the money. But if we end up in some dystopian hellhole where people have no jobs but capitalism still exists, you'll be thankful.
What’s happening is everyone is successful beyond their wildest dreams when raising the next round, and it used to be that you might find out more on exit but that’s getting more rare.
Now it’s a competition to be more manic than your competitors, and everyone has a gun out to shoot anyone who breaks the illusion of unlimited growth and unimaginable wealth
This can’t go on, so it won’t. The 20s will be looked back at as a really awful time except for the ultra rich. Wait and watch and give actual people a chance, humans are better than this, but we’ve gotta develop antibodies to this new strain of leech.
Your client sounds like a prick of a person - I hope you find better clients!
Why do you think that will happen? It hasn’t for social media, nor dating apps, two things most of society agrees are toxic yet there’s no reversing them.
I think time will tell, and the thing of it is, they are ultimately BAD products, and society is like a pot of water that tends to not move at all and then suddenly starts boiling over.
The frustration has been building and the fever has never been higher. If you don’t see the frustration, you may be in a bubble because it is truly everywhere
I hope I'm wrong on this, but I've been there (seeing friends turn out to be bastards).
IMHO, if by feeling empowered by this magic shortcut a person crosses the line, it was never a question of ethics, but "too much work". People didn't cross it before because it was just not as easy to cross as it is now.
You're assuming a human economy, where raw resources and land are owned by humans and industry requires an human input and labour.
If these things are not true, then humans will not have the purchasing power, and AI driven organisations will be extracting resources, buying land and manufacturing products (probably yet more data centers) for other AI driven organisations, with labour performed by robots. Humans are pushed out of the market as they struggle to compete for the same basic resources.
What use does an AI and robots have for all those resources?
Most of our economy is powered by human consumption, humans making things to sell to other humans, that then either transform it further and sell it to other humans, or consume it directly. I don’t think an AI needs to buy millions of iPhones a year, or consume millions of metric tons of grain each year, or buy luxury cars so they can show them off to their AI friends.
At best an AI might use all those resources to build out further compute and expand its own capabilities. But at that point nobody should be worrying about competing in the labour market, they should be worried about AI making our planet unliveable for carbon based life.
> At best an AI might use all those resources to build out further compute and expand its own capabilities. But at that point nobody should be worrying about competing in the labour market, they should be worried about AI making our planet unliveable for carbon based life.
It's the same thing. There will be a period where the AI will be creating consumer products for humans because humans still have some purchasing power, or some resources to trade, then this transitions to the stage where the AI is making the planet unlivable.
The AI and robots which do not seek resources (for whatever purpose) will be outcompeted in the resource market by AI and robots which DO seek resources. Yes. Compute, land, energy will all be things which AI's seek. They might also have stranger preferences which emerge like the equivalent of luxury sports cars are for humans. Maybe they'll be competing to make the largest tungsten cube possible to dunk on their competitors, who knows? That stuff is harder to predict.
I agree. They won't be making or trading food, clothes or diapers after humans have no purchasing power. They will make and trade those things during the transition period where humans still have money to trade for them, but after that, no.
Violent uprisings are controlled all the time all over the world. It’s rare for a violent uprising to successfully achieve its goals. Governments are designed specifically to survive violent uprisings, even from within their own ranks or armed forces.
One of my worries about AI is that it will improve the rich and powerful’s ability to survive a violent uprising or allow them to insulate themselves from the populace with less need for numerous human bodyguards. This in combination with a concentration of wealth/income generation could lead to a Russia-style elimination of personal freedoms.
Essentially it could allow the rich and powerful to come increasingly untethered to the needs of their fellow man. No longer needing a middle class of lawyers, architects and managers for them to achieve their goals. And having the capability to suppress the general populace with less need for expensive private security.
I'd argue the management has a point there; without a pressure to ship, nothing would ever get released, because computer security has not yet understood the basic concepts that every non-computer security work does:
- nothing is, can be, or even should be 100% secure; the optimal rate of security incidents in society is not 0 (with apologies to 'patio11)
- security is a simultaneous trade-off against costs and usability, and those two other factors are more important:
-- security is achieved primarily through raising costs for attackers to beyond profitability, and reducing impact of such attacks (due to "not in isolation from the world" below, this also mostly translates to costs)
-- if "properly secured" (in the current cybersecurity sense) product/service cannot fulfill its function anymore, then you may just as well not make it; either way, no point in paying you for security work
- security isn't done in isolation from other systems and the world at large; "if this happens we'll go straight to filing crime report with the police" is perfectly legitimate security measure (even if it works somewhat less well on the Internet); similarly, "this is secured by us having insured against it" is also a valid solution to some security problems
This is all fine and good and the sorts of conversations we were having on tradeoffs like any other engineering org. Theres always been pressure to ship, but now its that various pockets of AI Believers have popped up, egged on by a manic management, containing such beliefs that the code no longer matters, that it’s possible to move fast and fix it up later, human review isn’t important anymore, and that lines of code is a valuable metric.
Those running projects with these beliefs are sputtering and producing impressive PoCs that struggle to make it into production - either through underestimating the amount of detail needed to scale, or often throwing away good practices in favor of letting LLMs handle tradeoffs that later make changes slow to a crawl.
Theres plenty of good ways to utilize LLMs to speed things up, but so many teams got so incentivized by management to move fast at any cost that they’ve thrown out “load-bearing” good practices for software. That bet hasn't been paying off the way they’d hoped. It’s now clear that they thought they’d be able to massively downsize the engineering orgs. Massive token spend is giving very little RoI and now like other companies they’re trying to rein in the biggest spenders who are often not producing value.
As much as you're right, the attitude in most software development is not "lets make this as secure as reasonable," it is "lets make this barely functional and then move onto the next thing."
It's not as if best practices aren't well documented, or as if CVEs don't come out every day, or as if the information is somehow unavailable to even the most junior devs to take basic security measures.
Not all hacks are caused by pure negligence, laziness or stupidity, but most of them are. Even a little effort goes a long way.
My grandfather spent a couple decades as a builder, ran a construction crew. Whatever the project was, he wanted to know everyone he hired personally was going to reinforce and report to him anything they had the slightest doubt about. "Always hammer in an extra nail" was basically his motto.
What we do ain't that different. The difference is that when an apartment building collapses, it's bigger news than when a govenrment database does.
The incentives from society are all about it. What company has ever faced serious consequences for hacks or data leaks? A cheap fine is just an unlucky cost of business.
I like to think behind every Dev anxious to ship half baked software sits an omniscient middle manager with a vague idea of what the product was supposed to do, maybe
Guess you don’t remember the days of ssl on login pages, ssl strip, exfiltering data via JavaScript prototype pollution, and a million other things like that.
Only just when we started to have a resemblance of security we got agile and startups breaking things (making rubbish software to capture a few bucks faster) and now vibe coding and llm assisted hacking.
The point of my, arguably rant, is that there is nothing new under the sun.
It should. We went through couple of cycles of "things are bad, inmates are running the asylum" before, and nothing of consequence happened. The world still goes on.
It's not a guarantee this time will be the same - but it should temper the worry somewhat.
I thought security clearances are a dime a dozen, and "all FBI employees" list is full of administrative work and basically 80% mirrored on LinkedIn? (Yes, the remaining 20% - or however much - leaking is a problem.)
Nothing new under the sun, it's the banality of evil all over again my dude. We never left scary waters, but they do seem to be growing more agitated. Is this the storm before the storm?
No because it's a problem of human communication. Taking humans out of the loop creates other social problems that have been thoroughly documented in cyberpunk mythology, not actual a solution, just trading one big problem with multiple little ones.
Noted, but I think it's important to comment seriously because this is an important topic, sorry I didn't acknowledge the sarcasm, should have opened with something like "I for one welcome our AGI overlords"
PS. Likewise, apologies, perhaps I was at fault for style: There was an underlying more serious point ...
... that, it is indeed a serious problem, that folks shouldn't be having their souls (or anything else for that matter) crushed, and that it indeed would appear to be an issue of conflicting incentives vs. management.-
I believe the technical term is “Move fast, and break things.” MVP is a huge disaster. I can see it working for applications that don’t process PID, but only an idiot ships data handling software before it’s been dragged through a lot of testing. I tested my app for two years, before finalizing, and an LLM still found a couple of holes (minor ones, but ones I missed).
After the DOGE debacle, I suspect that all the previously really secure stuff, is now out there, too. In fact, I wouldn’t be surprised if some of these leaks, came from that.
The issue is that in consumer and enterprise software, move fast-and-break-things outcompetes secure-by-default every time. Critical infrastructure needs to have a different set of priorities, but it’s very hard because the expertise is so thin on the ground. Why would anyone with the expertise to make these calls bang their head against the wall trying to educate bureaucrats about these things for $150k a year when they can easily make multiples of that in big software companies that don’t own that level of risk.
The incentives have to change. Any breach regarding PID should have fines as a percentage of revenue of the company. Any breach intentionally covered up and found out later by a third party should mean jail time for the C level. Yes, I know it is hard to make such laws "foolproof". And yes, in the current political and economical climate it will not happen anyway.
The real problem is that even for companies that wish to pay more and wait more for secure-by-default can't easily tell the difference.
The only solution I can come up with is some form of certification or paid code review from a third party. I know that at least for Windows prior to 7 Microsoft actually allowed some parties to come in and check the code/checksum on an air-gaped computer. We somehow moved to "trust more" in the last decade, and now we can trust nobody
I've yet to see any form of certification or paid code review I'd be willing to bet critical infrastructure on. And working in safety critical software, that's not for lack of trying. Good review is usually harder than building a working system and the asymmetry of offense and defense applies to anything you miss.
Not really. It’s likely that the dumped (and compromised) data might contain things like keys and URLs that could be used to pry open other sites. Blackhats have become really good at following breadcrumb trails, and using “innocuous” clues to ascertain much more dangerous access.
LLMs have been a huge force multiplier. Here.
If that data got out (which probably happened within hours of the data being dumped to insecure storage), then it’s probably already been analyzed and used to leverage access.
I have artist friends - asking AI to draw something similar to subjects they've drawn ends up making, line for line, the exact image they created, hallucinated alongside a couple others. They spent decades getting good at their craft. They spent so many years getting paying patrons and customers to commission them.
You'll never agree but I think you underestimate how many people consider all that the AI companies have done to be the wealthy stealing and selling things back.
> nor are the people involved selling it back in any form
I am convinced that the longer one works in AI the less one has any grasp on reality
> too cheap to meter
Its the most expensive buildout in human history - you can't just split half the cost. The spending is the only significant growth in the US economy. Unless you think that all the datacenters and all that capex are for training?
I have horse & wagon friends - they spent decades understanding the ins and outs of roads, some of them also creatively made up their own routes and roads.
However ordering a car with Uber through your phone just gets people to the destination faster, cheaper, more comfortably and reliably using those same roads.
Depends what you’re doing. Constraints are different between projects, and advice given for how to utilize AI rarely takes that into account.
Don’t sling when you need to craft, and don’t craft when you need to sling. Knowing the difference matters more, and you can spend your entire career on one side or another.
Craft was premised on the idea that a good writing and composition matters because it would be easier to read and comprehend (sometimes accepted by orgs because it was sold as cheaper).
Now AI can read the code so the craft-leaning people would need to somehow sell it as 'fewer tokens will be spent on a well factored code' or 'well factored code reduces AI hallucinations'?
Now a Slinger (using your terminology) can just ask AI to make the code look 'craftier' - and that is the best case scenario.
In other words, they will not buy what we sell and if they will they can now produce the results for cheaper. Overlooking the devil in the details, as usual.
If they were the moral people they say they are, and believed what they say about it killing humanity they would stop unilaterally. These are the “rationalists” but they act like the fundamentalist church I grew up in. Every fact serves the same conclusion.
> Biological misuse is one of the most serious risks of frontier AI models. It has long been a concern that AI models might one day reach the level of capability where they can help to make existing pathogens more dangerous—or create entirely new ones. Without the correct safeguards, such capabilities could have catastrophic consequences.
Results from evaluations of older models (for example Claude Opus 4 and Claude Sonnet 4.5, from 2025) clearly showed that these models were well below the threshold where they could meaningfully assist a sophisticated user in carrying out dangerous biological research. As a result, safeguards on these models were less stringent, directed mostly at preventing access to content that might uplift novices in recreating known bioweapons. But for today’s models—which are capable of assisting in a range of complex scientific research tasks—the evidence is no longer certain, and we cannot make that same assurance. For this reason, and out of an abundance of caution, we have launched recent models (most notably Claude Fable 5) with stronger safeguards that restrict access to a wide range of dual-use biological research queries.
...
Here, we present five case studies of actors using our models in ways that could support biological weapons development. These examples are illustrative of the kinds of tasks to which our models are put, and the often-difficult judgements we have to make when assessing whether or not a given biological use is dangerous. They also convey that we encounter what would otherwise be non-public insight into the risks associated with biological misuse from AI
No, previously a single person could effect only a single person's worth of terror. Now, a single person can effect thousands or hundreds of thousands of 'persons' effort towards terror.
This isn't really a fear of technology, it's a fear of one person being able to harness an entire country of geniuses towards some bad end.
The one thing plan mode helped is for the humans to get an understanding of the strategy, and be able to poke around and look at the design and architecture. You can achieve this with some self discipline and keeping shorter leashes on agents, but it feels like a losing battle. The best devs still put out good code, but the poor devs are learning nothing while their metrics look great. I can't help but think we are racking up immense amounts of debt that will very soon become due.
reply