> Why have we not seen an improvements in products? [..] Is everyone just running full speed in circles or something?
The simplest explanation is that they don't give a flying flamingo about what you or I consider "improvements to products".
This report is an example.
There are several changes that modify CI behaviour, where the article gives no corresponding quality measurement.
They replaced type aware custom lint rules with AST-only static analysis. They don't say anything about what those new rules detect, didn't do old-vs-new rule comparison. They switched the TypeScript check from tsc to tsgo. Again, they are very proud of the performance improvement, but don't seem to care about diagnostic equivalence. The list goes on. They don't even report pass/fail agreement between the old and the new CI. They have 4x more tests, but no idea whether this big test suite works any better than the smaller old one, or even whether it works at all.
Are we holding it wrong? Mine does the same godawful external library tests. When it is having a particularly stupid day it also tests language features. For example, checking that using a callback executes the code supplied in the callback (it does).
Yeah, that's what not so many are talking about but a few have pointed out; it's the AI writing wayyyyy too many tests causing the new CI load.
Unless you prompt them otherwise, the models tend to write WAY too many useless tests and in the most inefficient ways imaginable. This balloons test counts and lines of code to insane levels and frankly, likely, slowly makes it more and more costly for the AI to make future changes.. To the point it can't wrap its context around the code base and effectively make necessary changes.
Why not? If you let the LLM run amok, you get 4 one line functions calling each other instead of one 4 line function. Run that for 24 hours and you'll get more than 2000 tests.
> imagine yourself living in the 1700s. how would you justify Newton and Leibniz's work on calculus?
They didn't have to, as there were no state grants or public research funds for mathematics during the 17th century.
Newton supported himself from his inheritance throughout the Great Plague, while he invented calculus, and then from flat salaries as teacher, then flat salaries from working at the London Mint. He later became fabulously wealthy after his appointment as Master of the Mint.
Not so relevant to this discussion, since if you are funded by patronage networks, you need to justify your work to your patron. If you are funded by grants or public research funds, you need to justify your work to the general public.
They did that... I take it you don't accept their justification, but that doesn't mean an attempt wasn't made.
I'm another part of the general public, and I accept their justification.
We can all vote, or use whatever levers our political system gives us to show our acceptance/rejection of basic research, but IMO basic research is absolutely necessary.
> The so-called "nonstandard analysis" hasn't caught on, because it's mostly the exact same arguments wrapped in slightly different language
No. Let's take a nonstandard proof of the intermediate value theorem on [0,1] by Nelson.
By the transfer principle it is enough to prove this for a standard continuous function f on [0,1] with f(0)<0<f(1).
Take a finite subset of [0,1] containing every standard point. Colour its points blue, green, or red according to whether f is negative, zero, or positive at tha point.
The first point of the interval is blue and the last red. Hence either awe can find some green point, or we can find two neighbouring points that have different colours, the first blue and the second red.
In the first case there is a zero, so we are done. In the second, let the neighbouring points be p and q. By the completeness of the real numbers, every nonstandard real in [0,1] is infinitesimally close to exactly one standard real. So p and q are infinitesimally close to some standard real number, let's call it z.
Standard continuous functions send infinitesimally close points to infinitesimally close points. So f(p) and f(q) are both infinitesimally close to f(z). But f(p) is negative and f(q) is positive. The only standard number infinitesimally close to both positive and negative numbers is zero. Thus f(z) is zero. This proves the theorem.
You tell me, which standard proof is this? It's certainly not the nested interval proof. Not the supremum proof. Not the bisection proof in disguise. Which argument does it wrap in slightly different language? Can you point to a single textbook, course note or lecture that gives such an argument?
No. One could of course argue that this is not simpler/shorter than the usual arguments. But it is very different from them. Saying that it's the same arguments repackaged in a different language is just wrong, and detracts from an otherwise valid point.
This is the standard nested interval proof, you’re just replacing the limiting step of taking smaller and smaller intervals with the nonstandard way of expressing the same thing.
I'll be honest: your one sentence response tells me you did not read the proof above in any detail.
I chose Nelson's proof precisely because its construction is well-studied and well-understood. The same construction of a mesh containing all standard points, with the coloring forcing a tiny multicolored cell, extends from the interval to the triangle. In one dimension you get two adjacent differently colored points; in two dimensions you get an infinitesimal triangle whose three vertices have the three relevant colors. Taking their common standard part and applying continuity gives a short proof of Brouwer's fxied-point theorem on the triangle.
But it is well-understood (there's a whole field studying such questions [2]) that the nested interval proof of the Intermediate Value Theorem does not generalize to proving Brouwer's fixed point theorem on the triangle [1]. This fact can be derived from a computability argument as well [3].
Nelson's argument does generalize to prove Brouwer, so it's not the nested intervals argument. But really, nobody cares about these technical reasons. It's obvious to most math undergraduates that Nelson's proof is not the nested interval proof, the clear absence of any nested construction kinda gives it away. The only reason it was necessary to get technical is that you did not really inspect the proof before claiming it was nested intervals. The technical results cited above are just a formal way to show that any correspondence you might imagine between the two proofs is just not there.
[1] Shioji/Tanaka: "Fixed Point Theory in Weak Second-Order Arithmetic", Annals of Pure and Applied Logic v47, pp 167188 (1990).
What you just described is a classic proof of Brouwer's fixed point theorem using Sperner's lemma. The proof you cited earlier does not generalize to it on its own, the Sperner's lemma is a crucial combinatorial ingredient. It's crucial, because it only works on spaces with the topology of the triangle; you cannot perform the same argument on, say, an annulus.
In the standard formulation, you apply the Sperner's lemma to find smaller and smaller triangles, and apply compactness, precisely as in the standard proof of intermediate value theorem.
The rest of your post, where you quote reverse mathematics stuff, is completely irrelevant to the point I was making. Nothing I said is about what theorems follows from what axioms, but rather whether nonstandard analysis is meaningfully different, clearer, or more useful language than standard one. It is not.
You made a sweeping claim that nonstandard analysis arguments are the exact same arguments, wrapped in nonstandard langauge. I explained that (while your other claims about simplicity may be valid) this is not so and detracts from the rest of your points. I challenged you to defend your "same arguments" claim by finding any standard analysis textbook which teaches a standard language version of Nelson's argument as a proof of the IVT. Let me recap what happened since then:
1. Two comments ago you confidently claimed that Nelson's IVT proof is "the standard nested interval proof" with the limiting step written in nonstandard language. That is a straightforward claim about the structure of the proof, one that you didn't bother to substantiate, and that is straightforwardly false.
2. After I explained why it's false (Nelson's construction proves BFPT, which no nested interval type proof can do), you changed your response: now the Sperner lemma was a "crucial additional ingredient". But Nelson's combinatorial step, that opposite endpoint colors force a blue-red interval, _is_ the one-dimensional instance of the Sperner lemma (and indeed the base case when you prove Sperner's lemma for arbitrary dimensional simplices by induction; the analytic part is independent of dimension, once you find an infinitesimal multicolored simplex, you take its common standard part and apply continuity exactly as before).
3. Then you wrote this:
> In the standard formulation, you apply the Sperner's lemma to find smaller and smaller triangles, and apply compactness, precisely as in the standard proof of intermediate value theorem.
There is a standard proof of Brouwer via the Sperner lemma, and it is _also_ not of the same form as the standard nested interval proof of the IVT. In the nested interval proof, you find a sign-change interval, then find a smaller sign-change interval inside it, and so on. The intersection of all of these contains a point, and that's your zero. Nelson's proof does not do this, and neither does the standard proof of Brouwer via the Sperner lemma: you do not, and cannot, take a 3-color interior triangle, then find a smaller 3-color interior triangle inside it and so on. Even the first step would not work, since the inherited labelling does not satisfy the right boundary condition relative to the small triangle!
And this is also why the computability paper I cited ("where I quote reverse mathematics stuff" ;) was very much relevant. There can be no effective "nested triangle" proofs of the Brouwer fixed point theorem at all, because such a proof would let you compute a Brouwer fixed point, and there are examples of computable maps on the triangle without computable fixed points. If Nelson's IVT proof was the nested interval proof, then swapping in the higher-dimensional Sperner step would give a nested-type proof of BFPT. No such proof can exist. Since Nelson's argument proves the BFPT without any change to the analytic part, it is not a nested interval type argument.
You first misidentified Nelson's proof as nested intervals, and then treated Sperner as an additional ingredient even though the coloring step in Nelson's proof is already the corresponding Sperner argument. Those are both fairly serious misunderstandings about these proof. Given this, I don't think our exchange leaves readers with much confidence in your assessment of NSA's drawbacks and benefits. That's a disappointing outcome, as far as I'm concerned. There are good arguments to make that NSA adds little value to undergraduate education, such as simplicity or the difficulty of the prerequisites, and good conversations to be had about them. But "NSA proofs are the same proofs wrapped in a different language" is not one, and I wish you had just narrowed it instead of doubling down.
I never said the triangles in the standard proof are going to be nested, so your whole segue into reverse mathematics is, just like I said, irrelevant. The point of the argument is that you can find a sequence of triangles with differently colored vertices, the vertices of which all converge to the same point (thanks to compactness), which contradicts continuity of the retraction on the boundary. The nonstandard version of this is exactly the same argument, it just replaces the explicit limiting step that contradicts continuity with the an argument that uses the nonstandard formulation of continuity in terms of infinitesimals.
If that makes it easier for you to understand it, in the standard proof, you also color every point of the rectangle, with the color of the edge it retracts to (picking the colors of the vertices of the big triangle arbitrarily, just making sure that the color of each vertex is a color of one of the edges it belongs to, not one of the opposite edges). Then, an easy argument from continuity shows that no interior point will have points of three different colors arbitrarily close to it. Finally, applying Sperner's lemma as above proves that such point must nevertheless exist, obtaining contradiction with the existence of the retraction.
and then treated Sperner as an additional ingredient even though the coloring step in Nelson's proof is already the corresponding Sperner argument.
I don't understand what are you saying here. What I'm saying is that for the coloring proof of BFPT to work, whether clothed in standard or nonstandard language, you must perform a combinatorial argument that uses a topology of a triangle as a necessary ingredient, similar in shape to the proof of Sperner's lemma.
You opened with the claim that nonstandard analysis hasn't caught on because it's "mostly the exact same arguments wrapped in slightly different language". I pointed out that the arguments are in fact very distinct: e.g. Nelson's proof of the intermediate value theorem is something that any NSA student would see, but no standard textbook teaches IVT by a standard language counterpart of it.
One post later, you answered that Nelson's IVT argument is in fact the "standard nested interval proof" with the limiting step rewritten in nonstandard language.
That claim is simply wrong. Why? Because Nelson's construction straightforwardly generalises to Brouwer, while the nested intervals proofs cannot. The discussion of reverse mathematics / computability is not a tangent, it explains precisely why Nelson's proof can generalise to give the BFPT in two dimensions, whereas the nested intervals proofs (which you claim is the same) cannot.
You then brought up that the BFPT generalisation of Nelson's argument needs the Sperner lemma as "crucial additional ingredient". Now you make the same point again:
> What I'm saying is that for the coloring proof of BFPT to work, whether clothed in standard or nonstandard language, you must perform a combinatorial argument that uses a topology of a triangle as a necessary ingredient, similar in shape to the proof of Sperner's lemma.
Presumably you keep pointing this out because you think it justifies some claim like '1D Nelson is actually nested intervals with the limiting step recast in nonstandard language, even if the 2D generalization of Nelson is not'.
But it does not. The combinatorial content is the same, the 1-dimensional interval case uses the topology of the domain just as much as the 2-dimensional triangle case does. The 2D argument wouldn't work on the annulus, and the 1D version would not work on the union of two disjoint intervals. The Sperner lemma is present in 1D, and present in 2D. If instead your point is only that proving the BFPT requires a harder case of the Sperner lemma than IVT, then of course it does. But what relevance does that have to the original claim that Nelson's IVT proof is the nested-interval proof? The proof of the Sperner lemma is pure combinatorics, it does not involve any (standard or nonstandard) analysis.
Or have you changed your mind on your earlier claim that Nelson's proof is "is the standard nested interval proof"?
If so, I think that's great, and closes the thread on whether NSA is largely the same arguments, since even the first proofs of the basic results are different.
If you still think that it's the nested interval proof, well, I am not sure what else to say, apart from linking the literature which studies this exact question, that I've already done, and that you dismissed as a tangent.
Either way, this discussion went on for too long at this point, so I won't monitor it further.
No. In the semiconductor industry, the "catch-up" player isn't normally spending less in absolute R&D terms.
Comparing the R&D costs of creating GPT-4o vs. DeepSeek V3 (the latest gen for which we already have good accurate numbers) it looks like the latter cost 1/20th as much to create.
If Samsung could catch up with TSMC for 1/20th of the cost, people definitely would say that TSMC has no moat.
That's the ratio the widely published numbers give [1]. One does not have to believe the numbers [2], but those who do believe them are then justified to conclude that there's no moat.
Which numbers you believe is of course going to affect whether you think there's a moat or not. That's largely orthogonal to your TSMC/Samsung analogy I responded to. If you think the "moatists" are wrong because they believe the wrong numbers, that's fine, but then there's no need for the analogy.
So why can't OpenAI/Anthropic also distill the good parts of free Chinese models? It's even better and easier for OpenAI and Anthropic. No poison pills as well.
Ultimately, that's what I need to be convinced. No one has put forth a good argument yet.
Clever architecture --> Ok but OpenAI/Anthropic can use these as well and they also have very smart people with their secret clever architectures
Distilling --> Ok but distilling means you will never be smarter than the original. Furthermore, reasoning is now hidden by private labs and they have poison pill answers for distilling if they can detect it. They will be able to detect distilling better and better.
Cheaper electricity --> Ok this is cancelled out by their chips being much less efficient due to not having ASML EUV machine access.
So I don't see why fundamentally their training costs are cheaper over the long term.
Mercor, Tacit Labs, Handshake AI...
I suspect companies like these play a big part in model improvements, generating high quality benchmark/task-focused data for training.
However, these do require educated, white collar, workers.
Considering that frontier scientists and engineers in the US are currently taking home seven (or even eight, in some cases) figure salaries - pretty high, I'd reckon.
I am not a "no-moatist" per se but one can argue their might be a plateau to how good a inference llm can become. If this is the case the playing field shifts to context, tools and harness, which are much cheaper to build an compete on.
Not for long. Too late to get Business now to exploit this, since EH and the small credit-free Pro allowance will soon be restricted to Premium Seats ($100/m).
I thought they just did this? People were using some loophole to use their Chat sessions to power their Codex usage after their Codex quotas had run out.
> Maybe it's because I'm American, but I can't imagine actually saying that to someone. I'd sorta expect them to throw hands if I talked like that about Jesus.
I mean, that's exactly what a "slur" is. They are not coined with the recipient's comfort in mind, and saying them can sometimes provoke exactly the sort of fight you imagine.
But yes, violent consequences ought to be particularly unsurprising to an American, since the US is, by Western standards, an exceptionally violent country. [1]
> You can tell that Claude really does grasp a wide array of highly specific scientific and mathematical nuances... where's codex is just basically for coding and that's it.
If you have time, can you elaborate or give some examples of mathematical nuances?
I am evaluating Sol and Fable on a fairly large dataset of subtly flawed informal mathematical arguments (task is to identify and name propositions with substantially incorrect proofs in a larger body of text), and Sol is saturating the benchmark, while Fable is below 50% even with the most generous grading.
I don't work in the natural sciences, so I suspect you mean something different by "mathematical nuance".
Is this benchmark public? Anecdotally I have had decent results asking Sol to nitpick my proofs (mostly probability theory but nothing super dense). I have never tried Claude seriously, so I am very curious about what the failures look like with Fable.
> If it is known that A is provably true then one can study the consequences of A being true
But one can already study the consequences of P=NP right now. You don't need to know that it's provably true in order to do that.
Knowing an actual proof would be useful, but an oracle revealing merely that it's true (or even provable) without telling you the proof does not let you do anything you couldn't do before.
Some people (almost all mathematicians) wouldn’t want to spend time on consequences of a false statement. In the present discussion it’s not about letting me do something I can’t do now but about whether or not the endeavor is worthwhile.
A lot of people spent a lot of time and effort to prove or disprove the Jacobian Conjecture. AI solved it easily. It is increasingly becoming the case that humans are not as good at mathematics as computers. You are free to ignore computer generated proofs but I don’t think this position will win out in the long run.
> Some people (almost all mathematicians) wouldn’t want to spend time on consequences of a false statement.
No, people constantly prove statements of the form "if P=NP, then strange implication X". They do not consider it wasted effort at all, because of the contrapositive: if X is indeed very strange, they might be able to prove that it is false, and then they've settled P!=NP.
If a counterexample to a conjecture is found then all work toward proving consequences of the conjecture will cease. No one is trying to discover consequences of the Jacobian Conjecture now.
At some point an AI will prove a result that is so long and complicated that no human will understand it. This should not preclude people from using that result. In general, whenever the body of knowledge is increased it is a good thing. Even if it isn’t increased by humans.
I’m not sure what point you’re trying to make, exactly, but a use case for better non-CS generators has always been stochastic simulation, especially simulation/sampling approaches that are bound by the number and quality of uniform variates per second.
As someone who has spent considerable time working in these areas, I still appreciate advances.
> especially simulation/sampling approaches that are bound by the number and quality of uniform variates per second
Sorry, nobody does stochastic simulations where the number of uniform random numbers obtained per second is any sort of bottleneck. If you've spent considerable time on stochastic simulation, you already know this.
But even if you insist that you alone are doing some very weird stochastic simulation which is somehow bottlenecked on sourcing random numbers fast enough, the falling in planes phenomenon linked above would make xorshift-type generators a poor choice for most sorts of simulations. It introduces spatial correlations into any sort of lattice dynamics simulation (Ising model, percolation) and every high dimensional Monte Carlo integration. Beyond falling in the planes, since xorshift is linear over GF(2), it is also a particularly bad choice for nondeterministic cellular automata and Boolean dynamical systems which use parity, bit masks, or xors.
AES-CTR throughput on a modern CPU is higher than that of xoshiro256++, and much higher quality. No advances in non-CS PRNGs can beat that while maintaining the same quality. If your stochastic simulation is bottlenecked on random bits, CSPRNGs are still the way to go, and they don't interact in nasty ways with any dynamical system you can actually sinulate quickly.
> Sorry, nobody does stochastic simulations where the number of uniform random numbers obtained per second is any sort of bottleneck. If you've spent considerable time on stochastic simulation, you already know this.
Actually, I spent a considerable amount of time in my doctorate and postdoc doing this.
Any kind of MCMC sampling of a simple model tends to be bound by the rate you can draw variates.
Examples of this include: Gillespie simulations of chemical kinetics, Ising and Potts lattice models (including their roughly bazillion variations), and anything resembling bootstrap or permutation sampling.
Just because your problems aren’t bound by the rate of drawing uniform variates doesn’t mean that these problems don’t exist. It just means that you have a narrow view.
I asked Vikash Mansinghka about this 15 years ago. He was using Xorshift as an RNG for a probabilistic inference on an FPGA. Why? It used few gates and was high (enough) quality.
You misrepresent the non cs ones by a factor of 5-10, even on cpu. First, you’re calling the prng twice per aes single call. That seems pretty dishonest already.
And using Golang? That ludicrously slow also.
To tell us what a cpu can do, do them using SIMD, you get pipelining and then many values per clock. Now try that with AES. Oh, you cannot, it’s not supported.
And people needing lots at full speed will do them on GPUs.
There is no world where even HW accelerated crypto comes close to non cs prngs on mainstream HPC systems.
I do a lot of testing and designing of things like hash tables and filters, and having a really fast, non-CS generator is incredibly useful for being able to clearly identify performance bottlenecks in designs. PCG has been spectacularly useful for that purpose for me.
When was the last time a new PRNG helped you clearly identify a performance bottleneck?
As in, you were using state of the art generator X, and you couldn't see the performance bottleneck, but updating to a newer (faster, or same speed but higher quality) generator Y, and could subsequently identify the performance bottleneck?
If you're using PCG, not in the last 12 years.
(In a parallel comment I suggest trying AES-CTR for this use case)
It's not critical but if you gave me something that behaved statistically like PCG (i.e., I didn't fret about whether it was going to cause me weird problems) but was twice as fast I'd be happy and would shift to it - it would speed up profiling and measuring and that would be nice. We still find ourselves often pre-generating a list into memory to keep the prng entirely off of the measurement path. It wouldn't be magic, but I don't need magic. I like nice things that make my life a little easier in a small corner of my research. :)
Most modern RNGs should be faster than memory bandwidth (when optimized), so unless your list is small enough to fit in cache, its unclear if this is faster?
dgacmu: if you're writing C on x64, try AES-128-CTR (AES-NI, 8 way) using the header wmmintrin.h which has hardware accelerated primitives for this. An LLM can implement the RNG for you based on this comment if you want to test it out quickly. It should be faster than PCG, and higher quality.
Will do. I'm on vacation right now and losing my laptop for a few days, but seems worth trying. My recollection from the RNGs a decade ago (I'm dating myself) was that the AES approaches had higher latency but were quite decent, though slower than PCG. Curious how that's evolved.
> When was the last time a new PRNG helped you clearly identify a performance bottleneck?
While not a bottleneck as such, I contributed to a photorealistic path tracer using the Metropolis algorithm[1], and we got a 10-15% increase in samples/second when we switched from a decent to a much faster and better PRNG. Like you we didn't think the performance of it mattered much until we profiled it.
Granted this was a decade or so ago, would be interesting to compare the state of the art PRNGs.
Anyway, just pointing out that there can be real-world cases.
As others here have pointed out, this is nonsense. The vast majority of PRNG calls on the planet are extremely high perf simulations, where crypto secure versions are a ludicrous cost in speed, energy, and sheer stupidity. That you and others do not understand is simply because you don’t see the places it’s required.
I’ve a PhD, have written papers on PRNGs, have worked in both cs prng and high perf prngs, have done decades of HPC projects, scientific sims. I get called in to develop precisely these high performance systems, and when you want to replace trillions to quadrillions of PRNG calls with one costing 10-1000x more, you’d get deservedly fired immediately.
You keep arguing about AES style code on a CPU. That’s not where people do high performance code. Try implementing AES and a fast prng on a GPU. You’ll soon find out how absolutely terrible cs-prngs are at performance. The measuremt isn’t how many ns per prng. It becomes how many thousands of prng generated per ns.
It’s bafflingly shortsighted for people with zero work in this area to continue to argue this. Choose the right tool for the job. Don’t project ignorance as knowledge. Both are useful advice.
Part of your argument was that there can't be an application for fast uniform pseudorandom numbers not just that xorshift by itself is not a very good PRNG(which I do agree with although it is an interesting sequence).
If Intel, AMD and Apple add a xoroshiro or PCG instruction and it produces pseudorandom numbers significantly faster than accelerated AES on those architectures, how does that affect your argument?
On the other hand, most simulations have moved on from random numbers to non-random space-filling sequences so the only non-CSPRNG application would be rolling fair dice for games and even there there is an argument to be made for CSPRNGs. So, perhaps I agree with you on the bottom line.
Yes there is. Not every system has the need or the resources to maintain a secure random sequence. You may also want a reproducible pseudo-random sequence in generative code that logs the seeds. Because of the misguided attitude that nobody needs these features, everyone who does need them has to roll their own now.
> Not every system has the need or the resources to maintain a secure random sequence.
I'm sure there's something, but that category has to be shrinking every year. What does such a system look like this decade, that needs random numbers but can't easily implement something like AES?
> You may also want a reproducible pseudo-random sequence in generative code that logs the seeds. Because of the misguided attitude that nobody needs these features, everyone who does need them has to roll their own now.
I don't know what difficulty you're referring to. Basically every CSPRNG can be seeded easily and you can log the seed.
I agree! For many games (especially the ones running on old devices), xorshifts are pretty good! Not all applications need cryptographically safe generators! And sometimes it's ok to trade complexity for speed!
But I agree that if you're building something new aimed for modern devices, Marsaglia's xorshift128 wouldn't be my first choice! But I'd definitely want it in my RNG library for backward compatibility!
If you are interested in performant retro PRNGs, you can find a link to mine in my profile, if I am not mistaken. The counter is based on xorshift. As of the time of release it passed all tests that could be passed with only a 2^32-1 period.
On Z80 it was only about as fast as RC4 which is also a good option in terms of performance and quality but whereas RC4 has a huge state, this one only has a 32bit state, the rest staying in ROM.
The simplest explanation is that they don't give a flying flamingo about what you or I consider "improvements to products".
This report is an example.
There are several changes that modify CI behaviour, where the article gives no corresponding quality measurement.
They replaced type aware custom lint rules with AST-only static analysis. They don't say anything about what those new rules detect, didn't do old-vs-new rule comparison. They switched the TypeScript check from tsc to tsgo. Again, they are very proud of the performance improvement, but don't seem to care about diagnostic equivalence. The list goes on. They don't even report pass/fail agreement between the old and the new CI. They have 4x more tests, but no idea whether this big test suite works any better than the smaller old one, or even whether it works at all.
reply