- superior performance
- superior operational overhead
- no second copy of data in tsvector form
- BM25 scoring support with optimized top-k output
- runtime configurable scoring knobs
- expression-attached score boosting
- sophisticated span query support -- this is proximity search on steroids (https://github.com/planetscale/lead/tree/main/tinql/docs)
- lossless term positions
- index-answerable negative expressions (find all docs that don't contain a word)
- full document hit highlighting
- optimized exact `count(\*)`
- term expansion via any of fuzzy matching, wildcards, regular expressions, and dictionary ranges
- intentionally smaller user-facing SQL API surface
There's a lot we didn't cover in the announcement blog. I'm sure we'll do more as time goes on.
As an aside, something I personally think is cool, and I suppose you can do this with Postgres' built-in `@@` too, is that you can use TIN's full query language (linked above) against any text datum. This is a valid query:
SELECT pid, query
FROM pg_stat_activity
WHERE query ==> 'select OR copy'
in other words, you don't need an index at all to use TIN's full query language against any text field in any query.
Building non-open extensions on top of it, it's not the license I don't like, the bad taste is that they use something open extend it and keep part of it closed.
The license allows it but on the flip side it's vendor lock-in predicated on using something open as the base.
Fully proprietary no issue with that, full open, no issue with that, building proprietary on top of open is where the bad taste comes in.
For completeness, it's not them specifically either, the other cloud companies do similar things and I suspect in part the reason they don't open these extensions up is because the others will but then they are doing the same thing themselves.
I find this attitude self-defeating because the lure of being able to provide some amount of proprietary software on top of open is what draws in the corporate investment in open software. And in the system we live in, it’s hard to imagine there’d be nearly as much open software as there is without that corporate investment. As a big fan of open software, this seems like a great trade to me.
(Disclosure: I work for a company with this business model, in part because I like working on open software)
So you are saying "don't hate the player", but hating the game is a big nono. That thing that isn't capitalism is inherently evil and should be avoided like a plague
I don’t understand your comment, can you elaborate? I’m saying that drawing corporate investment into open software is preferable to all of that investment going into proprietary software. I want as much open software as possible and I want us to be practical at all levels. Big swings like public grants and a better safety net (even UBI) are things I’m supportive of too, and can coexist with incentives for drawing in corporates. There’s a certain purism argument that a company that is involved in open software is constantly criticized for being insufficiently open but a proprietary software company receives no such similar criticism, which I think is irrational.
I don't know which exact company you are talking about. I see most proprietary companies heavily criticized and do so myself and used to heavily advocate for OSS at $work. My current work is 99.9% opensource.
As for my point, the main issue you are pointing out is the fundamental flaw in capitalism: companies focus on profit instead of technological progress. The two can be correlated but that isn't granted. See enshittification.
There are other economical models that could potentially address the shortcomings of capitalism but a lot of effort is put into discrediting socialist/communist ideas as they threaten the wealth of the wealthiest.
When one develops using open-source software they have an obligation to follow the licenses. They also have a moral obligation to be respectful of the work upon which they’re building. And they have a social obligation to help improve that software where they can.
Those that develop on top of open-source have no obligation to give you their work for free.
You’d be surprised as to the amount of open-source contributions TIN drove towards Postgres, LLVM, and pgrx. And you’d be speechless at the amount of upstream work across all sorts of open-source PlanetScale does. Postgres 18.6, for example, is better for you today, in part, because of TIN. You’re welcome.
I didn't say thank you and don't presume I would, Planetscale acting in their own self interest by improving postgres upstream isn't deserving of thanks, any more than Intel upstreaming a bunch of Linux kernel work is or myriad other examples.
Corporations acting in their self interest isn't worth giving thanks for, neither is the work of the people paid to do work on their behalf.
I don't expect my employer to thank me, I expect them to pay me, I don't expect users of software I was paid to write to thank me because I did it for the money not out of altruism towards those users.
> Those that develop on top of open-source have no obligation to give you their work for free.
This is a pretty narrow view of open source.
Just as a simple example, GPLv3 and AGPLv3 are both considered Open Source and, depending on how you hold them, may obligate releasing work to customers essentially for free.
"...are both considered Open Source" << Free Software (GPL) is the O.G. - it's like saying "George Washington is considered to be an American President" or "The Beatles are considered to be a pop/rock band" or "water is considered to be..."
I remember so well when "Open Source" branding started with Bruce Perens - all the business arguments. When you need a database, someone ELSE'S business decisions (how THEY are going to make money) never end up helping YOU. If you use proprietary extensions and become dependent on them, you inevitably will get BURNED when their business needs diverge from your needs.
- They close shop
- They refuse to interop with something you need
- They demand that you obey their arcane rules
- They rug-pull
- They get hacked as only they can
- They lie to you
- They stab you in the back
> When one develops using open-source software they have an obligation to follow the licenses.
Agreed!
> They also have a moral obligation to be respectful of the work upon which they’re building.
Respectfully, completely disagree!
Open source is a license, not a moral framework. Your only duty is to abide by the license. If you want to enforce that everything built on top of a particular open source package also be open source, that belongs in the license.
We live in a society. I don’t see any harm in believing we ought to be respectful of the work. That’s all I’m saying. You don’t have to be, but it’s a better world if you are.
> If you want to enforce that everything built on top of a particular open source package also be open source, that belongs in the license.
Yes! I fully agree. The one that creates the thing is the one that gets to choose its license.
Having opened-sourced some bit of work myself, that’s a very difficult decision.
They also end up with all the infrastructure and processes necessary to keep the external search system in sync, resync/reindex, pkey shipping back to their source of truth in queries, application-side joins and enrichment between both sources. It’s brutal.
Having everything in one place eliminates entire classes of development and especially operational problems.
I’m one of TIN’s developers and if you google my username you’ll see I’ve been in this space for a long time.
The answer to your first question is simply: yes
As far as your second question, what customization do you need that you believe TIN or PlanetScale doesn’t provide? These are things we can do, with alacrity.
I don’t think we even expose the TableAM APIs? They are incredibly hard to generate bindings for from the C headers — lots of inline functions and complex #define macros.
We have an ambitious goal with pgrx and it’s going to take many years and countless hours of developer effort to get there.
It can, however, serve as a way for newcomers to gain experience with Postgres internals.
Regarding the TableAM specifically, when we are able to create a safe Rust wrapper around it, that wrapper will be documented.
Those are the internals we currently expose as unsafe “sys” bindings.
As we/contributors identify more that are desired we add them.
pgrx’ focus is on providing safe wrappers and general interfaces to the Postgres internals, which is the bulk of our work and is what will take many years.
As unsafe bindings go, we could just expose everything, and likely eventually will. There’s just some practical management concerns around doing that without a better namespace organization —- something we’ve been working.
The Postgres sources are not small. They are very complex, inconsistent in places, and often follow patterns that are specific to Postgres and not easy to generalize.
If you’ve never built an extension with pgrx, give it a shot one afternoon. It’s very exciting to see your own code running in your database.
As an aside, something I personally think is cool, and I suppose you can do this with Postgres' built-in `@@` too, is that you can use TIN's full query language (linked above) against any text datum. This is a valid query:
in other words, you don't need an index at all to use TIN's full query language against any text field in any query.reply