For dumb simple select group by OLAP queries on medium data ? Sure clickhouse local or duckdb works perfectly well.
But if you need to construct dataset ? Or process existing dataset, do heavy filtering, transformation, reshaping, splitting? The proper ETL work, then pandas is really the perfect use case.
And pandas can work with small memory footprint as well, its actually trivial to do that, plus there are libraries like Modin that are upgrades over pandas with pandas api
Pandas is the swiss knife tool of data science that lets you do anything with the data and it integrates well with ML libraries
How would you use sql to move data from three excel files (some of them may have arbitrary number of worksheets), pack of json files, two csv files, one mysql, one postgres db, some parquets, some of them in S3) ???
And in the end save processed data in aws s3 in another format like iceberg or whatever
In pandas a lot of these are one-liners that are impossible in sql (depends on what sql engine you have?)
For dumb simple select group by OLAP queries on medium data ? Sure clickhouse local or duckdb works perfectly well.
But if you need to construct dataset ? Or process existing dataset, do heavy filtering, transformation, reshaping, splitting? The proper ETL work, then pandas is really the perfect use case.
And pandas can work with small memory footprint as well, its actually trivial to do that, plus there are libraries like Modin that are upgrades over pandas with pandas api
Pandas is the swiss knife tool of data science that lets you do anything with the data and it integrates well with ML libraries