If Pandas is slow, than you can use Spark. For such big files laptop is not an o... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		nooorofe on Dec 31, 2019 \| parent \| context \| favorite \| on: Doing a database join with CSV files If Pandas is slow, than you can use Spark. For such big files laptop is not an option anyway. SQLite can be fast if you index your data (but I've worked with files < 10G). Nowadays I am just uploading CSV to some cloud database and work with data there.

makapuf on Jan 1, 2020 [–]

> For such big files laptop is not an option anyway

Too big for excel is not big data, and my laptop can load this 10G in RAM (not that it necessarily need all of it) so why not if the data is here and the laptop on your lap ?

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact