> Yup, that's exactly what I'm saying there is something wrong with data model.
We are an Open Source project and know we can do better on the indexing speed. We already did much work on that subject. We enabled the auto-batching feature, which significantly improved the time to index documents.
Are you sending all your documents in one go, or are you waiting for the tasks to be indexed to send the next batch?
> I also noticed it's only using a single thread.
Unfortunately, one pass of the indexing process is currently single-threaded. We can do better, and we are aware of that. Software development takes time.
> Out of curiosity what dataset size you guys are testing on?
We are testing on a broad list of datasets, going from 200k movie datasets to 142 million song datasets.
> Other learning is, not every software written in Rust is fast :-)
It depends on what you want to be fast. Software programming is a matter of trade-offs, and we have excellent search speed results.
Yup, that's exactly what I'm saying there is something wrong with data model. I also noticed it's only using a single thread.
Out of curiosity what dataset size you guys are testing on? Indexing 2M documents on a server in 2023 should be instant, or at least <60 seconds.
Other learning is, not every software written in Rust is fast :-)