It is a cool project. S3 can be cost efficient, but only if you don't touch data :)
Their price calculation doesn't mention cost of S3 requests, which very quickly adds up and is often neglected.
It costs $1 for 2.5M GET requests to S3. They have 180 shards, in a general case query seems to fetch all of them. Presumably they don't download full shard per request, but download an index + some relevant ranges. Lets say that is 10 requests per shard. So that would be 1800 S3 GET requests = ~1400 search queries cost them $1.
Assuming their service is reasonably popular and serve 1 req/second on average, that would be $1,440 per 30 days in addition to advertised $1,000 spent on EC2 and S3 storage.
Seems comparable to AWS ElasticSearch service costs:
Don't forget S3 includes replication.
Also EBS throughput (even with SSD) is not good at all.
Also our memory footprint is tiny. This is necessary to make it run on two servers.
Finally, cpu-wise, our search engine is almost 2x faster than
lucene.
If you don't believe us, try to replicate our demo on an elastic search :D.
Chatnoir.eu is the only other common crawl cluster we know of. It consists of 120 nodes.
> If we get 1 req/s, even for a dataset of that size, this is not as cost efficient.
How many req/s do you have in mind for your system to be a viable option?
> Also EBS throughput (even with SSD) is not good at all.
It is not worse than S3 still, right?
> Chatnoir.eu is the only other common crawl cluster we know of. It consists of 120 nodes.
I have no deep ES experience. Are you saying, that to host 6TB of indexed data (before replication) you'd need 120 nodes ES cluster? If so, then reducing it to just 2 nodes is the real sales pitch, not S3 usage :)
What about d3en instances? Clustered, and together with minio you might reach similar performance. Only issue is the inter-region traffic, it would need to be inside the same AZ
I had the same feeling when reading the post. Their remark that they "estimated the cost" to be that low is in my experience a bad signal. Estimating costs on the cloud is really hard, there are so many (hidden) costs you may miss making it a lot more expensive.
For what it's worth, if you want to run ElasticSearch on AWS I would always go with local-NVMe instances from the i3 family, this is also what AWS and Elasticsearch themselves recommend.
4x i3en.2xlarge (64GB / 5TB NVMe) at $449 / month (1yr reserved) is $1796, or $2636 without reservation, but much better performance due to the NVMe drives.
It's easy to put a block cache in front of the index, and I'm sure they'll get to it sooner or later.
The benefit of using S3 in that case is that unlike e.g. Elastic, your block cache servers don't need replication, and you can tear them down when you're done. You can put them in a true autoscaling group as well.
A ridiculous blanket statement, despite the "almost never" cop-out...
It is cost-efficient in a wide array of scenarios. Many companies pay for it because they have calculated the different investment scenarios and AWS comes on top of alternatives such as owning the hardware or using competing cloud vendors.
I own a consultancy that builds complex web apps and while I appreciate how occasionally a dev has tried to save costs for me by cramming every piece of the stack (web server, cache, db, queue, etc.), in a single Docker image to host in a droplet, I'd much rather pay for separate services, as I consider it cheaper in the long run.
Yes, I charge $60-$90 per hour of dev time to my customers and the time saved from using simple Elastic Beanstalk deployments pays for itself in saved dev time. The architecture is also infinitely easier to reason about and scale than cramming parts in a single image.
Their price calculation doesn't mention cost of S3 requests, which very quickly adds up and is often neglected.
It costs $1 for 2.5M GET requests to S3. They have 180 shards, in a general case query seems to fetch all of them. Presumably they don't download full shard per request, but download an index + some relevant ranges. Lets say that is 10 requests per shard. So that would be 1800 S3 GET requests = ~1400 search queries cost them $1.
Assuming their service is reasonably popular and serve 1 req/second on average, that would be $1,440 per 30 days in addition to advertised $1,000 spent on EC2 and S3 storage.
Seems comparable to AWS ElasticSearch service costs:
- 3 nodes m5.2xlarge.elasticsearch = $1,200
- 20TB EBS storage = $1,638