Maybe I'm just spoiled because I can put crawl/parse tasks in a work queue for each url without much concern for latency but I've gotten 1000 r/s (on async calls) out of $100/€100 dedicated servers.
Their setup really only makes sense for WAN facing public architecture that can spike because you have no control over the load. Otherwise, I'd suggest you just get some leased dedicated server(s) to meet your workload.
Maybe I'm just spoiled because I can put crawl/parse tasks in a work queue for each url without much concern for latency but I've gotten 1000 r/s (on async calls) out of $100/€100 dedicated servers.
Their setup really only makes sense for WAN facing public architecture that can spike because you have no control over the load. Otherwise, I'd suggest you just get some leased dedicated server(s) to meet your workload.