Distributed filesystems are designed with commodity hardware and traditional data-centers in mind. Why SHOULD it be distributed across local drives on all nodes? These are computationally intensive workloads (like supercomputers), not storage I/O intensive workloads (like mainframes or Hadoop clusters). If you're managing hardware on this scale and you don't need local drives, why on earth wouldn't you put your storage in a separate, central location?