A TPUv3 pod is ~107 petaflops (Googles number from your paper). 512 Volta GPUs is ~64 petaflops (Nvidias number from [1]).
v3 pods don't seem to be publicly available. A 256 chip 11.5 petaflop v2 pod is $384 per hour, $3.366 million per year. [2]
Meanwhile Google Cloud Volta GPU prices (which are probably inflated over building your own cluster, but are hopefully close enough to a reasonable ballpark) are $1.736 per hour, would be $7.791 million per year for 512.
Unless Google GPU prices are really inflated, clusters are legitimately substantially cheaper than cloud GPUs, or these researchers did a poor job, it seems like this is a good advertisement for TPUs.
v3 pods don't seem to be publicly available. A 256 chip 11.5 petaflop v2 pod is $384 per hour, $3.366 million per year. [2]
Meanwhile Google Cloud Volta GPU prices (which are probably inflated over building your own cluster, but are hopefully close enough to a reasonable ballpark) are $1.736 per hour, would be $7.791 million per year for 512.
Unless Google GPU prices are really inflated, clusters are legitimately substantially cheaper than cloud GPUs, or these researchers did a poor job, it seems like this is a good advertisement for TPUs.
[1] https://www.nvidia.com/en-us/data-center/volta-gpu-architect...
[2] Pod availability / performance / pricing information here: https://cloud.google.com/tpu/
[3] GPU pricing info: https://cloud.google.com/gpu/