Are filesystems the correct abstraction to build databases on? Isn’t a filesystem a database in a way? Is there a reason to build a database on top of a filesystem abstraction rather than a block abstraction?
To say you can’t build an efficient database on top of S3 makes sense to me. S3 is already a certain kind of data-storing abstraction optimized for certain usages. If you try and build another data-storing abstraction optimized for incompatible usages on top of that, you are going to have a difficult time.
The traditional POSIX filesystem is the wrong abstraction for a database, but not filesystems per se. All databases that care about performance and scalability implement their own filesystems, either directly against raw block devices or as an overlay on top of a POSIX filesystem that bypasses some of its limitations. The performance and scalability gains by doing so are not small.
The issue with POSIX filesystems is that they are required to make a set of tradeoffs to support features a database engine doesn't need, to the significant detriment of scalability and performance in areas that databases care about a lot. For example, one such database filesystem I've used occasionally over the years, while a bit dated at this point, is designed such that you can have tens of millions of files in a single directory where you are creating and destroying tens of thousands of files every second, on upwards of a petabyte of storage. Very far from being POSIX compatible but you don't get anything like that type of scalability on POSIX.
Object storage is far from ideal as database storage. The biggest issue, though, is the terrible storage bandwidth available in the cloud. It is a small fraction of what is available in a normal server and modern database engines are capable of fully exploiting a large JBOD of NVMe.
> Is there a reason to build a database on top of a filesystem abstraction rather than a block abstraction?
Oracle DB for a long time supported running on raw partitions which I think suggests that the answer is "not really". Snowflake (and I hear Clickhouse) can run on S3 which I think is more evidence against running on a filesystem. Not to mention the torrid time Postgres has had with fsync on linux.
In my $dayjob as cloud architect I sometimes suggest S3 as an alternative to pulling massive JSON blobs from RDS Postgres/Redis etc. As long as their latency minimums are high enough there's no reason you can't.
To say you can’t build an efficient database on top of S3 makes sense to me. S3 is already a certain kind of data-storing abstraction optimized for certain usages. If you try and build another data-storing abstraction optimized for incompatible usages on top of that, you are going to have a difficult time.