Had great experience using PXE to boot HPC farms, mounting the OS from a NAS and using only a local disk in the machine for tmp and other writable locations. I am not sure how 'diskless' linux works these days on rocky flavours but was solid in centos 5 through 7.
I've had issues using root-on-nfs now that dnf uses a sqlite database. However, root-on-iscsi has been working really well for me.
At first I was PXE booting a kernel and custom initramfs over TFTP, and having the initramfs initiate the iscsi session. More recently I have been PXE booting iPXE, which then has support iscsi. iPXE initiates the iscsi session, reads the kernel and initramfs off the "disk", then once the kernel and initramfs have loaded, the initramfs uses some magic called iBFT to "rediscover" the session that the firmware initiated.
The nice part about that second approach is I don't need to set up any hooks to transfer a new kernel and initramfs to my server whenever they're updated on the machine.
Ive always used the feature(?) allowing you to mount parts of the fs read-write persistent on the local disk which might help the sqlite issue, though not sure if thats a read/write issue or something else.
Have never used scsi for this. i suspect it's faster?
> Ive always used the feature(?) allowing you to mount parts of the fs read-write persistent on the local disk which might help the sqlite issue, though not sure if thats a read/write issue or something else.
The entire nfs based rootfs is mounted rw at this point, so it's not simply a matter of it being a read-only filesystem. NFS has some weird file locking behavior that makes it behave noticeably different from a local filesystem for certain functions. Some sources seem to claim NFSv4 has better support for file locking, but I was unable to get root-on-nfs working reliably under any permutation of NFS parameters I attempted.
Loading the NFS rootfs wasn't a problem though, so I figure a distro that was a bit more intentional about not making nfs-unfriendly software part of the core would be able to do it. Although I also tried setting up Arch in this way and had similar issues.
> Have never used scsi for this. i suspect it's faster?
I'm not really sure whether it's faster. A lot of online sources say "iscsi is faster because block device" and it's not immediately obvious to me how that would make it faster.
I would say that in my experience iSCSI has had fewer compatibility issues. Applications seem to be happier with a high latency block device and a traditional filesystem, than doing file operations over the network.
Had great experience using PXE to boot HPC farms, mounting the OS from a NAS and using only a local disk in the machine for tmp and other writable locations. I am not sure how 'diskless' linux works these days on rocky flavours but was solid in centos 5 through 7.