Well, I was able quickly remember and find just this: "The Box: A Replacement for Files", Francisco J. Ballesteros et al. (http://lsub.org/ls/export/2kblocks/index.html)

While I was writting the comment above I was mostly thinking of Plan 9 and GNU Hurd. Couldn't quickly remember exact names, but you should look in direction of those two OSes. They were made to address the shortcomings in UNIX design, and had some fancy concepts.