I do pretty much that using GNU Stow. Of course I then build packages and libraries on my own, in my $HOME, because I don't have root access on that machine.

I install any PACKAGE of version VERSION in:

    ~/stow/pkgs/PACKAGE/VERSION/

So taking nim for example, I'd have:

    ~/stow/pkgs/nim/0.17.1/

Under that, I'd have the whole FHS system for just nim 0.17.1:

    ~/stow/pkgs/nim/0.17.1/bin/
    ~/stow/pkgs/nim/0.17.1/lib/
    ~/stow/pkgs/nim/0.17.1/share/

I set ~/stowed/ as my STOW_TARGET. So on running stow, symlinks to all:

    ~/stow/pkgs/PACKAGE/VERSION/{bin,lib,share,..}/..

get created in:

    ~/stowed/{bin,lib,share,..}/..

I then have aliases set up to do something like "unstow PKG/OLD_VER" and "restow PKG/NEW_VER" when I want to change the package version.

I skip the "restow" step if I just want to "soft uninstall" a package i.e. remove it from PATH, PKG_CONFIG_PATH, MAN_PATH, .. by removing it only from the STOW_TARGET ~/stowed.

"Hard uninstall" would be simply:

    \rm -rf ~/stow/pkgs/PACKAGE/VERSION_I_DONT_WANT/

Or even:

    \rm -rf ~/stow/pkgs/PACKAGE_I_DONT_WANT/