The best C and [insert any language here] package manager is Nix. Or rather, it should be[1]. I'm fairly confident something like it will be. If you haven't, please do try it out (or maybe don't[1], but at least read the paper[2] and let its ideas sink in). I honestly just sigh and think "yet another package manager and they still haven't learned..." whenever I see a new one now.
[1]: Unfortunately documentation is famously bad, the repo that holds package declarations is full of undocumented conventions, there often isn't a "blessed" way of doing things, and there are some other warts, such as all the difficulty one may experience from having to learn a new functional, lazy, dynamically typed language. I'm not aware of active, coordinated efforts to improve on these fronts, except for new languages such as Nickel.
Nix is an academic's solution to a large problem space based on a single principle ignoring several real limitations in order to shoehorn itself into an existing box of a problem. Of the many problems it has are assumptions about development models, deployment models, and operational models. Moreover, it seeks to be a monolithic proprietary solution rather than a collection of loosely-coupled layers that can be interchanged.
But what irks me is that it doesn't fundamentally change anything. There are clearly major inherent limitations in how software is developed that led us here. Nix is intended to find complicated ways to work around the problems in those development patterns. But we can't call out the pink elephant. Rather, let's make pink wallpaper, pink sofas, pink chairs, pink rugs, and then say that we've solved the problem, because you can barely even see the elephant now.
The problem isn't packaging. The problem is software design itself. The dependency hell, the conflicting versions, the filesystem hash trees, the DSL. It all exists because the software itself has no way to express its own compatibility with other functions or dependencies. We just kick the can down the road to a random package manager and hope for the best. But clearly the solution needs to happen in the software, not in how we install it.
You're stating a lot of interesting opinions, but I do not see any arguments to help describe why you feel that way. How exactly is Nix monolithic or proprietary? And what is complicated about it? It seems pretty simple to me, if not at odds with how other package managers do things. It's one of the few package managers that can successfully work across multiple Linux distributions, some BSD systems, MacOS, and even Windows for some software. In my opinion it's the least proprietary or monolithic package managers around, the other being Guix which follows the same principles.
But I think the biggest issue is in your last paragraph where you don't describe why or how the solution needs to happen in the software itself rather than at a higher level.
If you need something that exists outside the repository it can get complicated. Even playing within the repository can have pitfalls at least on Guix which is what I've mainly used (like having a package appear in two profiles can cause issues that are non-obvious or like using "guix shell gcc binutils" instead of " guix shell gcc-toolchain"). Some of that has to do with the fact of what an expected *nix install is supposed to be and expectations of what should be installed on a users system and so are not marked as dependencies or requirements when they really are.
I am curious about how a system designed from the ground up with this in mind might be better.
The ability to start a shell with all the right dependencies configured, without requiring root access or permanently installing new dependencies, seems like something useful.
You can also do that with rootless containers via Podman. There’s even a convenient wrapper for that workflow made by Red Hat called Toolbox (https://github.com/containers/toolbox)
unless you need specific versions of dependencies? The last time I looked to try and port any of our own software the nix answer to needing specific versions dependencies seemed to be "you are holding it wrong".
With Nix flakes it's even easier and you can just mix and match inputs as you want. For example if you want to replace libfoo with your own fork you just do something like:
And the best part is that none of this requires any changes to your system. If you need multiple different specific versions at once, Nix can just do it, as every software is sitting around in /nix/store/{HASH} and there is no risk of anything colliding.
Glibc allows specifying the version of a function at call time, so that you can call a specific version of a function in a library. This means that regardless of when you released some software, you can pin how it works, so even if you upgrade the library 10 years later, it will work the same way (as long as the library keeps the old version of the function). This needs to be extended, made easier, and every programming language should do it by default.
The filesystem is also a limitation. We need to refer to versions in files and directories in the same way, due to application dependence on files. This can be accomplished probably just by adding versions in paths and then defining version compatibility in code. Would be nice if the filesystems all got versioning built in, though, so we could just call file functions with version arguments rather than hacking around path names.
Data formats and protocols also need explicit versions for all operations, so they can advertise their compatibility and thus be backwards and forwards compatible.
We must also be able to execute software with a specific combination of versioned functions at exec time. This requires kernel modifications / new syscalls.
Libraries and applications should keep old versions of functions indefinitely, and only load code when needed. This will cause some consequences which we need solutions for, such as how to handle the reduced disk space and shared memory. It will require rethinking how applications are built, stored, and executed.
All of this rethinking of software design requires a lot of effort. But it would provide a world of benefits that we currently need hacks for.
We could upgrade randomly to bleeding edge software without any change to existing applications; any software interacting would continue to function as before. But newer function versions would be available and could be enabled via feature flags. We would no longer need containers, maybe not even package managers. Software could be upgraded continuously and remain perfectly stable. Users could define what versions they will execute depending on their wishes. And legacy software could be supported for decades, maybe centuries, with no extra effort.
Have you seen Unison lang? It sounds like it overlaps (at least) with what you're talking about.
> Unison’s core idea is that code is immutable and identified by its content. This lets us reimagine many aspects of how a programming language works. We simplify codebase management — Unison has no builds, no dependency conflicts, and renaming things is trivial.
FWIW, I agree with you that the way we do things now is, uh, not the best of all possible worlds. I think Nix (and Guix) is a step in the right direction, though, but they don't go far enough.
> you can call a specific version of a function in a library
A library has some functions. Later, the library gets refactored, and those functions are thereby rewritten; but since no changes have actually been made, the version number stays the same. However, there exists some legacy code that horrendously breaks when given a non-ABI-compatible function from this library, _even though_ the functionality and version of those functions the same. Oh dear! What can we do? Well, clearly exact binary compatibility is important, so someone has an idea: we'll specify functions with exact hashes of their code instead of versioning, so this problem doesn't happen.
This is all well and good, but then the library decides to update some internal utility subroutine, changing its hash, but _not_ changing its call sites. Oh dear, now some other functions have hashes that are still the same, but they have different functionality due to that utility update, and legacy code breaks horribly again! In order to make sure this doesn't happen, someone comes up with the idea that a function's hash should be dependent on the hashes of all of the functions, variables, etc. it depends on _and_ its code, so all of the "inputs" that go towards defining the function end up updating its hash, thereby insuring legacy code stability.
Now we run into another problem: where do we put all of these functions? They need to be stored somewhere on the system so applications can have access to them, so we'll put them in a repository that we'll call a "store". Now we can do cool stuff with shared stores such as "only load code when needed"!
Oh dear, now the user sitting at the shell of the system wants to run programs too. Our function infrastructure works so well, why not extend it to programs too?
Some programs really like filesystems and environments and depend on specific structures. We can't really encode these structures as an "input" to our programs' hashes, because they could be highly dynamic (or even undescribable), so instead we can put software into a little box where it sees the filesystem structure it wants, so it is happy, and the box itself could be specified with input programs/functions in the same way that programs or functions can!
Hmm.. we seem to be recreating this "hash" structure that depends on inputs a lot. We need a common name for this... hmm, how about "derivations"?
Congratulations, you have just reinvented Nix! The solutions in your comment that you talk about are things that Nix already was designed to provide. The only discrepancy is that Nix doesn't actually provide hashes for individual functions, rather they work for libraries, but this is because of a limitation on how libraries work: that they can have shared, global, mutable state, making the idea of individual function versions useless.
The main problem why Nix doesn't obviously seem to solve these problems is that Nix's tools are not very well designed. For example, it's very non-obvious how to specify older versions of packages. However, flakes makes this easier, and although I don't believe _Nix_ specifically will be the solution, the holy grail at the end of the road will be very similar to Nix in core ideas.
1) foo.lang: class foo { func bar(a) { return b } }
2) baz.lang: import foo && print( foo.bar(1) )
Generally `foo` needs to have unit tests (self-tests), but `baz` needs to have "acceptance tests" for `foo` because `baz` imports `foo` (external validation / fit-for-purpose). Blah blah "law of demeter" aka: self.do_foo_bar() { foo.bar(1) }.
I've come to the mental model that code should test itself, and (some) code should test its dependencies. Code fundamentally can't test code that uses it, but can attempt to be "well-behaved" and share (via documentation or checked interface) how it can be / should be used.
eg: `foo_test.lang: assert( foo.bar(1) == 42 );`
But if `baz` calls `foo.bar(2)` or `foo.bar(0.3)` or `foo.bar("four")`, then `foo` cannot possibly be responsible for how `baz` interoperates with it in all cases.
"Software has no way to express its own compatibility with other functions or dependencies".
I've heard that at a certain point all internal behavior can turn out to "bake" and be depended on by an external user. An extreme example is `*.append(...)` having O(n), O(log n), O(n^2) performance characteristics in either memory or speed. Some (ab-)use means that you may get a bug report b/c your performance characteristics changed in a way that the _user_ of your code didn't expect, and I struggle to see how anything other than snapshotting the world [never mind network-call dependencies] would ever "fix" things?
I think it's an impossible problem:
1) code can test itself
2) by definition, code can't [exhaustively] test how it is used by "others"
3) therefore: any changes to dependent code are potentially "breaking" changes from the perspective of the user of the dependent code (even bugfixes!)
4) a generalized "really good" outcome is possible (eg: imagine something like TypeScript but for dependency management), but once you loop around to point #1 again... you either exactly match like a puzzle piece (eg: frozen dependency), or there's some sort of wiggle-room where any sort of change to the dependency has the _potential_ to break the user, or there is guaranteed slack/wastage in between the two dependencies (ie: size, performance, memory, operating range, etc) where changes can be relatively safely made.
5) therefore: code that uses a dependency should have acceptance tests (fit-for-purpose) of how the dependency behaves.
We're relying on version numbers and transitive dependencies at the moment, but I can't imagine anything other than "freeze everything" being a viable, generalized solution?
It feels like the solution to most of this is static typing, not writing a ton of tests by hand for every single dependency. Observable changes in a dependency's logic should be caught by your own unit tests. Changes in dependency function names or signatures should be caught by the compiler.
But static typing implies shared typing, and then you run into issues like: <type:Username> and how do you marshal that across system or library boundaries?
Nix is nice as a catch-all, and especially good when languages like C and C++ don't have popular package managers. It's definitely not better than cargo or pip when dealing with just Rust or Python, but it's definitely relevant when you're building multiple languages.
I would love to see an evolution of nixpkgs with more of a starlark-style for package declarations.
This likely wouldn't work on practice. Nix is aggressively lazy, so if you just want a single package, you don't have to evaluate the builds of all 60k+ packages, just the package you wanted and its dependencies.
This just might be me as a functional-programming-maximalist, but I do not like imperative languages for configuration management. You implicitly have to hold the execution state of all actions to know what is going on.
BUILD file evaluation is very similar to nix expression evaluation. I think the bazel 'analysis' phase is equivalent to the nix derivation expansion step
Maybe nix is more lazy, but I would be surprised if nix ends up doing substantially less work than bazel when a single target in a large graph is built.
For managing python dependencies, nix avoids all the headaches of pip, poetry, etc. Builds always work, and one has confidence that they have exactly the same bits as everyone else. nix-shell gives you a virtualenv that also has all your non-python dependencies as well. Instead of dealing with 10 different package managers across a codebase, there's just one.
The trade-off is that it's much harder to select a set of versions with nix, and requires some knowledge of nix lang.
If you can just take all of your packages from a tag of nixpkgs, then that experience is very nice. On the other hand, if you end up wanting a package that is but up to date or missing entirely, that can be a bit more daunting.
It's heading in exactly that direction. In fact we specifically want to start using Starlark for module declaration (mind -- optionally. It's still all declarative JSON API at the bottom! APIs FTW!).
We're also going full hermetic and aiming for reproducible-by-default. Those should be "duh" things in modern world. The comparisons with the good parts of Nix should be obvious.
It's such a deep issue with the C and C++ ecosystem that it's probably what will finally lead systems programming to other languages. It wasn't an issue until the Internet, and still isn't an issue if you are linking against binaries, but definitely is a problem now.
Windows shouldn't be treated like a freak, something too outside the norm to support compiling on.
I have a colleague who insists that that docker is the C++ package manager. I dunno, I've tried to see his point of view but as a usability issue it seems like far too much complexity to just use Docker as a package manager. Compared to something like Cargo for Rust, Docker is incredibly complicated and nags you all the time about installing upgrades. And my understanding is that Docker has non-zero performance impact, right?
Docker performance impact might as well be zero. You're actually using the system kernel, with the application's filesystem (thus dependencies) and other resources (CPU, memory, network access) namespaced away from the rest of the system. So there is no e.g. mapping on top of the extant kernel mapping. I've not noticed and performance impact, though obviously my workflow isn't your workflow.
Seriously, try it. I personally use docker-compose, even for single containers.
Docker is reasonably performant only on Linux. On macOS it's a fat slow VM that bogs down the whole machine (unless you make the VM even slower by giving it fewer CPU cores), has a horrendously slow network file system, and it's managed by a needy Electron app.
And don't forget that all new Macs are now ARM-based, so x86-based Docker images don't work.
My gripe with C is that everyone pretends it's portable, but only care about porting to one (their) platform. It's easy: just use pkg-config/docker/nix. Windows? macOS? Who cares!? Just run a Linux VM!
I always wonder how do people develop in Docker entirely deal with their own shell configuration? Actually install all the tools needed by your .zshrc (or .bashrc) in the container or there is a better solution?
I personally mount a volume and run my normal shell on it. Some operations I do on the container, especially those which relate to deploy (e.g. PHP Composer), but for development the application directory is a normal filesystem directory and I'm using my own VIM and Bash.
Docker is definitely useful, and I have worked with people who have very nice Docker workflows.
Personally, I prefer VMs in GCP, Azure, or AWS. VMs can be stopped when not in use so their use can be inexpensive enough for both hobby projects and work. Instead of writing a Dockerfile, etc. for a new project, I create a VM. This has the advantage of using VMs much more powerful that any laptop I own. This does require being OK with a mosh/ssh, Emacs/Vim, development style.
How do you develop on a VM running on a remote machine? I could make some guesses but I'd love to hear the specifics, especially things that I could adapt into my own workflows. Are you compiling remotely but coding locally?
How is docker a package manager? It may well give you a nice even ground layer to execute your package manager on, but it doesn't do any of the traditional tasks that a language package manager does like e.g. dependency resolution.
You are correct, I skirted a bit around the idea of what a package manager is in response to Nix. Docker does not resolve dependencies, but it does provide a platform in which each application can have its own version of dependencies that won't clobber other things that I'm working on or my system as a whole. Furthermore, the Dockerfile serves as a record (even versioned in git) of the dependencies added for future deploys, like a Composer or NPM lock file would. And that Dockerfile serves additionally as a reference for future projects.
The VM performance hit is far too hard to recommend it for actually running applications. I'll use it to dev for another system (e.g. a Python backend for a web app) but I would not spin up a VM to e.g. compile LibreOffice or -shudder- actually run LibreOffice.
Would it really be an issue to run LibreOffice in a VM on a modern machine? Back in the bad old days of ten years ago, I used to run a Windows 7 VM on my Ubuntu laptop of RAM to watch Netflix. It was pretty tight with 2GB of RAM, but it worked fine.
> The wiki listing of packages acts as the "registry" and populates the clib-search(1) results.
This seems to be an extremely bad idea. Since everyone can edit the GitHub wiki pages, it means its registry is vulnerable to all kinds of malicious attack.
vcpkg [1] is also a workable C package manager. It mainly has C++ packages but does include a few C packages e.g. glib [2]. In addition, you can use it on a separate list of package recipes (portfiles as vcpkg calls them), in case you want to use packages not in the central repo. It's based on CMake and runs on Windows and Linux (and other platforms I think but I haven't tried).
At least the five items below should probably have an answer upfront before embarking on the creation of package manager. Particularly one intended for general widespread use:
- How do plan to address the issues of trust, security, auditing, copyright
and licensing, trough the life-cycle of the artifacts in your registry.
- How do you plan to enforce the artifacts integrity both
registry side as well as client side while using unreliable hardware.
- How do you plan to manage your dependencies and limit the blast radius
of unforeseen events.
- Do you have a community model for namespace management.
- Do you have a model to implement Two or Three Person/Group
rule for critical changes?
Edit: this latest is usually called the Two-Man Rule...
> ...all of them under different licenses, carrying different code standards...This issue needed to be fixed. [1]
Ehrm I don't see how that issue should be solved. I gave it a first look and there's clearly no way of filtering by license. Most of the sources in deps are MIT license but there are also BSD. So after you copy all the libs together you have to go through the package.json to find all the different licenses and if obliged copy them into your archive. Same can hold for the code from various sources. I don't think they all follow the same standard (It's an assumption. I didn't bother checking it.).
The best C and C++ package manager I have used is Xmake/Xrepo. They support all the major package repositories out of the box including Vcpkg and Conan.
This looks nice, something I would have really appreciated way back when I used to use C and C++. In the 1980s, my “package management” was a huge collection of file lists with descriptions for public FTP sites, grab relevant source code as needed…
I like the side conversation here about Nix. I have not tried Nix in its own, but when I used to use non-M1 Macs, I had NixOS running in VirtualBox.
I may be wrong about this, but I love the huge variety in programming languages, operating systems, package managers, development tools, etc. A world with a tech monoculture would be boring.
Great, it's on version 0.0.1. It's hard to get stable API right on the first try for projects of this magnitude, so I think it's great they keep the room for adjustments before going to 1.0. /s
[1]: Unfortunately documentation is famously bad, the repo that holds package declarations is full of undocumented conventions, there often isn't a "blessed" way of doing things, and there are some other warts, such as all the difficulty one may experience from having to learn a new functional, lazy, dynamically typed language. I'm not aware of active, coordinated efforts to improve on these fronts, except for new languages such as Nickel.
[2]: https://edolstra.github.io/pubs/phd-thesis.pdf (don't be discouraged by the number of pages, it's fine if you read just the first part)