The "why" behind the trie is explained in the comments in the code. :-) The core of my hypothesis was to avoid hashing and hash table lookups at all.
The problem with the trie, though, is that it does a memory access per byte.
Whether and when this trade is beneficial is not totally clear to me, but clearly, it can vary.
Whether my hypothesis is actually correct is also not something I'm certain of. Verifying this would take some time with: 1) looking at the codegen and 2) some use of `perf` to extract CPU counters for things like branch and cache misses, and see if a correlation can be established.
The problem with the trie, though, is that it does a memory access per byte.
Whether and when this trade is beneficial is not totally clear to me, but clearly, it can vary.
Whether my hypothesis is actually correct is also not something I'm certain of. Verifying this would take some time with: 1) looking at the codegen and 2) some use of `perf` to extract CPU counters for things like branch and cache misses, and see if a correlation can be established.