LuaTeX can be used as a PDF writing library that has a "perfect" line breaking algorithm inside along with many other nice features. We use it as a base for our database publishing tool. Its absolutely reliable, has modern features (unicode, OTF) and can be programmed in a very decent language (Lua). You need to know TeX if you want to unleash all the power, but you don't need to program in TeX's macro language (which is fun but also a PITA).
I think that a new generation of TeX based applications will show up in the next 10 years.
Agreed. Luatex makes the following division of labour practical: work with data structures and manipulate fonts and the backend in Lua, handle moving about text and setting parameters in the macro language.
One thing I would like to see is proper static analysis tools for Luatex. Lua has tools like https://github.com/fab13n/metalua/tree/master/src/samples/me... but there's nothing of comparable sophistication for Tex, let alone code combining the two, which makes debugging needlessly hard.
The problem is that you can change TeX's syntax during the run time. I am not sure if it is theoretically possible to create such a tool for TeX input. That is one of the reason why I use (except for approx 10 lines of TeX code) only Lua code in my software. These 10 lines will disappear once there is a tex.initialize() in LuaTeX.
I'm quite sure flow-based static analysis can be done for Tex, the question is how expensive it is to perform and how useful the results are.
With respect to catcode-changing code, you can model its possible execution paths: while you can construct example Tex code where you can't infer anything past this point without having complete information from before, in practice with, say a package, you do have example documents which the package is meant to work with, and you can use that in the static analysis.
I've thought about this a fair bit, at least in the context of typical Plain Tex, but not written much code. I should try and enough time for a proof of concept: there's a lot that could be built on it.
I think that a new generation of TeX based applications will show up in the next 10 years.
Edit: some links:
http://luatex.org/ - official site
http://en.wikipedia.org/wiki/LuaTeX - wikipedia
http://wiki.contextgarden.net/LuaTeX - LuaTeX is now also the base of ConTeXt, a LaTeX alternative