We use lua 5.3 on an embedded Platform for scripting and it has been a roller coaster ride. Not a fun one, unfortunately.
You want luasocket? The stable one is not compatible with 5.3.
Packages are sometimes outdated (for years no updates) and there is no replacement.
The lua point releases have breaking changes.
The source code itself is a macro hell which is hard to debug. And the code is not very readable. The documentation lacks for some topics of you use the c api. I tried to implement some scheduling for c and lua threads.
Oh and one thing if you must use Windows... you better quit right away. I had to help my co worker Installing it with luarocks and it is a mess. to be fair it was easy on my Ubuntu machine.
The thing is lua on embedded has no rival. but god did it cost me some nerves.
> The documentation lacks for some topics of you use the c api.
I'm surprised to hear this. I'm doing a lot of Lua and I always thought the documentation is very succinct and complete. I really curious when issues you ran into.
> The thing is lua on embedded has no rival. but god did it cost me some nerves.
My info might be a bit out of date, but something that surprised me when I looked into it was how little control I had over the embedded lua interpreter from my outer app. I basically had to hand it control by calling lua_pcall, and either wait for a result or manage a timeout from another thread.
I expected to be able to do things like tell Lua to run for X milliseconds or Y opcodes or whatever, but it didn't seem that anything like this existed.
So perhaps my disappointment with the documentation was really just disappointment that the implementation wasn't set up to do what I wanted in the way I wanted to do it. But in the course of coming to this realization, I also found the documentation frustrating and opaque.
Running for X opcodes isn't too difficult, but isn't immediately straightforward either. Instead of calling a function with lua_pcall, start a coroutine using lua_resume. Then, you can have a callback using lua_sethook and the LUA_MASKCOUNT option, which triggers the callback after a fixed number of opcodes. Inside the hook, you make a call to lua_yield. That forces the coroutine that was running to yield control to the outer app. The outer app can then resume the coroutine if desired.
There was a while that I wanted use this to design a programming-based video game, where simulated agents worked in real time. Most of the programming games I have seen use a fixed timeout, and kill any scripts still running after that time. What I was thinking was to instead have the script constantly running, and any slow scripts continue running on the next frame (e.g. all players get 100 opcodes per frame).
Thats clever. I used the same approach with lua_sethook to run a hook every X opcodes and check if time or memory constraints have been exceeded. But instead of yielding I used longjmp to terminate execution (basically like raising an exception and catching it a few levels up in the stack).
It's not bulletproof though and the script can still block e.g. when calling into native code.
Yeah, native code can block, and in those cases requires additional checks in that exposed native code. That runs into all the usual multithreading headaches, and depends on exactly which functions have been exposed to the lua interpreter.
The advantage to this method over longjmp is that the function call is still valid and can be resumed at any point.
Ah. Yeah applying resource limitations outside of memory usage is always a bit awkward. The only really supported way to my knowledge is to externally set a hook and trigger an error when called: https://github.com/lua/lua/blob/master/lua.c#L40-L59. I think this doesn't always work in tight loops though.
I remember patching Lua over 10 years ago for a programming game. I essentially counted VM instructions and stopped executing when reaching the limit. It survived malicious players on a hacker conference back then ;-)
I'm not sure which embedded platform you're using but give AngelScript [0] a look.
I've been using it to extend some C++ code (bsnes emulator) and it's been fantastic. It's so nice to have a C++-like scripting language with natural script-host bindings that I don't have to think hard about creating or spend a lot of time writing marshalling code or worrying about memory layouts. On top of that nice script-host interface you get an actual usable scripting language without silly things like 1-based array indices.
That said, it does not come with any sort of standard packages like luasocket but there is support for defining your own modules.
Depending on the embedded platform you're using, there may not be support for the native calling convention and you may have to fall back to the generic calling convention, or you could roll your own code to accommodate your platform's calling convention.
AngelScript is not a transpiler. It's a runtime scripting language with an interpreter designed for easy script reloading at runtime, and I think there's even a JIT out there for it too. It's a valid competitor to Lua for the same use cases Lua serves. The only difference for integration may be that AngelScript is implemented in C++ and not C. Probably a C wrapper API could be (or has been) developed but that might not solve the problem of lacking a C++ compiler for your embedded platform.
I share some of your pain. We use Lua 5.1 on our embedded platform and at one point did development on 5.2.
There's a lot of (seemingly) undocumented changes, even between two minor versions that make you question the semver (and your sanity) at times.
For example, the patterns operate just different enough between the two minors, that I had to resort to dumping everything into JSON and back since that was the only way that one odd case worked similarly between the two versions (albeit sharing the same JSON library).
In regards to the new version, coworker of mine glanced at the 5.4 changelog and could not make out how things had changed, only the fact that they had been changed.
Lua point releases, 5.1, 5.2, etc. are major releases. The numbering scheme isn't semver. It is only accidentally that Lua even has an ecosystem. It is meant entirely for embedding within a larger application and not used as a stand alone application server.
I just had a short glimpse, but it seems that this is rather "transpiling" to C, right? We needed something that is being interpreted to allow easy updates of scripts.
Comment on all subjects (I will just link it to other commenters):
A lot of commenters posted some projects with JS interpreter. But we had to chose something that is stable and is being maintained. The last thing is always hard to argue about because you never know how long maintainers stay commited to the project. But the LUA interpreter is widely used for a lot of (commercial) projects. So there is high chance that it stays for some time. We tried out some other projects, sometimes looking into mailing lists and github issues where we would find open bugs where the interpreter leaks under some circumstances or other problems that you just want to avoid.
LUA seemed a perfect fit and i think still is, but there are still things that didn't work as planned. I really love the concept of the interpreter with the stack. But, again, scheduling is a very complex topic and that is where i found the lua documentation lacking. Especially scheduling between LUA <-> C. I found some projects that solve this issue but then we stumbled upon other issues:
Some projects were for 5.1 and didn't work for us. Others used features we didn't have on embedded (pthreads).
So we decided to do a very basic scheduling but had to understand more what happens under the hood. And that is where the source code is not easily readable. Don't get me wrong i am convinced that the devs are great at their work, but holy moly you can't use more readable name for you variables? Or the use of macros, is it really neccessary to use it for every conversion?
Interesting. Didn't Unicode restrict UTF-8 to allow encoding only 21 bits? Does it mean that it can now do 6 byte UTF-8 encodings? What kind of restrictions did it have before?
Lua 5.4 has a new feature called to-be-closed variables. It is intended to allow for deterministic freeing of resources, even in the face of possible runtime exceptions. (Sort of like RAII in C++).
The canonical example is that you can mark a variable holding a file handle as to-be-closed and then as soon as you exit the variable's scope the file gets automatically closed (including if exit early due to break, return, or error).
But what is supposed to happen if you are inside a coroutine and pause the execution before reaching the end of the scope, and never resume again? If there are any to-be-closed variables their destructors will never run! Or they might only run after the containing coroutine gets garbage collected, which is not a timely solution. To cover this situation, Lua 5.4 introduced a new coroutine.close function that kills a paused coroutine and runs the destructors of any to-be-closed variables inside it, if there were any.
Just recall that Lua versioning schema allows for breaking changes to point releases.
This has been both great for Lua as a Lua and one of its biggest challenges. It's allowed the language be constantly refined and tweaked; however, at the expense that it's super common for applications to complete break when upgraded to a point release.
LuaJIT is an entirely different VM. It is binary compatible with Lua 5.1.x modules (shared libraries) and can run most Lua 5.1 and 5.2 code. A good amount of the 5.2 changes, specifically those which do not break 5.1 code, have been added to the VM.
I use FFI and LuaJIT is significantly faster on my embedded platform. Also, you can yield in coroutines in places where you can not with the standard Lua VM, such as across pcall, xpcall, iterators, and in metamethods.
They have function variants which can do this, but you need to call lua_pcallk instead of lua_pcall, which doesn't help me much with third party libraries.
Indeed. Mike Pall blows them out of the water single-handedly. I wonder if Lua developers have learnt anything from him with respect to performance. I would like to see how Lua 5.4.0 would perform in comparison.
It is no surprise that the benchmarks on the LuaJIT website are going to show it ahead of PUC-Lua, is it? In truth the performance comparison is a bit more complicated than that. :)
For a pure Lua comparison LuaJIT is typically going to beat Lua by a large margin. For code that can be JIT compiled you might see ~10x performance improvements and for code that doesn't you might still see around a ~2x improvement because the LuaJIT interpreter is written in hand-crafted assembly language (PUC-Lua sticks to portable and standards-compliant C90).
But things get a bit more complicated if you add foreign C code into the mix. Under LuaJIT, the JIT compiled code that uses the LuaJIT FFI interface is the fastest, but interpreted code using the FFI is slower than interpreted code using the traditional Lua-C interface, and can even be slower than PUC-Lua using the traditional Lua-C interface. This means that when you are using LuaJIT you need to be very careful to make sure that your code is the sort of code that can be JIT compiled. If you hit a "not yet implemented" feature you can have a big performance degradation.
------
Nevertheless, Lua 5.4 has brought significant performance improvements compared to Lua 5.3, specially in integer operations (including for loops). Another big change was that the `#` is now faster. For example, on my machine the following loop runs in 1.96 seconds in Lua 5.3, in 1.26 seconds under LuaJIT, and in 0.96 seconds in Lua 5.4
local a = {}
for i = 1, 1e7 do
a[#a +1] = i
end
print(#a)
The new generational garbage collector is also a very big improvement. I've measured a 1.5x speedup on some GC-dominated workloads, where the final result was that Lua 5.4 would even edge out LuaJIT, which is still using the old incremental collector.
> I wonder if Lua developers have learnt anything from him with respect to performance.
It cannot be emphasized enough that PUC-Lua and LuaJIT have different goals. One goal for PUC-Lua is that it must be portable everywhere and be implementable in pure ANSI C (no platform or architecture specific calls). LuaJIT in contrast contains much handwritten assembly for the specific architectures it supports.
PUC-Lua also cares deeply about interoperating with C and hence why the Lua-C interface is formally part of the language spec, and as the parent thread mentioned, LuaJIT using the traditional Lua-C interface can be really slow.
That said, PUC-Lua has learned a lot from Mike Pall. It was Pall that helped them get fully yieldable coroutines across pcall boundaries. I'm sure Pall's lengthy analysis of unimpressive garbage collection performance in both Lua 5.1 and LuaJIT was read by all of them and a motivation for them to try the generational garbage collector now finally working in 5.4.
But finally, there is a new project called Pallene (not being pedantic, formerly known as Titan). It is inspired by all the lessons of LuaJIT. The name 'Pallene' is a nod to Mike Pall. Pallene is a statically typed sister language to Lua that is designed for performance and easy C/Lua bridging, but in a way that doesn't bring in all the downsides of JIT. The language is glued at the hip to Lua so there is also easy interoperability between Pallene and Lua code. Pallene introduces static types not for the sake of type safety, but for the purpose of inferring what things can be optimized. Pallene brings in a AOT compiler into the mix and generates optimized binaries. But these binaries also look like any other Lua/C native library so they are callable from normal Lua too and the user doesn't know if they were implemented in C or Pallene.
We chose the Pallene name because it was another one of Saturn's moon names that sounded nice, and is pronounced the same way in English and Portuguese. The Mike Pall thing was just a coincidence although we certainly can't deny his influence, as you have said :)
I use Lua via OpenResty / lua-nginx-module and in personal C projects for plugins and scripting. Some of these seem useful, like utf8 support and const variables.
For the multiple user data values feature, I've always found lightuserdata more useful than userdata, because it's not often that you need just one bunch of simple memory that can be freed without any other work. Rather, I almost always have more complex data, such as things that need manual cleanup like sockets, handles, etc. Or, the structure has pointers to other things that must also be cleaned up. What I'd like instead is the ability to pass a destructor callback to `lua_pushlightuserdata` so that it get's called when the pointer falls off the stack.
I don't use Lua threads or coroutines currently, so I don't have much to add there. What I do wish is that there were a way to clone or pass objects from one lua state to another to support parallelism. Basically, I'm thinking of it like the work Eric Snow is doing on subinterpreters in Python. Passing values (by copying, no shared memory) between lua instances.
One use case I had is loading a plugin that you then want to run multiple instances of in parallel with different arguments. Ideally, you wouldn't have to load the plugin multiple times (therefore calling the module level code multiple times). So I'd like to copy the entire lua state and then run each one with different arguments. There's no userdata, coroutines, or lua threads, so I don't have to worry about things that aren't possible to copy.
I got as far as looking into how to copy functions and then got busy with other things and stopped. Is there anyone else out there trying to introspect Lua function structures and copy all the opcodes, upvalues, etc?
I think that's what Lua Lanes if for. Haven't used it myself, but the descriptions sounds exactly like what you're looking for: http://lualanes.github.io/lanes/
Wow, thanks for sharing! Not sure why I didn't find this when I was looking. It does look like a very good match to my use case. I'll have to try it out.
It’s a simple, but powerful language that uses basically two features (functions and hashmaps) to implement everything. Ex: script files are implicitly functions. Global variables are in a hashmap. Semantically it is very much JavaScript without all the surprises. Semi-technical users (artists and designers) can figure it out and go on to make surprisingly powerful features.
The binary code is small and fast. The standard library is small and most of it can be removed easily. Parsing source to bytecode is quick. Loading bytecode is very fast.
It is easy to embed. It’s easy to contain it’s memory allocations in an arena. It’s easy to restrict what it can and cannot do. It’s easy to interface with C/C++. It’s extremely portable.
Your last line there is really one of the big keys. No other language is as easy to embed and interface with C/C++. The only real exceptions are some varieties of Lisp, but the "easily embeddable" ones tend not to be as mature, performant and robust as Lua.
I'm no particular fan of the language, but they do the "embed and interface with C/C++" better than anything else I've seen.
I use it as a configuration and data language, mostly for building APIs for other libraries (like SDL2).
The syntax is simple and clean, like JSON but without the most commonly cited warts (no requirement to quote keys, and comments are allowed.) The VM is lightweight enough that including it in an application is practically inconsequential. I like its modular syntax as well. There are also bindings for just about every language out there, making Lua code more easily portable and embeddable.
It's not without its flaws as a language but to me, it's better suited as a config language than TOML, YAML or XML for most use cases where you don't actually need their complexity, or JSON if you're not on the web.
For KVdb [1], my managed key-value store database, I opted to integrate Lua as a scripting language instead of JavaScript because it felt super lightweight and the syntax is easy to learn in a few hours for even newbies. I'll probably end up adding JavaScript, too, since most developers know it, but there is a simplicity about Lua that I haven't seen in many other somewhat popular languages.
I've built a collection of Android musical instruments [0] and Lua was big win for the project. I was previously familiar with Lua and Love2D framework and I wanted to see how far I can push it. The language is expressive and lightning fast. Very few building blocks that fit together nicely. I dig tables as first class objects, and meta-tables to turn them into complex objects or call delegation mechanism.
My favorite feature is that it's interpreted. The Android app is just a runtime. I can edit source files right on the phone and re-execute. It's awesome prototyping platform.
BNF is amazingly short and therefore language is easy to learn [1]
VM is tiny and can communicate bi-directionally with C / C++ code very easyly, makes it the ideal candidate for embedding in a larger app as a scripting language.
I use it as the scripting, and implementation, language for a console-based mail-client. (Think "mutt", but with real scripting instead of the ad-hoc support that mutt has.)
I use it because it's a simple language that is quick to code in, and trivial to embed inside a C, or C++, host application.
If I were to begin the project again I'd have no qualms about a similar approach. (Writing the core in C++ with a notion of display-modes, email-parsing, etc, and the actual control in Lua. The biggest painpoints were dealing with MIME, and similar broken emails which was largely irrelevant for the lua-side.)
(If you haven't heard of that, then I implore you, for the love of God, don't look it up and try it out and get addicted, whatever you do! And if you do, then don't blame me for getting you hooked. Oh dammit, now I made myself want to play it again!)
I use it as a scripting language for my game engine, but also for all the tooling around it. Premake for generating the VS solution/makefiles, but also my own tools using IUPLua [1] and a few other modules (lfs, sqlite etc). Pretty much my whole ecosystem is C++/Lua.
I used it because that's what I needed for configuring Hammerspoon. Been rather pleased with the language overall, but don't have any other uses for it. Hammerspoon has been killer, especially once I figured out some simple metaprogramming.
I embed it as a simple interactive query language for a time system at the local computer club. I wanted to use guile, but my fellow computer clubbers lose all their intelligence when they see parentheses.
Pretty much only for modding software/games that provide a lua API. If I were to create a software like that I would definitely consider implementing a scripting interface with LUA.
Most of it I'm pretty okay with. The one big pain point is lambda definitions: you have to do `function(x) print(x) end` each time, instead of `\x. print(x)` or something.
It sounds small, but with callback heavy or embedded DSL code it very quickly adds up.
I love lua syntax, its the only language I feel I actually dont have to look up syntax when I return to it a year later. Its just stupid simple, and that has a big value for me. Maybe Java < 8 also .. also pretty simple but very verbose :)
It seems that quite a lot of people use Lua.
I liked the language, it's tiny and well-made. However, I wasn't happy about the tooling support. IDE plugins, documentation generators, lint checkers – all of this seem abandoned.
Is there any similar language (embeddable, good C interoperability) with better tooling besides JavaScript?
If you want Lua under the hood, but a more industrial-strength language on top, you might have a look at Haxe. It can compile to Lua (among other languages/platforms).
The language looks really nice. I especially like its strong-typing nature. It seems, though, that it won't be easy to embed – Nim compiler requires either gcc or clang.
Well, I'm not a big fan of JavaScript :) I like the direction ECMAScript is going. However, the clumsy things are going to stay because of backward compatibility.
Also, QuickJS is a good initiative, but I afraid it's not production-ready yet. It doesn't even have versioning and/or changelog.
This is probably not the right place to ask but ..., given that Lua is commently embedded in games, anyone know of a sandboxed version? I'd love to provide a scripting language for user mods to a game but I'd like as much as possible to not have to trust the mods.
You can choose what parts of the standard library scripts allowed to use. Just exclude os/debug libraries, and that's it. Newer versions of the Programming in Lua book contain detailed instructions.
Lua is extremely sandboxable from hosting C. Just only add modules, functions and features you want added. Makes good use of the C preprocessor flags too. But also check out QuickJS as a neat alternative. But it probably doesn’t compare in speed to LuaJIT which is mostly the flavor used I think.
I'm looking through the new reference manual and there doesn't seem to be any mention of const variables, or any mention of how the new to-be-closed variables work. Do you just define the __close metamethod and lua takes care of the rest?
We use Lua 5.1 in an embedded system and it works great. We use only a few standard libraries and have several dozen of our custom libraries exposed as Lua APIs. Predictable memory usage is really good. It's several years since we picked the then current version of Lua and have had no reason to upgrade. What would be the motivations to upgrade to a new Lua version on an embedded system where there is a working version?
I don't think there's any reason to upgrade unless some of your dependencies require a higher version and backporting is more expensive that porting over all your existing code. Since it sounds like you expose libraries with your own code and you have full control over your ecosystem, I don't think it makes sense to upgrade. At least in my project I see no reason to switch away from LuaJIT (so ~5.1/5.2). It works perfectly well and I don't see why that would change.
Does this release influence LuaTex? Or rather, maybe, how is it going with LuaTex?
I've previously opted for LuaTex rather than "base" Tex or XeLaTex and I'm at the point where I'd happily stick to LuaTex and also learn some of it's more technical aspects.
Oh and one thing if you must use Windows... you better quit right away. I had to help my co worker Installing it with luarocks and it is a mess. to be fair it was easy on my Ubuntu machine.
The thing is lua on embedded has no rival. but god did it cost me some nerves.