This is full of great advice. After a brief skim I think I agree with everything except the advice on mutability.
Haoyi says "if you are actually modeling something which changes over time, using mutable [variables] is fine". I think this is absolutely wrong. You should even use immutable variables to model things which change over time (after all, almost everything does). Only use mutable variables when performance means you can't get away without them.
Other than that the guide is pretty much how I would program (as a Haskell developer, rather than a Scala one).
in a pure immutable style and show me how it's better. You will start off with lots of `.copy`s, maybe transition into lenses and other optics, and soon you'll be on the bleeding edge of research and still not have a working game that's maybe high-school difficulty if written in a mutable style
Research is cool and immutability is cool but if I can get what I want only needing some highschool kid instead of a PL-PhD that's cool too
Ah, but that's a different claim. The claim in the article is "You can use mutability to model things that change with time.". Your claim here is "You can use mutability if it makes programming easier.".
How on earth are those things related? Because using mutability to model mutable things makes programming easier, while using mutability to model immutable things makes it harder. It's not rocket science...
My intention is to try to tease out the finer distinction between where immutability should be used and where mutability should be used, because I'm personally very interested in the tradeoffs.
I'm genuinely sorry if my comment touched a raw nerve. As I said, I think it's an excellent article.
One thing I got wrong in my original comment: use mutability where performance means you can't get away without it, but also where code complexity means you can't get away without it. (My personal view is that there are many cases of the former but few of the latter, but that's an argument for another day.)
You didn't touch a nerve, I'm just being snarky because it's the internet and if I don't the internet police will arrest me.
"code complexity means you can't get away with it" seems very common. Five nested `.copy` calls is probably "too much". Lenses, in most of the incantations I've seen so far, are probably "too much". I'd love to see your solution for making
The apparent impossibility of a simple lens syntax is one place where Scala is letting me down. I hugely appreciate all the work that's going into things like Dotty, but I wish people were equally as enthusiastic about basic language usability improvements.
One of the most important things a language can do, IMO, is make basic operations easily expressible. Lensing is one of those things.
I'll add to this that it's precisely when values change over time that you want immutability most.
Because when dealing with time, you get a lot of non-determinism when modifying values in variables. This happens because the ordering of non-commutative operations is important, mutable things have identity because their "value" changes with the history of those operations and so with variables and mutable data-structures it gets very hard to reproduce a certain state or to reason about the sequence of events. So in case of errors, it's hard to reproduce a bug or to reason about what ordering you had. Note this isn't to say that operation ordering issues won't happen with immutable data-structures and values - it can happen of course, though having values and pure functions in that chain will make it easier to debug.
But then if you introduce concurrency by multi-threading in the mix, that's when the shit really hits the fan. Because then you have to guarantee a certain ordering and you do that by concurrency primitives, most commonly by intrinsic locks and for one dealing with this stuff is really hard and non-intuitive and also locks are not composable and in general a bottleneck (Amdahl's law ftw). This is why people end up using `Future` and actors and other higher level abstractions for dealing with concurrency, because they make it easier to reason about ordering. But for those that ever worked with Erlang, the "receive" function of actors is meant to be pure, instead of modifying mutable variables, like people normally do in Akka. And in Akka you can work like in Erlang (e.g. context.become), but people rarely do that. And of course, you then get the most horrible mutation known in the software industry, because in addition to having shared mutable state, you're now also dealing with asynchronous message passing on top of that.
As a final word: having the ability to mutate is having power and when mutating variables you're using that power. Sometimes it's necessary, sometimes it's convenient, but many times it's more power than you need. You don't get the luxury of mutability when dealing with accounting for example ;-)
it's precisely when values change
over time that you want immutability
most.
Values always change over time, otherwise you could replace variables by constants.
I suggest that immutability is most useful for data that crosses into a context that you (the programmer) have no/little control over. There is nothing wrong with a bit of state, like so
def pow ( n : Int ) : Int = {
var result = 1
var i = n
while ( i > 0 ) {
result *= 2
i -= 1 }
return result }
The reason this is OK is that the state changes are local, while globally (from the outside) pow remains functional. There is no need to coordinate with other programmers, or other functions or modules, to get i and n to behave properly.
In contrast, if you pass shared state into a context that you have limited knowledge or control over (e.g. shared memory concurrency) then things become hairy.
That's why I advocate: "locally stateful, globally stateless" programming.
Just a note, variables are always replaceable by constants ;-)
@tailrec
def pow(n: Int, result: Int = 1): Int =
if (n > 0) pow(n - 1, result * 2)
else result
But yes, mutability isn't problematic if it's well encapsulated. Works well for simple functions, but when you get to more complicated things (e.g. classes/modules) the problem is that proper encapsulation is very hard to do, sometimes next to impossible.
Sure, we can always translate away state and pass it explicitly.
Some people find some algorithms can be more naturally expressed using state. (And "some people" is a polite way of saying "alot".) The question of whether state can always be replaced by pure functions without loss of asymptotic complexity is open as of April 2016. All I wanted to point out is that there is no need of becoming a functional Taliban: if state is kept local, that's usually fine.
when you get to more complicated things
Yes. Then don't use it. My rule of thumb is: definition and all uses of state should be visible to the programmer without scrolling.
Note that although many prominent Scala people program in a very Haskelly style, mutable variables are significantly more idiomatic/first-class in Scala than they are in Haskell.
The fact that Scala is proper multi-paradigm, and has a first-class functional sublanguage as well as a first-class imperative sublanguage is part of its great appeal.
I have mentored quite a few people over the last couple of years in Scala. In my experience, going from imperative Java, C, Python straight to Scalaz/Typelevel/higher-kinded monad style pure functional programming never works, and drives Scala-beginners away.
What works extremely well is to start with Scala as a nice Java without semicolons, and then, as they become more proficient in the OO/imperative Scala fragment, gradually to introduce more advanced features, starting with case classes.
The reputation Scala has for being too difficult comes from certain parts of the Scala elite trying to convince everybody, including beginners, that higher-kinded purely functional Scala is the only way to use the language. Nothing wrong with this style of programming per se and it's great that people are pushing the envelope, but the conceptual gap is too big for a language learner hailing from the imperative world (which is most of them).
Oh I'm sure that's true, but as a recovering Python programmer I'm uncomfortable with justifications of mutability under any except the most extreme circumstances. ("In our codebase we avoid writing side effecting code", my colleagues said, whilst filling the codebase with side effecting code.)
This, too, will pass. Mutability is fine, if isolated and carefully accounted for. However, to understand when it's fine and when it isn't, you have to bring years of experience both in imperative and functional world.
I just sidestepped the issue by becoming a Haskell programmer and I'm glad I no longer have to deplete my judgement by making those kind of decisions :)
There's value in that, sure. But when programming one is generally modeling a business process, and some of those processes really are easiest to understand by thinking in terms of a thing that changes. So it's nice to have a programming language that lets you talk about things that change in a natural way.
Coming from an OO background the idea of hard-coding dependencies makes my palms sweat a bit. Your example is pretty simple and I agree with it but for objects/packages that are "reaching out" to the wider system rather than staying in their bounds do you not think this principle could create a loosely cohesive system? I would say that dependency injection isn't just about swapping behaviour, but it's about cohesion and bounded contexts and all that stuff that makes systems easier to reason about.
- If you want to extract them out, it's safe and straightforward. Still a tedious refactor, but a safe tedious refactor. It's not any harder than injecting stuff up-front. Given that, why do the work now if it's not any harder to do later, and you may not need to do it?
- If you hardcode dependencies except for those you can't hardcode, that means that if you come across something which isn't hard coded, you know immediately there must be some reason it needs to be injected in! This makes reading code easier, since following this convention the code itself tells you how it's used, vs having to guess whether an injected dependency is actually necessary or whether it has just one instantiation and can be hardcoded
- You are already hardcoding all sorta of things anyway. The `Seq` object? The `String` type? These, and other things, can be injected in as params or type-params if we really want to. The question is then when do you stop and draw the line? You obviously can't inject everything. "Only inject stuff that needs to be injected" provides a clear, unambiguous guideline that applies equally to dependencies on your code, on the std lib, or on third party libraries
Yeah that makes a lot of sense. I think points two and three would be beneficial to the article, because point one by itself could be said of a lot of things (eg. large methods and classes, or deeply nested and complex algorithms).
On a semi-related note, here[0] is a talk given by a gentleman named Runar Bjarnason discussing a Dependency Injection technique in Scala.
Related in that you mentioned dependency injection, unrelated in that it is not directly relevant to the content of Strategic Scala Style. (maybe of interest to some though).
Great write-up regarding good software engineering practices in general, and ones applicable to Scala development specifically.
Two things I submit to be clarified/added to Strategic Scala Style are:
1 - Explicitly declare the return type of all public methods.
Doing so ensures that type inference does not propagate implementation types to the caller (often eliminating compilation errors) and, in my experience, greatly assists in reduction of errors when working with higher-kinded types. IMHO, it also assists in understanding a system due to an explicit contract being expressed.
2 - As "Joe Clueless" wrote in the comments, why isn't Either presented as a viable alternative to Option?
Again, IMHO, while Option is an exceptionally useful type, when indicating errors at the "public API level" it proves to be insufficient. The distjoint union type Either allows a collaboration to be both explicit that a failure may occur as well as indicate what the failure was (if any).
Hopefully these comments help and, again, kudos to a great write-up.
I really dislike Either personally. I much prefer Scalatic's "Either with attitude" Or. It's also the only natural use of infix type notation I'm familiar with so that's kind of neat.
I frequently see Either abused to represent the lack of Union Types. But then I also see it used for errors. And it's not really a perfect fit for either (no pun intended).
Every (a non-Empty List) is also nice. I like Scalactic. It adds just a couple of small useful things. Very lightweight.
Option (Some or None), Or (Good or Bad) and Try (Success or Failure[T <: Throwable]) cover the bases for me.
Just in case others were unaware of scalactic. (sister library to scalatest, so if you're already using scalatest, you're already using scalactic under the covers I believe)
I don't type-ascribe all public methods. Maybe I should but I'm lazy. I don't even type-ascribe my implicit vals or defs... i probably should but i don't
As to Either, I didn't mention it because I don't use it much, and don't see many other people using it either. Sure you can use it and it'll work, but it's not the style I use or the style I see most open source libraries using.
As for Scalaz, yeah \/ is great, or cats' Xor. But this doc is all about Vanilla Scala: for every library out there, they will have their own best practices, that are totally uninteresting to people not using hat library. If someone wants to write similar documents for Scalaz, Cats, Akka, Play, Finagle, Play, they would each probably be just as long as this one, and probably just as valuable!
I get why you didn't go over Scalaz or any other of the awesome libraries out there. Which is why I didn't mention \/ and friends except as part of a tangential sub-thread :-).
IMHO your scope is spot on for Vanilla Scala and will help many.
Regarding the similar documents shout-out, here's one for Scalaz which I found very well written:
Perhaps others know of write-ups similar to yours and this one for some of the others you mentioned. Could be interesting to compile them into a "see also" kind of thing.
I rarely use Either. If logic is internal to my program, often the cause of the deviation from the happy path is implicit, and I can just use Option. But if I do want to propagate the cause, usually I prefer to model directly within my types as a sealed trait, because then I can easily model multiple causes. Of course, this comes up short if I want to reuse the same cause for multiple distinct outcomes. Either could be nice for this, but it comes up short for multiple causes, because you have to resort to nested Either. Consequently, I very rarely use Either.
What I'd really like is a simple type union syntax, so I could union the happy path result with any number of failure modes, where there isn't necessarily a common supertype for anything.
Using `Either` instead of `Option` would violate "Use the least complex approach that solves your problem".
If all you need is represent an optional value, why use `Either`, which needs a second type parameter representing the failure? If you don't care about what caused the failure (because there was no failure, an absence of value is a valid result), no need to represent it.
Besides, these two classes still represent fundamentally different concepts and they are not interchangeable at all.
My question was implicitly framed in the context of the "Error Handling" section of the paper. I agree that Option is an ideal choice for 0/1 cardinality situations and use if frequently for that purpose.
Sorry for any confusion I may have introduced when re-stating the question originally posed by "Joe Clueless" in the comments section.
> >Again, IMHO, while Option is an exceptionally useful type, when indicating errors at the "public API level" it proves to be insufficient.
> Agreed. Either, or even better, \/ (Scalaz either/disjunction) is much easier to work with and more expressive.
I, too, recommend using the Scalaz \/[0] Either type as it is "right leaning" and lends itself quite nicely to processing due to it being both a model of Bifunctor[1] (making handling either a success or failure situation quite clean) and contributing nicely to defining an EitherT monad transformer capable of abstracting both latency (the Future part) and errors (the \/ part).
For example, I often have the following defined in projects:
type FutureEither[+T] = EitherT[Future, DomainError, T]
Where "DomainError" is a top-level type representing an error encountered during processing.
All this adds up to being able to work with "happy path" logic, deferring error handling until it can be handled/reported and network request/response latency absorbed until it becomes an error (addressed as any other error would be).
No specific questions about your blog post but want to echo the sentiment that Scala is incredibly easy for refactoring. With Python (or even C++) it can be quite the chore, but as you said, building simple and using the compiler to guide you has great benefits. I'm currently going through the process right now with an AWS database backup utility I've been writing.
Since you're here, on another subject, I've been toying with using Scala.js with Duktape to integrate Scala into a C++ project as a scripting language. Have you seen or heard anyone who has successfully done this or any pointers you may have?
p.s. Thank you for all you've contributed to the Scala community. Your posts are always enlightening and your projects incredibly forward thinking!
Not relevant to your post, but I kept getting asked by data engineers who are interested picking up scala (thanks to spark): how shall I learn scala? Do you happen to have any advice for them?
Find something you want to do, try to do it, find out what you don't know, and learn it.
You could try learning from a book or a course, but personally I've never found a better way of learning "what I need to learn for X" than trying to do X and then googling all the things I don't know.
Sure, you'll forget a whole bunch of that stuff, but if it's at all important you'll find yourself re-learning it again and again until it sticks.
Scala by example and the coursera courses by Martin Odersky have helped me, but like Li said, there is no substitute to having something in mind and getting your hands dirty using Akka, Play or Spark and Googling what you don't know.
Guys please have a quick look at Concepts, Techniques and Models of Computer Programming, it's incredibly valuable. Basically the main point of the post is the principle of least expressiveness. It really makes sense after you know more about multi-paradigm programming.
The fun part of developing in scala is that it supports multiple completely different styles and paradigms, each has its own strength and weakness. You can pick the most suitable one for the functionality you are working on. On other hand it also make learning more complex. To be most effective, you need to learn these styles and paradigms and the subset of the language features you limit yourself to when in that style. Otherwise, scala is definitely one of the languages that is most capable of producing messy programs.
> The fun part of developing in scala is that it supports multiple completely different styles and paradigms.
Putting on my management hat that thought fills me with horror.
For example pull requests become a nightmare when you have extremely experienced Scala devs using every trick in the book and producing code which no one else on the team understands except those with similar levels of Scala knowledge.
Honestly, it's greek for the rest of us and furthermore it seems each Scala expert has their own way of implementing the same functionality.
I enjoy programming in Scala, I write code in it (when I have to) but in all honesty as cool as it is we find ourselves needing to stick with Python, Java and JavaScript to ensure we have some sort of consistency between new hires and experienced devs.
This is a very good point. I work on a team with 7 Scala developers of varied experience. When someone commits code it is reviewed. If the reviewer doesn't understand the code we will talk it over as a group and either learn the new technique or decide that the complexity is not worth the benefit and maybe even rewrite the commit. We encourage people to talk about new techniques before implementing so that rewrites are rare.
I agree with this 100%. Having too many ways of doing things, and no way to pick between them, results in everyone doing things every which way, all different.
That's why this document exists at all. We can definitely make things better, but to do so requires putting together guidelines, proposals, and consensus (things developers are historically pretty poor at), rather than just cranking out code and complaining (both of which developers excel at, especially fueled by coffee)
But we went through that with C++ already: all companies used to publish their own guidelines of what C++ features were allowed and not allowed to use. Of course, all these subsets were different. And of course, if you work on open source project, it's just impossible to be productive in such a language without knowing all of it.
Scala allows you to restrict yourself to a much greater extent than many other languages. You could say use assembly for everything - there's nothing that assembly can't do, so in a sense assembly is more powerful than any other language.
For me the reasons I use Scala rather than Java/Python/etc. are the same reasons I would use Java/Python/etc. rather than assembly. To write code with the same guarantees as the core of my system would be theoretically possible in a "simpler" language, but it would be incredibly verbose: it would take longer to write and be much harder to maintain (because you'd have to keep a lot more in your head to understand each piece of functionality). If I weakened the guarantees (i.e. didn't keep track of as many possible side effects) I'd have a higher defect rate, especially when refactoring.
That's the small heart of a few projects, but I see a lot of merit in using the same language for your whole codebase: http://www.teamten.com/lawrence/writings/java-for-everything... . And Scala is actually a pretty nice language for one-off throwaway scripts where you don't care about tracking any effects or similar as well.
> To write code with the same guarantees as the core of my system would be theoretically possible in a "simpler" language, but it would be incredibly verbose: it would take longer to write and be much harder to maintain (because you'd have to keep a lot more in your head to understand each piece of functionality).
I could use the same reasoning to justify writing my code in Common Lisp instead. The thing about Turing-complete languages is that the only real dimmension of power is their expressiveness - i.e. how much boilerplate do you have to write to implement something (and therefore how much of that cruft other people have to deal with when trying to understand your code).
I therefore find this principle as applied to programming languages to be illogical.
It makes sense in non-Turing-complete languages though (configurations, DSLs) - because there by "least power" one means power in the computational sense. Accidentally making a language too powerful when it's not needed can lead to security issues.
> The thing about Turing-complete languages is that the only real dimmension of power is their expressiveness - i.e. how much boilerplate do you have to write to implement something
Have fun implementing your compile-time Common Lisp typechecker.
I mean, you can do it. I have friends implementing a typechecker for Python, and it works. As for me, I'm very happy to have someone else implementing/debugging/maintaining a typechecker for me so i can do other things with my time, like watch cats climb into boxes on youtube.
In practical terms I need a typechecker to write code at anything like the rate I do, with anything like the defect rate I achieve. But that's possibly sophistry. I agree the principle is not quite right (or at least not quite absolute) as stated; using extra power has a high but finite cost, it is occasionally worth using more power than you strictly need if the benefit is high enough.
I think there's a second dimension of languages to consider, namely how hard it is to write incorrect code. To some extent these goals are conflicting, in that preventing incorrect things often means disallowing some shorter correct solutions.
> I could use the same reasoning to justify writing my code in Common Lisp instead.
Maybe you should then. I find it too hard to restrict my code in lisps: if I want a certain part of my code to be in a DSL-like constrained sublanguage, there's no standardized way to express that, only ad-hoc macros.
> It makes sense in non-Turing-complete languages though (configurations, DSLs) - because there by "least power" one means power in the computational sense.
I think non-Turing-complete general-purpose languages are possible. I'm very excited for Idris.
> I find it too hard to restrict my code in lisps: if I want a certain part of my code to be in a DSL-like constrained sublanguage, there's no standardized way to express that, only ad-hoc macros.
There is a standardized way - via "ad-hoc" macros. Except they're not really more "ad-hoc" than using classes and traits in Scala is "ad-hoc".
Maybe 'aninhumer has a point - restricting the language is kind of the opposite goal of making it expressive/powerful.
> I think non-Turing-complete general-purpose languages are possible. I'm very excited for Idris.
> There is a standardized way - via "ad-hoc" macros. Except they're not really more "ad-hoc" than using classes and traits in Scala is "ad-hoc".
They're not standardized enough to have good common tooling around restricted embedded DSLs. If you use the type system to enforce restrictions around certain areas of Scala code, every Scala tool understands it. IME a lot of lisp programmers end up writing their own tool integration, because there is no standard for that kind of restriction (macros can be arbitrary code; in practice programmers restrict themselves to sensible macros, but in a way that isn't exposed to tooling).
> Idris is not Turing-complete? :o.
In practice there are escape hatches, but it's a total language: your function must come with a proof that it terminates.
> So, you can't express the programs that never terminate, but can you express all the programs that would terminate?
This gets philosophical - would the program "Y {f => x => if(x is a proof of the inconsistency of PA via the Goedel encoding) 0 else f(x+1) } 0" terminate? If you assume consistency of PA then no.
> Point in case, you probably shouldn't ever want to use a language for the specific feature that could express a non-deterministic program.
Well the mu operator or equivalent (informally, the inverse operator) is an extremely useful feature. Many useful programs can't be written strictly primitive recursively (as a trivial example, you can't compute the Ackermann function). I have hope that we can find more restricted versions of mu that let us express all the programs we want to, but that's a decidedly nontrivial problem.
I guess the point of the person you're responding to is that following the article's advice, Kotlin would probably be a better choice than Scala in most cases.
For example, Kotlin doesn't support higher kinded types so if your problem doesn't require highed kinded types, Kotlin would be a better fit than Scala since it's the "Least powerful language that will solve your problem".
Which will probably lead to the controversial claim that "All programs need higher kinds" followed by the realization that the perceived power of a technology is extremely subjective.
Haoyi says "if you are actually modeling something which changes over time, using mutable [variables] is fine". I think this is absolutely wrong. You should even use immutable variables to model things which change over time (after all, almost everything does). Only use mutable variables when performance means you can't get away without them.
Other than that the guide is pretty much how I would program (as a Haskell developer, rather than a Scala one).