TL;DNR, use a language your company can support. It doesn't matter how suited to the job a language is, if it's single Engineer or small team, what happens when they move on? How do you support it? Who's on call?
Not Elixir, but a cautionary tale from our Erlang project. ~8 years ago a our IoT backend was written in Erlang, this was the early days of IoT, so sure it made sense as a technology, could scale well, handle all the symmetric sessions, etc. It was a good tool for the job and in theory could scale well.
But, there's always a but. We're a Python shop on the backend and C on embedded devices. Engineers move on, there some lean times, and after a few years there was no one left who could support an Erlang system. So we hired for the position, and it's key infrastructure, but not really a full time project, so we hired a Python+Erland person.
But! Now they're on call 24/7 since the regular on call roster are python people and when the Erlang service goes wrong, it's call the one engineer. So, do you hire 2 or 3 people to support the service? No, you design it out. At the time is was 2018, and IoT services were a plenty, so we could use an existing service and python.
Another way of looking at it, let's say it's a critical service and you need at least 3 people to support it/be on call. If each engineer costs $200k/year, that's $600k/year. Does this new language save you that much over using a more generic and widly known language in the org?
Your comment was so spot-on that I suspected we worked at the same company for a minute.
Erlang / Elixir are languages that are easy to begin learning, but difficult to master. In our case, the team was able to get the basics up and running quickly, but the service became increasingly difficult to manage as our product requirements expanded beyond the strengths of the Erlang ecosystem's core competencies.
In our case, the original team of developers was excited to write a greenfield service in Erlang, but much less excited to maintain and scale it long-term. Some of them used their experience to speak at Erlang conferences, which led to them being recruited away by companies who were desperate for Erlang developers to maintain their legacy code after the core team left. Ironic.
At first we tried to train existing engineers on Erlang. Doesn't work. Not because Erlang is bad, but because people don't want their companies changing the direction of their career development just to support someone else's legacy project that requires an on-call team to support.
So your two options are: Hire at least 2, preferably 3 Erlang engineers for the reasons discussed above. Harder than it sounds, because AFAICT there are a lot of companies with 2010-era Erlang backends that made sense at the time but now require dedicated maintenance engineers.
Or you can rewrite it in a more common language, which is the route we chose. Rewrites are a difficult decision for well-known reasons, but ultimately it was the correct choice. Being able to drop 10+ engineers into a project as needed is a huge accelerator compared to funneling 100% of your work through the 2 Erlang gurus in the corner who maintain a codebase that no one else wants to touch.
Hiring is nearly impossible, come support our legacy system, and what ever you do, don't introduce Erlang into any other part of the company. Why? We feel burnt on this one project, and god forbid we get any deeper than we are now. What this isn't appealing?
... whom an inexperienced hiring department entirely trust when they assure that the technologies they are already familiar with are the best fit for the company, let's hire only people who think the same?
I'm not sure if you're being obtuse. But rewriting your stack in a language no one on the team knows because someone on HN said it's a better fit for your usecase is something no good engineers/managers would think is a good idea. If a potential hire thought that was a reasonable thing to do, that would be a major red flag - a sign this person lacks experience, and any kind of business understanding.
at some point you end up with path dependence -- you're in the situation you're in, and it would be big-bang rewrite or incremental migration to port all the code to a new stack, not to mention the costs of retraining existing staff to be able to effectively develop in and support the new stack, or replacing the staff with different staff who know the new stack but don't understand the existing business and code. so maybe the previous decision to pick tech stack A over tech stack B when starting the product was pretty sound given the circumstances, or maybe it wasn't, but it doesn't matter that much from the perspective of deciding what to do next.
Oh yeah, I know all too well about path dependence. That's how we end up with a Rails monolith that's written in a very standard/structured way so that many devs can contribute, add type annotations for safely/docs, etc. Using no types is like having no tests and thinking about adding them later. Usually better to have it in place as you go.
> there was no one left who could support an Erlang system.
If you have $200k "python engineers" on the payroll who wouldn't jump at the opportunity to do some additional Erlang, maybe it's time to reconsider your hiring practices and that is the real cautionary tale.
Back in the day people would balk at hiring Python programmers saying, "there are so many more Java programmers", and I used to say, "Why would you hire a Java programmer who was unwilling or unable to learn Python?"
Same logic applies here: Why would you hire a Python programmer who was unwilling or unable to learn Erlang? (Especially if you're going to pay them to do it!)
If you can't switch languages you're a kind of technician not a programmer.
FWIW, having done it professionally for ~15 years I'll never write another large project in Python again. Erlang's runtime is so much better than Python's it's just ridiculous. I feel stupid for not realizing this sooner.
If I had a Python job (a language I enjoy) and I was forced to work in a language I don't (say, Perl), I'd start complaining about it and looking for something else to do. I personally wouldn't mind working with Erlang, but that isn't the point. Many developers "can" do these things but it doesn't mean they want to.
It's pretty much how we treat our Python projects. All of us hate it, but data science is done in Python so we still need some percentage of the engineering team fluent enough in it to sustain operations with it (on-call, maintenance, bugs, eventual decommissioning).
The moment we can get rid of it, you bet we will. But it's much easier tearing it down when you understand it enough to discern designs driven by the limitations of the tech vs requirements for the product. Black box ground up replacements are hard and expensive compared to ports. Also, thankfully it's Python and nothing something more difficult/complex.
At the end of the day, it's a job, and there's an expectation of professionalism to do what's necessary. A lot of people on HN like treating tech jobs like a paid hobby which I think is detrimental to career development.
Regrettably, also back in the day and probably around the same time, companies who wanted to hire "outside of the box" asked for Python. Not because they were python shops, but because they figured that people who had an interest in a new but not-quite-commercially viable language must have some sort of intrinsic interest in programming, and would likely make for better programmers.
This is because those shops read the PG essays, and think that the the shibboleth is Python, it is not. The marker is that that someone is at the fringe and not in the middle of the flock.
I have met lots of engineers who are smack dab in the middle of the flock when it comes to tech stacks, or even _behind_, but they were more than capable. Hire for what you need. But what you think you need and what you actually need is probably not know to you.
Depends on the reason for switching. If I'm asked to switch because the new stack is going to have measurable improvements plus the language is going to stay and grow (just as examples may be Go or Rust right now), then sure. On the other hand, if the reason is because the company fucked up by relying on one idiot with an elitist mindset about programming languages, then that request going to have to come wrapped with very good incentives. If I'm a great java developer with tons of jobs to choose from, you are going to have to give me a very good reason to narrow my opportunities down for you. Otherwise, I'd say no and move on and you can go hunting for devs for a niche language and pray to Gods that the hiring situation will improve for you.
Your example of Java to Python doesn't represent the gravity of switching from Java to say Erlang or Haskell. You assume someone says no because they are unwilling or unable. We are unwilling, sometimes, because of the opportunity cost we have to pay for your fuck up.
> If I'm a great java developer with tons of jobs to choose from, you are going to have to give me a very good reason to narrow my opportunities down for you.
Why would learning a new tool narrow rather than widen your opportunities?
Learning an obscure tool that eventually dies is a waste of time. Think learning a language only you and a few of your friends speak. This in itself is good enough reason to be cautious about what you invest your time in learning.
Languages and specially the tool chain and the way of doing things around them evolve. If you spend like 5 years doing something else and come back to it, you'll find things have changed a lot. You may argue that it's easy to relearn this. But from a job perspective, the guy who has been using the exact same tech stack in the recent years will often be a better bet than a guy who wrote Java in version 1. This is very true for C#. Syntax, design principles, tools, method of deployments all have changed rapidly. Professionally, you want to be as up to date as possible in the area you worked in or you get rusty. You can learn new things on your time for their own utility. Flushing down 40-60 hours a week on a useless piece of knowledge is bad all round to me.
> Flushing down 40-60 hours a week on a useless piece of knowledge
That is only if it's actually useless. I haven't programmed in haskell for many years, yet I am very glad I did. It was hard at first, but it made me think about code, data and transformation from a different perspective, expanding the repertoire of paradigms, regardless of which language I code in.
YMMV, but personally, I'd much rather hire or work with someone with 3y java + 1y erlang experience, than just 5y java experience. In general a polyglot rather than monoglot, especially if it's a different coding paradigm. Nevertheless, I may not actually want them coding random bits of work in erlang or whatever random language.
Seriously, why doesn't python have a maven equivalent yet?
And no fat-binaries. Nor self executing binaries. It's just as old and having em basically just boils down to enforcing conventions. But nope, nothing. There have been attempts by some such as shiv, but really... None are even remotely as usable as maven is.
I've been a happy Python programmer for over fifteen years, so this comes from a place of love and respect.
Beyond roughly 150k LoC you start to run into difficulties keeping everything straight over time, especially if you use a lot of indirection or other magic. The aspects of the language that help you go fast when you're prototyping (is "RAD" still a thing? Rapid Application Development) start to trip you up when the codebase matures. There are a lot of fancy footguns in Python that catch you if you stop dancing.
Tooling is getting a little better (MyPy, et. al.) but at the same time the overhead of adding explicit type information erodes the advantages Python has over, say, Java or Haskell.
This is true of all dynamic languages. Rails codebases become a nightmare beyond a certain LOC. It’s taken a long time but now that we have languages with strong static typing but good type inference like Kotlin, Swift, Typescript etc the old fully dynamic languages are obsolete technology and will eventually be replaced.
from personal experience with mypy, the great thing about gradual typing in a dynamic language is that, well, it's gradual! when I want to do something extremely dynamic, there's no horrible reflection API in my way. but most of the time I can live happily inside the walled garden mypy provides.
I have adopted this attitude for personal projects & small companies, and it was mostly great.
There was at least 1 really large error that slipped into production because we were mocking an untyped service incorrectly (a mistyped property name), and the test passed, but prod exploded. It was embarrassing, and I certainly would have caught it with more mypy discipline, but overall the pros outweighed the cons.
At big companies that use mypy, however, this is problematic. If teams only define types for the straightforward parts of the system, then the types become less useful, especially as the company scales. From what I’ve seen, individual teams use an all-or-nothing approach with MyPy, so it’s unusual for an individual dev to sparsely type anything.
agreed - I am thinking more in terms of the common distinction between infrastructure/libraries and application code - I can write dynamic, well-tested libraries that are used by heavily annotated, checked application code. The types are generally more meaningful at that layer, and it tends to evolve more quickly anyway.
Completely agree. I worked at a company on a Common Lisp project where the Python crew refused to learn Common Lisp because they thought it would look bad on their resumes. This should be a litmus test for programmer quality: If they don't want to learn something because it's not "fashionable", they're probably lousy programmers who lack confidence in their own abilities. Do you really want these people working for you?
> Completely agree. I worked at a company on a Common Lisp project where the Python crew refused to learn Common Lisp because they thought it would look bad on their resumes.
They can put only "Python" on their résumés, right?
The reasoning is simple: if someone thinks something would look bad on their resume, they probably don't want to spend time on it, either.
For example, I wouldn't want PHP or Fortran on my resume, and, consistently with that, I avoid them. I don't have an opportunity to actively avoid them, but that defense mechanism would leap into action, if called upon.
However, I know a thing or two about these; I'm not simply applying "meme-based reasoning".
This really is resume driven development into a whole new level. No wonder why some sane Programming languages Never made it. And it is the reason why I am worry about Ruby's future.
It's not a resume driven development but a proper allocation of personal time and skills planning for the future. What's the point of learning something you won't use in the future? If for the sake of learning then there are a lot of better choices.
Just don't base your choices on the spreading of ignorant memes.
Would you want to pass on Lisp because of the ramblings of a rookie web developer with no Lisp experience? That's what you might do if you believe the "Lisp Curse".
I'll be the first to admit... I'm a little conflicted by this..
I've almost exclusively worked on the usual procedural /OO languages professionally (java /c#)..I like to experiment and I've picked up a bunch of languages along the way (perl, python, coffeescript, f#, nim etc).. And its usually because I want to see what's different and spend enough time to do something useful (but not large) with it. Also, it helps you become a better programmer IMO. otoh, if I'm hired for my skills but then have to work on js( don't actually like the js eco system) or haskell (I gave up after trying and I doubt I'd become very good at it anytime soon if I were forced into it) I'd be not happy either (and IMO, with valid cause that doesn't reflect on me or my abilities)
> If you have $200k "python engineers" on the payroll who wouldn't jump at the opportunity to do some additional Erlang
Ah, sure they might be willing, but as a lead I'm not willing to invest in it.
1. I'm not sure I want someone who's got 4 weeks of experience in a language working on system critical infrastructure. Pick your time, does it take 6mo?
2. They have existing work, learning and working on an entire new language _AND_ new project is a huge task. Especially for non-trivial projects.
We could, but in our case it was a better engineering decision was to retire the old system and move on.
This kind of sentiment come up every so often on HN and I'm just sick of it. Sure, programmers should strive to improve their craft, just like professionals in every industry. But there are plenty of ways to improve yourself and learning a new language is hardly the most effective choice (and in fact could be counterproductive).
How does learning a new way to think about a problem become counterproductive? Each language you learn gives you a new angle from which you can approach a problem. Is it counter productive to learn a second spoken language? I haven’t come across an argument against that, even learning a dead language like Latin or Ancient Greek has benefits.
Perhaps counterproductive is too strong a word, but there could definitely be problems in the short term. Programmers love to apply things they learned in their work, and taking a concept or pattern from language A and applying them to language B is a natural reaction. The most common culprits are monads and actors. Ergonomics (readability, cognitive cost, debugging, IDE support, etc) became afterthoughts in the midst of excitement, and the long term health of the codebase (and teammates) suffers.
I have made mistakes like that (introducing monads/future and making the team learn guava) and I still regret it. Its not to say futures are useless - there were a lot of async logic in the codebase and futures solve a real problem, but its not clear if the benefits outweigh the cost.
To be fair, if you’re rewriting an Erlang system in Python/JS and you don’t feel like you’re losing much, Erlang was the wrong choice from the beginning. The cases were Erlang shines are very hard to replicate in these languages.
> The cases were Erlang shines are very hard to replicate in these languages.
I'm just beginning to get interested in the Erlang ecosystem. Could you please add more color to this and give some examples where Erlang shines? IIRC, Whatsapp backend used to be on Erlang.
That would make sense that whatsapp backend is in Erlang. Erlang was developed for telecommunication systems where you need performance and the ability to hot swap code.
If you're claiming that Erlang is no better at hot-swapping code than "all languages", you haven't seen Erlang's hot-swapping abilities.
"All languages" provide you the ability to run arbitrary code, and thus theoretically support hot-swapping, but if you try to put it into practice you'll soon run into real obstacles you hadn't considered that either bite you or at the least make your job a lot harder. Erlang provides those practical solutions that make hot-swapping easy (in other words, practically possible).
TL;DR: On the question of buy vs. build when it comes to hot-swapping abilities, Erlang falls on the left while "all languages" start at the right.
> I have seen it, it is no better than Python's, exactly as I said.
I'm going to have to call your bluff and ask you to substantiate your claim. Please show where Python provides, as a part of the language and/or the standard library, the equivalents of these Erlang built-ins:
1. Module versioning (via vsn)
2. The ability to run multiple versions of the same module at the same time, transparently and for as long as needed.
3. The versioning (via Module vsn), tracking (via handle_call/3 and derivatives), and migrating (via code_change/3) of State.
4. The dependency tracking and ordering between code updates of multiple modules (via release_handler)
These are the native features that make hot-swapping in production not only feasible, but also powerful and easy. Of course, these are built on top of foundational primitives that are possible in any language, sure, but not all languages (including Python) have these natively available.
----
> Hot swapping Erlang in production is rarely done, and if it is done, it mostly done during debugging, not serving production traffic.
> I invite you to do a straw-poll on the erlang mailing list.
I'm sorry, but this point you're introducing now is unrelated to the discussion we're having, so I'll refuse to respond to this in any way except to remind you of the point we _are_ discussing:
2. To which you responded: "Erlang's hotswap and Python's hotswap are of comparable power, ie, both are crappy. Being able to do something isn't the same as doing it well.
Hot swapping support in Erlang is first class, but even with that, it is not an easy feat to accomplish in large systems. Which is why, in my experience, people tend to avoid it if they can do without it. Systems i have worked with either had scheduled downtime periods or one could easily take down a node for upgrades without impacting the general availability of the system. In such cases, there is no real incentive to invest time and energy in managing hot swapping. But if you absolutely need all parts of your system be up 24/7, Erlang has the best story I've heard so far.
Basically anything that is protocol-oriented like WhatsApp. A lot of people will say fault tolerant, low latency, highly available and massively concurrent systems, and they’ll be right, but I think that’s only half the story. You also get a battle-tested blueprint for building these kinds of distributed systems. You will find a pattern here and there in other languages and VMs, but the cohesiveness of the package (language + BEAM + OTP + observability + patterns) is something really unique. Compare this to the Java ecosystem, where there are dozens of concurrency primitives and an even greater number of distributed systems libraries and frameworks like Akka, Mesos, Zookeeper, Storm, Flink, Samza, Heron, Helix...
If the problem you're solving can be easily split into many independent, low volume, maybe long running, communication oriented tasks, you owe it to yourself to look into the Erlang ecosystem. Think handling signaling for calls, chat connections, inputs from millions of IoT endpoints.
Used to be? I know WhatsApp switched from FreeBSD to Linux and swapped hundreds bare metal servers to thousands of VMs.
I dont think WhatsApp ever swapped / rewrote their system in another language under Facebook.
Key phrase right there. Because it doesn’t matter what language a system is written in: if management doesn’t bother to hire replacements until after everyone who understands that system and why it works the way it does has already left, of course it’s going to crater shortly thereafter.
That’s not a language problem, that’s a business continuity problem. And it’s lamentably common as dirt.
Any monkey can learn to churn code. Learning the business domain, what its problems are, and how to solve them; that’s the bit that actually matters.
Because engineers don't take kindly to companies changing the direction of their career development.
If you're an up-and-coming Golang developer, how happy are you going to be when your company takes you off of a Golang project and makes you the new Erlang person?
Now how happy are you going to be when you're informed that you're learning Erlang to maintain an old, flakey codebase written by 2 guys who left the company and are now unreachable?
Oh, and you're going to need to be on call for this because no one else knows how to fix it.
> If you're an up-and-coming Golang developer, how happy are you going to be when your company takes you off of a Golang project and makes you the new Erlang person?
In my mind good developers are not <language> developer, they are engineers that solve problems with the right tools.
> In my mind good developers are not <language> developer, they are engineers that solve problems with the right tools.
Given infinite time and no deadlines, sure.
But in the real, you can't expect your engineers to learn entire programming languages and associated ecosystems on the fly as they debug issues in production.
If your Erlang-based cloud backend starts going down, you can't afford to wait around while engineers teach themselves Erlang so they can begin to even debug the problem.
The point is that if you write critical infrastructure in a specific language/framework/ecosystem, you need to have people proficient in that ecosystem who are ready to go. When dealing with production systems at scale, it would be downright negligent to assume you could simply learn the language and framework on the fly if any problems come up.
I can kind of get this if we were talking about COBOL or MUMPS or something. But being paid to learn Erlang isn't really in the same category, or the same universe, really.
Maybe the emphasis should be more on the disappeal of your role being changed to maintain an old, flakey codebase than on the language.
So how about don't be stupid about it and instead make three engineers devote 1/3 of their time to this? That way it's a broadening rather than a redirection of their career, they can help each other, the on call duties are shared, and the business isn't dependant on a single person.
To me, this is the sniff test of a good engineer. All the good engineers I've known would've jumped at the opportunity to learn a new technology on company time/money. The engineers that refuse to learn generally don't grow.
Yup pretty much. Esp when it’s something like Erlang that passes the sniff test as something that likely will expand your knowledge in a meaningful way, and be a long lived technology.
Yes, it doesn't matter if it's a programming language or another technology. That's engineers not worth keeping around, stuck in their old ways, sooner or later they'll make themselves obsolete.
I'm not a <language> programmer. I'm a programmer period.
The term used within my circles is 10x1 engineers -- they've replicated their first year of experience/learning 10 times over. Biggest thing to watch out for when hiring "senior" devs
Dedicating a third of my time to a language I had no previous intention of learning is going to be a hard sell, even if you label it as "broadening."
If I'm a python developer, I'm now spending less time on my core competency, giving other python developers an edge. Seems like you're actually narrowing my career, unless I wish to apply for jobs seeking mediocre Erlang programmers...
If you see your identity as an engineer to be one grounded in the languages you use, you have commoditized yourself, or are at least very junior. Programmers solve problems, usually with a mix of deep skill sets revolving around domain knowledge or knowing how to use computation, systems, etc to solve them. Competency in a specific programming language is far down the list of what defines the value of a specific individual engineer to an organization or project.
Well, that’s not how programmers are recruited in the real world though. Java shops will hire Java programmers, C# shops will hire C# hands, etc etc. Marketing yourself as a generalist problem-solver will get you, at best, in one of those nightmare scenarios where you’re the only programmer in a non-tech company.
I specifically was referring to your identity as a programmer, not how you market yourself. Those are separate things.
The OP said "I'm a python programmer", and discounted the value of learning something new like elixir because it would undermine this due to the opportunity cost, which implies they may be falling into the trap I mentioned. All other things being equal, someone who is reasonably competent at both Python and Elixir is going to probably solve many problems in a better way than someone who didn't have a broader perspective. (In part because learning Elixir stretches well beyond learning a programming language, it is an introduction to an entire distinct continent of distributed systems knowledge and technology - OTP, BEAM, etc.)
That's why you don't market yourself as a generalist problem-solver. You market yourself as an expert in C# AND Python AND Swift AND... whatever else you know how to use. With appropriate signalling to communicate your experience levels with each.
There are lots of senior roles that require oversight of multiple projects that are working with different tech stacks. Some companies specialise in a single language. But lots of companies have say 2 or 3 backend languages (say C++, Go, and Python), plus a website in JavaScript plus mobile apps in Java/Kotlin and Swift/Objective-C. Being able to dive into all of those makes you incredibly valuable to this kind of company.
>You market yourself as an expert in C# AND Python AND Swift AND... whatever else you know how to use
So that everyone knows you're full of shit? It takes years in a technology to become an expert at it. There's no replacement for the experience of taking multiple projects from inception to production to scale. That's how you learn about the inscrutable errors, fiddly configuration issues, and pitfalls that make you an expert at something.
"Javascript" though isn't Javascript. I mean it could be, but it's worthless in that context because what you actually need to be productive is familiarity with the browser DOM, current frameworks for frontend or backend, and basic project structures.
For Java it might be enterprise Java, it might be Android, in which case you need familiarity with those frameworks to be productive. For Swift it'll be iOS.
No one's "diving into" the codebase for an iOS app starting from "I mostly do backend web app development in Golang".
Are you an "expert" in all those things? I doubt it. In fact the worst possible thing you can generally write anywhere is that you're an expert in C++ which at this point is oh-so-many subdialects.
> No one's "diving into" the codebase for an iOS app starting from "I mostly do backend web app development in Golang".
I mean, I've certainly been expected to at previous jobs (my background being backend and frontend web). It was a steep learning curve, but in retrospect it was also an excellent learning experience.
> Are you an "expert" in all those things? I doubt it.
No, but give me time. I'd consider myself an expert at frontend JavaScript/TypeScript and backend JavaScript/PHP. I have a decent amount of experience (enough to get a job an X developer, not enough to call myself an expert) in a number of other technologies (e.g. Python, Rust). And at my current day job, I'm responsible for a React-Native project through which I'm getting exposed to a lot of the quirks of the Android and iOS platforms.
Go I've not used. But Go's a relatively small ecosystem to learn. I don't put Go on my CV, but I'd be confident applying to a Go job. C++ I've done a little of, but I'm absolutely not an expert. Although Rust experience means that I understand most of the low-level concepts and best-practises that aren't C++-specific. Still, I absolutely don't advertise that on my CV either. One day though maybe. I'd like to do some embedded stuff at some point.
> In fact the worst possible thing you can generally write anywhere is that you're an expert in C++ which at this point is oh-so-many subdialects.
C++ is a bit of a special case there though. Because
1. C++ is huge: it takes a lot longer to master C++ than it does other technologies.
2. C++ is wildly unsafe. Therefore the consequences of not knowing what you are doing with C++ are much greater. You'll likely end up with a crashy program riddled with security holes.
In contrast, I wrote a production Rust project with no prior experience, and all the things that would have been crashes or security issues in C++ were compile errors. So the only consequence of my lack of experience was that it took longer to complete the project.
>If I'm a python developer, I'm now spending less time on my core competency, giving other python developers an edge.
so this argues against the theory that learning other languages and other programming paradigms makes you a better programmer in the languages you already know.
That depends what you learn. Spending 3 months working on a Rust project has done a lot more for my ability as a JavaScript developer (my strongest language) than spending those 3 months working on a JavaScript project would have.
FWIW I agree with you. I'm a Java dev and learning a language that's similar to it (C, C#, Python, whatever) does nothing for me but take away from learning more of my language.
Other comments say an engineer is supposed to solve issues with the tools given to them. I can solve issues in other languages, I just don't want to. How does that make me a bad engineer?
Productivity and working knowledge of a tool do not count as rational merits? It takes years to really learn a language; a new framework and language per week is a stupid fad.
If I run a Java shop I want a clique of Java devs, not Golang or Rust devs.
There is some truth in this, but less than commonly believed.
A "multi cultural", Jack-of-all-trades kind of team, that have a broader knowledge of many tools and techniques but less in depth familiarity with any of them, is more likely to use less features from the languages and tools that they have picked for a few of their strong points only, and less likely to indulge themselves with a level of sophistication that benefits their ego (and job security) more than the project.
> If I run a Jave shop
So I know a Python shop must be a business that sells reptiles, but what does a Java shop sell?
A good engineer will prefer some language(s), but be able to figure out anything should the job require it. This career track shift stuff is complete bullshit.
The workers are settled and and don't want change. And there's nothing wrong with clocking in and out and not giving a shit about your "career". But there is something wrong with the managers falling for this.
> So how about don't be stupid about it and instead make three engineers devote 1/3 of their time to this?
If your product-critical cloud backend depends on people being knowledgeable in a specific programming language, you don't make it a 1/3 time side project for multiple engineers.
Also, if you start giving people "1/3 time" responsibilities, you're one step away from "everything is top priority" territory.
The point is: If you're going to write critical pieces of your infrastructure in a specific language, it's a requirement to have at least one person full-time dedicated to that piece of infrastructure.
When things go wrong at scale, you can't depend on a group of people for whom this is a part-time side project.
My least favorite part about HN is trying to share advice based on real-world experience, only to people show up and armchair quarterback the situation with solutions that sound easy on paper.
You'll have to trust me that we didn't just throw our hands up in the air when dealing with the Erlang situation. We tried a lot of the suggestions here and more.
Erlang is one of those languages that, for whatever reason, lulls engineers into a false expectation that they can pick it up over a few weekends and start knocking out IoT-scale solutions serving 6- and 7-figure connection counts. Speaking from direct experience, that's not only untrue, but it's a dangerous misconception that leads to a lot of pitfalls.
Don't get me wrong, Erlang is great if your problem set matches up neatly to core OTP functionality and well-defined patterns, but you don't have to stray far until you start finding difficult pitfalls that aren't easily addressed by engineers learning Erlang on the fly.
Sorry if I wasn't clear. I'm not questioning why you switched from an otherwise unused technology. Sometimes it's the best way forward. My comment is meant for this sentence:
> Also, if you start giving people "1/3 time" responsibilities, you're one step away from "everything is top priority" territory.
I like to maintain and improve legacy systems and usually happy to do it. When I got a separate side-project I had constantly fight with my team why I should work on project X instead of contributing to our core project. If it was my whole team responsibility they were happy that somebody does it. Also, they wanted to invest enough time so it won't break.
Because "learning" Erlang is like learning any language. Sure, you can learn the language in short period of time, but really being able to professionally support a production level app to the level needed will take a long time and a lot of practice. And that practice isn't a week long course where you suddenly learned Erlang and then that's it. You aren't working with Erlang any more.
You don't need someone "learning" Erlang. You need someone who works with it.
Probably cost of training. You'd have to hire consultants to teach best practices and that could cost significantly more than hiring a new engineer. Plus there are other risks to maintaining two codebases in different languages.
Whether you hire an outside consultant or not doesn’t matter. You have to pay the cost one way or another, either by your employees spending time learning the language + experimenting + making mistakes, or by going through some training and hopefully making less mistakes (not guaranteed).
Not saying this applies to every company, but I've worked at two different companies that have done something similar:
* Healthcare startup that basically outsourced its entire infra/security team from a consulting company. The consultants rewrote everything and took over engineering leadership, basically acting as gatekeepers for anything SRE-related.
* Worked at an older DNS company that hired a bunch of consultants to help engineers migrate their web framework from Play to Spring. Said engineers used Play because it was shiny but didn't have the Scala expertise to keep maintaining it, so they moved to pure Java.
I had the same reaction. However, I work for a small team with driven developers. It is expected from our engineers to learn “best practices” on their own time when we introduce new technologies. We are also compensated extremely well according to these expectations. If you have a team of engineers with a different mindset, you will need consultants.
Yes that. But also only use paradigms which the company can support.
For example good static typing is not just about adding type declarations but about taking advantage of the type system to the right degree. (Like don't ever overdo it, it's killing productivity most times).
But what degree falls into the productivity boosting spot and from which point on it will hinder productivity depends a lot on the team.
The problem often starts when the team changes and e.g. a pure scala/rust/C++/Ocml team get people cross entering from languages without any usefull static typing.
What had been reasonable and useful might now be a hindrance to the team!
I ran into a situation where this caused massive delay. (Well through the actual root cause was miscommunications, people not complaining about problems and instead trying to find ways to fix them behind the back of other people force pushing nice looking simple solutions which just happen to fail many edge cases, killed the write, refactor, test cycle and increase maintenance cost longtime :( )
So well 1st never overuse type systems. 2nd communicate problems, especially if you have problems understanding anything even if it just takes longer then you think it should.
Our adoption of TypeScript led to a significant reduction in defects, and it definitely isn't killing productivity. Most of the time we take advantage of existing types so if anything it speeds things up in the long run.
You are misunderstanding what I mean. Through I haven't be to clear I think.
What I mean is code which not only uses types and takes advantage of the type system but overdoes it. In many languages with a powerful type system you can encode much more then just basic types and this is useful to reduce possible error cases. But if you overdo it the type system part becomes too complex, type error messages too unreadable and it hinders productivity and on-boarding.
I'm a big fan of static typing and believe that more encoding more constraints in the type system is _generally_ good. But if the resulting types become to complex it often not worth it.
I think I understand this one. I would restate it as "Powerful static type systems usually encode more constraints than you can easily reason about." And that's good(it catches your errors) and bad(you start to fight the compiler instead of solving the problem).
With simpler, more gradual designs like what are in Typescript or Haxe or even static, native-code languages like D, you have more escape hatches available, so you can iterate on the specification a few times and get code running quickly but also aim to make it tighter over time.
No what I mean is not that it encodes more constraints than you can reason about, but that the way some libraries abuses the type system to encode some of the constraints can cause more problems then it solves.
> Does this new language save you that much over using a more generic and widly known language in the org?
This is a great question to ask. It’s worth extending beyond language to frameworks and libraries as well.
It’s important to optimize for the development experience across all the apps in a team. Especially once the excitement of using a new language is gone and there are bugs to fix.
I‘ve rewritten a few projects in a common language for the org. Each time the biggest benefit was how straightforward it became to work on. Problems you’ve found somewhere else become easy to spot (and search for). Writing another test is the same as the last project you had open. Anyone in the org can contribute with a minimal learning curve. This can be the difference in getting bugs fixed in a day or a month.
I think one approach when choosing more niche language might be, first look for a book like https://www.erlang-in-anger.com/, read through it and consider, if this is the sort of stuff my colleagues will want/be capable to deal with.
Then consider how hard it would be to get more people that have used such language/ecosystem in anger.
Side note: Pagerduty comes with many hilarious sounds, including a very realistic meow. Thus we practice meow driven development: I don't want my phone to meow desperately.
Not Elixir, but a cautionary tale from our Erlang project. ~8 years ago a our IoT backend was written in Erlang, this was the early days of IoT, so sure it made sense as a technology, could scale well, handle all the symmetric sessions, etc. It was a good tool for the job and in theory could scale well.
But, there's always a but. We're a Python shop on the backend and C on embedded devices. Engineers move on, there some lean times, and after a few years there was no one left who could support an Erlang system. So we hired for the position, and it's key infrastructure, but not really a full time project, so we hired a Python+Erland person.
But! Now they're on call 24/7 since the regular on call roster are python people and when the Erlang service goes wrong, it's call the one engineer. So, do you hire 2 or 3 people to support the service? No, you design it out. At the time is was 2018, and IoT services were a plenty, so we could use an existing service and python.
Another way of looking at it, let's say it's a critical service and you need at least 3 people to support it/be on call. If each engineer costs $200k/year, that's $600k/year. Does this new language save you that much over using a more generic and widly known language in the org?