I find systems design and complex adaptive systems fascinating. I really recommend the book, "Hidden Order: How Adaptation Builds Complexity" by John H Holland. It is technical but aimed at the educated, non-technical reader, with enough description that you could code along to his models. I really wish he had written more books. RIP.
Another good source of systems thinking are the books by Jane Jacobs on urban planning.
I think anyone wanting to study this area rigorously would need to spend time doing detailed discrete event simulations, understand probability theory, time series analysis, phase space analysis, experimental design, graph theory and queueing theory.
Like databases or operating systems, complex systems embody many fields of study simultaneously. One of the funnest parts is that they can be modeled at so many different levels.
I find it frustrating that this article doesn't explore the tension between the "never rewrite" and ARM anecdotes. Iterating is good, but iterating to a whole new place is often impossible, and the biggest disruptions come from being in a different place. It's a core tension in systems design, and one that this (otherwise good) article doesn't acknowledge.
Do read "The Tyranny of Structurelessness", though. Great read. I enjoyed the analogies to distributed systems here, too.
Intel's StrongARM (originally DEC) chip had the highest perf/watt of any cpu in the industry. They later sold StrongARM to Marvell for 600M. Not only couldn't they create it, they couldn't continue to sell it even though it was widely successful.
The "tyranny essay" is wonderful on many levels. It is mostly about human social-political structures, if one wanted to be true to "structurelessness" you would have to have a constructed double-blind system to enforce it. Otherwise you get fascist overlay that breaks along existing personal power dynamics. The in-group, central cadre and then everyone else on the periphery.
Structureless movements can be extremely effect at scaling an organization, but at some point you need to collapse that chaos into something cohesive.
> This lack of structure, Freeman writes, disguised an informal, unacknowledged, and unaccountable leadership, and in this way ensured its malefaction by denying its existence.[3] As a solution, Freeman suggests formalizing the existing hierarchies in the group and subjecting them to democratic control.
Due to the writing style, I found it really effective for my understanding was to have chatgpt summarize and explain passages, and then have it explain it again.
Well I think if Intel had decided to start from scratch to make "x86 only more power efficient, it'll only take one product cycle" they would have failed miserably, but if they said "let's make a phone cpu, that's our new top priority" that program could have grown into making great laptop cpus a decade later.
Making a new product that fills a new niche in a short timespan is viable. Making a product to directly replace what the old system does after decades of iteration is usually a delusion.
This is fun from the article: "Normally we think of the CAP theorem as applying to databases, but it applies to all distributed systems. Centralized databases do well at consistency and availability, but suck at partition tolerance; so do authoritarian government structures."
> Back in university, students used to tease the Systems Design Engineers, calling it "boxes and arrows" engineering. Not real engineering, you see, since it didn't touch anything tangible, like buildings, motors, hydrochloric acid, or, uh, electrons.
Hey, I resemble that remark! Quite true though. The common paths I saw was adding a specialization or continuing on getting an MBA for others.
When working on any system: people who built the abstraction layer you're working with - enlightened geniuses; anyone using the abstraction layer you create - literal monkeys. It's a recursive relation that applies across full stack.
In the social sciences you have Talcott Parsons, and Niklas Luhmann, that see society as a social system. To some extend the author does explain social systems, with taking the system theory from technical systems, which is kind of the origin of those theories. It is a very useful theory, but lacking in a lot of ways. For example social systems theory doesn't account for the dynamism of society. It more or less shows a static picture of what we know. But societal structures have a history and a movement in the future. Also human agency is absent, that we are not determined by some laws like machines, but have agency. Lastly what can be critisized is the inherent functionalism. In a system everything is assigned a function, and mostly exists because of its function. This doesn't need to be true in reality.
Does anyone have good pointers on if/how systems design can be applied on a smaller scale in software engineering and API design? Granted, that would by definition exclude some very important factors on company level, but then, they are out of reach for me anyway.
EDIT: I realized that the above sounds like using SD to solve technical issues, but I meant to ask specifically about using SD to guide things like evolution and usage of a software architecture or API.
I think the core tenet would be "design every process(*) with feedback loops, so that you can detect when it needs to be updated". This can be as simple as having good tests that align with the business cases the API needs to support, to logging every data that goes through it and performing heavy data analysis on it.
(*) (Not just the processes in the program, but also processes in the company needed to find out what to do and how to do it).
This leads to organic growth, which can overcome problems caused by an upfront design that goes bad, because it couldn't have planned in advance every problem it will face in the future. Systems Design is all about adapting the system to changes in its environment. Learn about the Viable Systems Model for a mental framework on what to look for.
Funny anecdote from my time at the University of Waterloo:
The systems design engineering major was abbreviated to SYDE. It was a popular joke to ask friends in that major “so what is systems design engineering?”
They had a self deprecating chant in response: “S Y D E what is systems? Don’t ask me!”
On this second-systems effect: Okay, what if the initial system has reached a point of being unmaintainable and a new system is needed, and there just is no way to get it to where it needs to go incrementally, because the pathways just aren't there; it's a from-scratch do-over. Now what? Abandon hope? I don't get that.
I work at an organization where the second-system is being attempted and jealousy, sabotage, resistance to change and more come into play (why do they get to build it? we should build it! let's tell everybody how much it's going to suck because those guys are stupid! etc) but the danged thing is badly needed.
Every time someone has told me that "it sounds nice to do it incrementally but that just won't work for us" it's because either they (a) lack expertise they would need, or (b) haven't really tried for political reasons.
Maybe it's actually true in your specific case, and I'm curious what makes your case different, if you're at liberty to share!
Sorry for the extraordinarily late reply, but: I'm only peripherally involved in this application, but I think it's in part because of hideously bad architecture from the ground up. I know it was written in Cold Fusion, for one thing, which has graduated to the ever-popular Tremendously Expensive Licensing Era, and has major performance and database problems.
But yes, politically: There is no desire to make it a "migration" type of problem; transitioning users isn't important from the business perspective. That might be hard to believe, but I think the argument is that dragging the old product through its own muck is a lot slower than building a new product that brings in new business. So, who am I to argue with that? Are we saying, no, no, if you have an old product you must migrate even when the business says you don't need to?
The second system effect is basically death by 1,000 cuts. You avoid it by designing a new system that you need rather than designing a new system you think you might need in the future.
It becomes a problem because you think "we're doing this whole new system, lets add two dozen bells and whistles we don't have or need yet, surely it won't affect time to delivery!" Bzzzt, wrong.
That's part of the system. Management is failing if it's not acknowledging and tackling this part of the system that's prepared to destroy the whole to preserve itself.
Seems like systems design is just studying a potpourri of engineering disciplines and the job entails stringing it all together the best that you can. Is there a formal theory about systems? I don't think so, and if there is, there's definitely no way to quantify if one system is "better" than another.
Since no formal theory about the field exists.. every conjecture is just opinion. So someone working in systems design won't necessarily come up with a better "design" then some other person who doesn't specifically work in the field given that there's a lack of a formal theory to prove it either way. In software these guys who do "designs" are the system architects and I think the ridicule is justified given that anyone can do it.
Category theory is the closest thing I've seen to a fundamental theory of design and I'm positive it's not well known among "systems design" engineers. Honestly it's not even that useful either because the theory doesn't include any measurement to determine if one structure is "better" than another.
I mean more like you don't need any "design" training or specific "systems" training because no actual theory about this stuff actually exists. It's bullshit.
You did 15 years of C++ so you have domain knowledge of C++ and whatever area you work in. You don't need any extra "design" training to get good at it. So a better way to say it is this: Anyone who is knowledgeable about whatever they're working on can do "design" you don't need to hire someone who's an expert in "design."
Your title "Systems Architect" is more of a symbol of your rank and years of experience then it is your ability to design.
Even though there isn't a formal theory, there definitely is a big difference between a person capable of doing good system design and one who isn't. I don't think all people are capable of becoming a systems designer or architect. It is not always required for projects, but when it is necessary it can make a great difference, putting a project on the right track from the beginning and keeping it on track, making sure it all becomes something good in the end.
I've done it enough to know that I couldn't do it while keeping my IC-type role as well. For me, personally, I approach systems design in a sort of breadth first search for a solution to the entire problem space, whilst deep-diving into particular areas where I am less certain. For other parts my experience lets me brush over details quite quickly.
However, that is quite different from an IC-type role, where you'll typically run into a multitude of practical engineering issues that can be frustrating and take a lot of time. Tool issues, cloud issues, bug hunting, smoking out every little detail, write unit tests, review others code, etc. That would completely throw your mental cycles in the wrong bucket.
I don't think you should do system design without having plenty of experience from IC-type roles though. You have to have understood so many different aspects; capabilities of the people at hand, the organization at large, tools, frameworks, technologies, etc, and be very, very willing to communicate the picture over and over again and take in feedback as the project progresses. It's technical leadership at one of the hardest levels.
But realise at the same time that even software development itself doesn't have anything resembling a universal theory, common processes or frameworks. It's all changing, being reinvented, and rediscovered on a continuous basis. It is equally true that many things even in medicine or construction aren't based on any solid science.
>Even though there isn't a formal theory, there definitely is a big difference between a person capable of doing good system design and one who isn't.
Yeah? Prove it. You can't. That's the big problem here. The only thing you can give me at best is some vague metric on some anecdotal experience. We don't even have data on this. Replace "system designers" with ICs who have the same breadth of IC experience who produces the better design? What does "better" even mean?
My personal opinion on Systems design is that it's pretty easy if you got the "IC" part down. It is largely just arrows and boxes.
>I've done it enough to know that I couldn't do it while keeping my IC-type role as well. For me, personally, I approach systems design in a sort of breadth first search for a solution to the entire problem space, whilst deep-diving into particular areas where I am less certain. For other parts my experience lets me brush over details quite quickly.
I think of system design as something that is tediously hard. It's not something that requires a massive amount of skill. But it does take a lot of time to come up with a design. It's like building the Eiffel tower out of toothpicks. Anyone with the relevant domain knowledge (as opposed to design knowledge) can do it. So yeah it does make sense you can't be an IC at the same time.
>But realise at the same time that even software development itself doesn't have anything resembling a universal theory, common processes or frameworks. It's all changing, being reinvented, and rediscovered on a continuous basis.
This is the whole point. Anything referred to as "design" is largely an artistic endeavor. Arguably much of these frameworks have been changing in a manner that cannot be characterized as "improvement". Just endless horizontal progress and endless genetic drift because we can't know if one design is better than the other.
This is the same with system designers. The skill level is horizontally stacked because we literally can't prove shit.
>It is equally true that many things even in medicine or construction aren't based on any solid science.
Medicine is a highly quantitative endeavor. All medicines go through rigorous quantitative verification for efficacy. The same is definitely not done with system designers.
I can't argue with your points about not having evidence, but I refuse to brush it off as "merely anecdotal" as well.
> I think of system design as something that is tediously hard. It's not something that requires a massive amount of skill. But it does take a lot of time to come up with a design.
Well, zoom out a bit, and imagine that this tedious and hard work where you are documenting and explaining things with arrows and boxes actually takes up most of your time. That's the birth of the explicit role.
I am also sure you can imagine people you have worked with that you would never want to have above yourself in such a role when you are in an IC-role, because they for example always come up with crazy ideas that won't work, or don't come up with ideas at all, or just brush over everything with simple arrows and boxes and assume someone else will figure the hard details of the overall picture out, and in fact in doing so will find no use for the arrows and boxes they got handed other than being some picassoesque requirements input from which you have to make real investigations and conclusions.
That's the people that aren't mature enough for such a role and I claimed many never will be.
But yes, I resonate with a lot of what you are saying.
I still disagree. It's just not that hard. I can explain by comparing it to art.
Artistry is hard and artistry is basically the same thing as design but harder.
Let's take something like say painting. Painting requires a lot of skill. Why? Two reasons. The sheer amount of ways to compose paint into a painting is astronomical. Much much much More then the amount of atoms in the universe. The amount of these paintings that would be considered "art" is also astronomical in number but the ratio of art to all possible paintings is minuscule in number and can basically be rounded to zero. This ratio and the actual huge numbers plugged into the ratio illustrate how hard artistry is.
How about building art or designs with legos? Suddenly it's easier. Why? Because the number of ways legos can be composed in a limited space is much smaller then the ways you can compose paint on a canvas. This is because lego Blocks have specific rules. Two lego blocks have a countable number of ways they can be composed, two brush strokes can be positioned with enough variation that composition is more or less infinite. This is why designing something in legos is EASIER than painting. In fact this is the part where the word "artistry" starts to transition to "design". In legos it can be said you are "designing" a structure. Basically when the skill involved with the creative endeavor becomes significantly easier we tend to transition from artistry to the word "design". It's not a hard rule but definitely a generality that exists.
I look at these constructs made by a "lego artist" and I know I can ALSO make those big constructs if I had the will and time to do it. It's a feat of enduring tedium and not much skill. But the mona lisa in oil paints? you need skill to render that.
Do you see where I'm going here? What's system design if not putting together lego like primitives? It's trivial. The only thing you need here is knowledge about how the primitives work and how they compose. That's really the only challenge.
Heck you can even write a program that has all possible "boxes" as primitives with the right composition rules and just brute force evolve a design through random compositions. Each design has what? at most 40 primitives in a typical design? Maybe 200 total primitives? Let's make it 10,000 total system design primitives just to be excessively generous. Even at that number, system design is easy enough that it's a candidate for genetic programming.
Do you think such a thing can be accomplished with pixels? Randomly generate a square of 1000x1000 color pixels until you get something that looks legit? Running it at 5000 generations per second you probably won't get anything legit before the sun goes super nova. You'll need to switch out of random walk to machine learning to get anything artistic.
Usually formal theories need a logical mathematical language to fully characterize it. I scanned the wikipedia of cybernetics and it's just not it.
Control theory or signals is the closest formal theory to cybernetics. The problem with this theory is that it only characterizes very specific systems that have ordinal values as inputs and outputs. What if the input is text? What if it's tuples of tree like tokens?
The closest theory that encompasses the intuition of what we think of when we use the word "systems" is category theory in my experience. And this theory is in fact sort of too general to be very useful.
> you consider scrolling through wikipedia a review of the formality of the field? I think you may need to consider how formal you are :)
Yeah I did. Not one math equation and two informal diagrams describing the same thing.
Most formal theories outside of programming languages use western mathematical syntax for the foundational dialect and I don't see anything related to that here. Cybernetics similar in a sense to "system design" in that it's not formal imo.
>And no, category theory is not even approaching explaining this. At all.
It does. It unifies all of mathematics at the lowest level and operate at all levels of abstraction and even recursively. Category theory can describe category theory. The problem is we have no definition in this theory about what is a better design? It's just a description of designs.
Cybernetics is fairly formalized, but a lot of research into it was dropped. Cybernetics is based on information theory and approaches systems in a probabilistic way which is in direct contrast to the AI approach of the 50s and 60s which were really about processing discrete pieces of data.
If you read cybernetics texts today, they paint a picture very similar to neural networks and the statistical methods for processing data. "Introduction to Cybernetics" by W. Ross Ashby is a decent text with a lot of good exercises. I still think there's a lot to learn here.
>It does. It unifies all of mathematics at the lowest level
This was attempted before by Russel & Whitehead before Godel proved that any sufficiently complicated system can produce non-truths and false statements that cannot be verified by that same system.
>This was attempted before by Russel & Whitehead before Godel proved that any sufficiently complicated system can produce non-truths and false statements that cannot be verified by that same system.
I never made the claim the system was complete.
>If you read cybernetics texts today, they paint a picture very similar to neural networks and the statistical methods for processing data. "Introduction to Cybernetics" by W. Ross Ashby is a decent text with a lot of good exercises. I still think there's a lot to learn here.
well unfortunately I don't see anything formal in the Wikipedia. I'll take a deeper look at that text though.
“Systems thinking” is why Seattle turned into a mess:
People patted themselves on the back for their “deep understanding”, failed to include all the voices, didn’t actually understand the complex dynamics, and drove straight into a wall.
Empirically, they’re not good at what they do — if a government paying for their advice ends up with a disaster.
We don’t call civil engineers who collapse skyscrapers successful.
I find systems design and complex adaptive systems fascinating. I really recommend the book, "Hidden Order: How Adaptation Builds Complexity" by John H Holland. It is technical but aimed at the educated, non-technical reader, with enough description that you could code along to his models. I really wish he had written more books. RIP.
Another good source of systems thinking are the books by Jane Jacobs on urban planning.
I think anyone wanting to study this area rigorously would need to spend time doing detailed discrete event simulations, understand probability theory, time series analysis, phase space analysis, experimental design, graph theory and queueing theory.
Like databases or operating systems, complex systems embody many fields of study simultaneously. One of the funnest parts is that they can be modeled at so many different levels.
Complex Adaptive Systems https://en.wikipedia.org/wiki/John_Henry_Holland
Urban Design https://en.wikipedia.org/wiki/Jane_Jacobs
Ants https://theconversation.com/e-o-wilsons-lifelong-passion-for...