The paper and the system that caused this incident are different. Google has a ton of different automated systems for maintaining production.
The paper you linked is for a system called autopilot, which can scale a specific jobs memory and CPU usage up or down depending on historical load of that job. Think of it as having different instance sizes on GCP or AWS, then you have a tool that monitors how much actual CPU and memory your job uses on the instance, in will upsize or downsize your machine depending on load over time.
The system that broke based on the incident report above had to do with quota that a given production system could use as a maximum. Similar resource management automation, but very different systems.
So one is about fine-tuning, while the other is about preventing excessive resource usage.
Great, I feel like an idiot. I advocated slack, because it granted the freedom to choose a client. Now I helped lock-in others.
Suggestions for alternatives, that I could migrate to? The requirements are: mid sized teams, desktop and mobile, all major os. People used web based client, native clients, irc gw, and bots. We need search and archive. Self-hosting is an option.
Hey, I just wanted to say that it is brave of you to admit to making this mistake. I would have been one of those protesting your advocating, imploring, perhaps loudly, to consider the consequences :)
Anyway, yeah, a lot of us are now looking for practical solutions to this very unfortunate and serious problem.
Prosody is an XMPP server Kaiwa is a web frontend that works really well with it. For a quick "try it out" Kaiwa makes a docker image that bundles prosody.
There are lots of bot options. Conversations is the android client to use. There are no great iOS clients though.
Out of curiosity, have you looked at Mattermost? How does it compare? About three years ago when open source Slack alternatives were starting to hit, Mattermost and Rocket.Chat were the top options. GitLab even ended up bundling Mattermost with their Omnibus installer (and were going to bundle Rocket.Chat as well, had they made it possible to support Postgres, but I don't think that's the plan any longer).
To be frank, you should feel like an idiot. This was entirely foreseeable and people have been warning you not only about Slack, but about proprietary software and vendor lock-in for _literally decades_. Try not to make this mistake again. Advocate for free and open source.
Every single one of these "scoops" carries the narrative "the authorities / the police / people who don't share our opinion collaborate with or are nazis"; laying the mental foundation for further G20 type riots and their self-justification. This narrative is getting quite old for most people these days, even though most support non-violent anti-neonazi and anti-extremist organizations.
Outlets like indymedia and the notorious local establishments appear to be more interested in searching in their local neighbourhood politics for justification to riot than in novel kinds of extremist or outright fascist movements developing on a global scale. This makes it a bit difficult to believe their self-proclaimed goals.
any evidence for that first charge there? There have been some well documented collusion between elements of the police and far right wing types in Germany. AfD is also winning local elections there.
To provide a more balanced picture, there's a competition held at that site:
Riot-Bundesliga [1]. Who fucks up cities the most? Throw paint at buildings! Set cars on fire! Destroy property of political enemies! And banks! And stores! Beat up (probably) right-wing people! And cops! Go win the crown!
The problem is that it's the hot phase right before the election, and Interior Minister de Maiziere, who himself comes from a family with close Nazi ties (https://twitter.com/wasloshamburg/status/901010204598222848) wants to prove, especially after G20, that he is the personification of "law and order".
What I am really afraid of is Joachim Herrmann (CSU, Interior Minister of Bavaria) succeeding that moron after the election. Herrmann recently introduced the possibility of "indefinite jailing" ("Endloshaft") for people that the police deems as "dangerous".
The German Conservatives always have been rightwing-authoritarian, but the pressure from AfD neonazis has shifted their entire politics even more towards right-authoritarian than I would ever have thought possible.
To make matters worse, the SPD has not been able to produce a qualified candidate who actually wants to take back stuff like Hartz IV which the working class really hates, the Green Party got their policies stolen / made irrelevant (environmental protection, exit from nuclear energy), and the Left Party (as well as the non-parliamentary lefties) are rather battling themselves than the Conservatives and the Nazis.
In theory there are alternatives where you do not need to trust the server. The keyword is proxy re-encryption (e.g. [SELS]).
On the other hand schleuder exists today and can be used by normal people (for example without having a custom private key per list, as you need afaik for [SELS]).
The other often overlooked fact about schleuder is that it's also a re-mailer. You can use it in a one-to-many setting, where the external party does only know the cryptographic key and email address of the list and a group of people can communicate with this person using schleuder as the proxy. This is a great alternative to shared mailboxes and a feature that is probably impossible with proxy encryption.
I fully agree with your analysis. But cryptocurrencies are just the latest tool that will serve well the interests of the demagogues. Technocracy will not improve affairs that we don't resolve as a society in the first place. More powerful tools means just more power to those who can afford the longest lever. Any other interpretation is just a blatant neglect on the history of technology.
Cryptocurrencies, growing pains aside, will arrest the trend toward economic centralization. They are not just more powerful tools. They are tools that shift the control over money from large trusted third parties to individuals. They literally allow an individual to be their own bank.
They are a counter-force against the trend toward eliminating cash, and with it, financial privacy [1]:
>Any future cashless bank-payments society will be the outcome of a deliberate war on cash waged by an alliance of three elite groups with deep interests in seeing it emerge.
These statements from an executive at MasterCard are a great window into the company's designs on cash and its electronic corollaries:
>There's huge interest in cryptocurrencies and what perhaps they can create in the market place. Now we at MasterCard are not completely comfortable with the idea of cryptocurrencies largely because they go against the whole principle that we've established our business on which is really moving to a world beyond cash and ensuring greater transparency.. If you think about it, cash is a problem for a number of countries. Cash really facilitates anonymity, it facilitates illegal activity, it facilitates tax avoidance and a range of other things that aren't going to drive efficiency in an economy
> They are tools that shift the control over money from large trusted third parties to individuals. They literally allow an individual to be their own bank.
You're not thinking big enough. All I need to be a bank is some money to lend, or safe to put deposits in.
Cryptocurrencies let people be their own nation state.
Saying that bitcoin allows users to act as their own banks is like saying a mattress with your cash under it is your own bank. It's not meaningful in anyway. Debt is the meaningful differentiator.
No, ability to issue a controlled lot of physical currency is the meaningful differentiator, which is why the relation to banks is irrelevant.
I can buy $1 worth of Bitcoin, declare it ErikCoin, acceptable only in my stores at a 1/1,000,000 denomination, and I instantly "minted" $1,000,000 in cryptocurrency, using only the Bitcoin blockchain.
Bitcoin doesn't let people be banks, it let's people be the mint.
I believe from looking at the fix [0] I was able to trace back the origin of the bug. This is my (unverified) theory. Can anybody familiar with serpent confirm?
There is a catch-all [1] function in the public API (why???) of the wallet contract which uses delegatecall to delegate to the library class.
"In a similar way, the function delegatecall can be used: the difference is that only the code of the given address is used, all other aspects (storage, balance, ...) are taken from the current contract." [2] (again, WHY???)
So calling through this catch-all function the "internal" modifier on "initMultiowned" does apparently not prevent it from being called, since the delegation happens from a function inside Wallet.
So the "attack" is to just tell the wallet to reset its owners to myself. This would be so embarrassingly trivial, that it's more like picking the money up from the floor, than a "heist".
This wallet contract is insane and the programming language too. Why would a language for such a critical application have such super unsafe constructs? This can't be true. Please, serpent community, talk to your local PL people!
Talk: https://eventsonair.withgoogle.com/events/autopilot-research...
Paper: https://research.google/pubs/pub49174/