While we run through the normal portions of our update, I really want to focus on something tangential, but related tonight. Specifically, the effect of the Starlink Snowflake destroying Twitter on the Ukrainian’s efforts to defend themselves against Putin’s re-invasion, others attempts to free themselves from tyranny and oppression, and just everyday people trying to survive natural disasters. We’ll get to most of that after the jump, but first let’s start with a quick update from Iran:
Astonishing scenes from #Iran. Protesters have burned down the house of Ruhollah Khomeini, the Islamic Republic’s founder.
The house has been a museum for the past 30 years. This is an attack in the essence of the republic itself. pic.twitter.com/Qtk5jr5AR6
— David Patrikarakos (@dpatrikarakos) November 18, 2022
Well now it’s a party!
The BBC has the details:
Protesters in Iran have set fire to the ancestral home of the founder of the Islamic Republic, Ayatollah Ruhollah Khomeini.
Images posted on social media show part of the structure in the city of Khomein being set ablaze.
News agencies have verified the videos’ location, but regional authorities denied there had been an arson attack.
Ayatollah Khomeini is said to have been born in the house, which is now a museum that commemorates his life.
The fire at his ancestral house is one of the latest incidents in a wave of nationwide demonstrations directed at his successor, Ayatollah Ali Khamenei, and his government.
The protests against Iran’s clerical establishment erupted two months ago after the death in custody of Mahsa Amini, a 22-year-old woman who was detained by morality police for allegedly breaking the strict hijab rules.
Five members of the security forces were killed in the latest unrest on Thursday, according to Iranian state media.
Meanwhile, funerals for young Iranians said to have been killed by security forces sparked fresh demonstrations on Friday.
Crowds chanting “death to Ali Khamenei” gathered in the south-western city of Izeh for the funeral of a nine-year-old boy, Kian Pirfalak, who was shot dead by security forces, according to his family, although officials have denied this.
More at the link!
Here is President Zelenskyy’s address from earlier today. Video below, English transcript after the jump:
Good health to you, fellow Ukrainians!
All day long, the energy staff worked to restore the normal technical possibility of electricity supply, as a result today there are already significantly fewer emergency shutdowns.
As of the evening, in most regions where shutdowns continue, stabilization hourly schedules were in effect. Emergency shutdowns were used to a greater extent in the Odesa region and Kyiv.
The difficult situation with energy supply persists in a total of 17 regions and in the capital. Kyiv region and Kyiv, it’s very difficult in the Odesa region, and also the Vinnytsia region and Ternopil region.
Special “Points of Invincibility” were opened in Kherson today. The first two points. There will be more.
While electricity is restored in the city, people can charge their phones, stay warm, drink tea and get help. We have provided communication service there, there are “Starlinks”, etc.
We know that it is very difficult for people, because the occupiers destroyed everything before fleeing. But we will connect everything, restore everything.
Vice-President of the European Commission Dombrovskis was in Kyiv today. We talked with him, in particular, about energy. Financial cooperation between Ukraine and the European Union, the financing needs of our country until the end of the year and in the next year were also discussed in detail.
This year, we expect two tranches of macro-financial aid – EUR 2.5 billion in November and EUR 0.5 billion in early December.
We also discussed new sanctions against Russia, which may be introduced at the European level.
I spoke today with the student community of Ireland – a country that supports us fundamentally. Although it is a neutral country militarily, Ireland strongly defends European values and helps us politically and humanitarianly.
I made two requests to the Irish. The first is to promote at the pan-European level the need to strengthen sanctions against Russia. The second is to help us spread the truth about the events in Ukraine and about Russian terror.
I also addressed the participants of the security forum in Halifax, Canada, which is one of the world’s security forums. Once again, I presented the Ukrainian peace formula and invited Canada and our other partners to choose for themselves a component of the peace formula in which they can show their leadership as much as possible.
This is the point of the way we present our proposals for restoring peace.
We have broken them down into elements – what Russian aggression consists of. And we do everything to neutralize each of these elements and to neutralize the aggressor because of it.
Every day, I receive very positive feedback from the world about this approach of ours – major leaders support Ukrainian constructiveness. And I believe that we will be able to implement our peace formula.
The fierce fighting in the Donetsk region continues to this day, there has been no alleviation of hostilities or respite. About a hundred Russian attacks were repelled in the Donetsk region only yesterday. All our warriors who are holding out in Donbas are true heroes.
Today, I would like to pay particular attention to the border guards of the Kharkiv and Sumy detachments, who are currently operating in the Donetsk region. Their mortars help a lot, giving the invaders no rest. Nice job, guys! Thank you!
I also thank the warriors of the 4th operational brigade of the National Guard, which protects Ukraine in the direction of Bakhmut. Smart initiative and persistence are two things that produce extremely useful results.
Nowhere on the frontline do we give in to the enemy. We respond everywhere, we hold positions everywhere. We are preparing future successes in certain areas.
I spoke today with the President of Türkiye Erdoğan. Thanked him for the efforts to extend our grain export initiative.
We also discussed the details of our new humanitarian initiative Grain from Ukraine. I hope that Türkiye will join its implementation.
Of course, attention was paid to both security cooperation and energy situation.
Tomorrow we will have a very active day, not a day off at all. We are expecting news and opportunities for Ukraine. And we are already preparing the appropriate soil for them.
And today I want to say one more thing – congratulations. Congratulations to all sergeants of the Armed Forces of Ukraine on their professional day. Today, many kind words were said both in the army and in society.
Sergeants are truly the people who ensure real unity and understanding between commanders and soldiers. The sergeant corps is the true backbone of the Army, and I thank all of our sergeants who live up to those words.
Thank you to each and everyone who is currently serving in our defense forces!
Thank you to everyone who made the defense of Ukraine a part of their personal destiny!
Glory to our heroes!
Eternal memory and honor to those who gave their lives for the freedom of Ukrainians!
Glory to Ukraine!
Here is former NAVDEVGRU Squadron Leader Chuck Pfarrer’s most recent assessment of the situation in Kherson:
KHERSON AXIS/ 1430 UTC 18 NOV/ RU airstrikes hit Kherson area. UKR air defenses down 5 Shahed-136 UAVs, four RU cruise missiles and two RU guided missiles. UKR Gen’l Staff reports RU airstrike against Antonivka, a location deep in occupied S bank territory. pic.twitter.com/nSSBsfbU2r
— Chuck Pfarrer | Indications & Warnings | (@ChuckPfarrer) November 18, 2022
Last night a couple of people emailed and a couple of people texted to ask me what I thought was going on with the Starlink Snowflake and his new toy. My response was that we may be watching a serious psychological breakdown play out in real time, a very bad and extended reaction to any number or combination of drugs, or a combination of both. However, at the end of the day, it doesn’t really matter as there are only two questions to ask. The first is the age old classic of cui bono/who benefits? The second is that even if we stipulate that the Starlink Snowflake is acting independently for whatever reasons he has, given that what he’s doing is going to make it harder for the Ukrainians to communicate to the world what is going on and their take on it, as well as people protesting tyranny and oppression like the Iranians in the tweet up top.
The only people that benefit from what the Starlink Snowflake are doing are tyrants, autocrats, and authoritarians. Destroying the platform, which is for all its issues and faults, is often the first and quickest way to get the word out about breaking news events – natural or manmade – just makes it harder to get the word out about them. Here are the abstracts from two different research articles on Twitter’s use for crisis communication:
This article examines the use of social media, specifically Twitter, in crisis communications during a natural disaster and how it can provide information, guidance, reassurance and hope to victims while keeping others across the nation and the world apprised of the situation so they can provide assistance, as needed. A case study looks at how the mayor of Houston, Texas, Sylvester Turner, used Twitter during Hurricane Harvey in August and September of 2017. The case study is analyzed using restorative rhetoric theory, revealing the use of Twitter by Mayor Turner to be a strong example of successful restorative rhetoric during a natural disaster. This research affirms the findings of other researchers that the restorative rhetoric stages overlap, and that the theory may be improved with some variation based on crisis type. This research also shows that Mayor Turner’s use of Twitter exemplifies best practices for using social media in crisis communications with very few opportunities for improvement. This article offers suggestions to crisis managers on how to use Twitter to prepare for, communicate during, and go forward following a natural disaster.
Twitter, a popular communications platform, is identified as contributing to improved mortality and morbidity outcomes resulting from the 2013 Hattiesburg, Mississippi EF-4 Tornado. This study describes the methodology by which Twitter was investigated as a potential disaster risk reduction and management tool at the community level and the process by which the at-risk population was identified from the broader Twitter user population. By understanding how various factors contribute to the superspreading of messages, one can better optimize Twitter as an essential communications and risk reduction tool. This study introduces Parts II, III and IV which further define the technological and scientific knowledge base necessary for developing future competency base curriculum and content for Twitter assisted disaster management education and training at the community level.
For good, bad, or otherwise, Twitter has become the default and go to for breaking news. And not just by reporters doing their initial reporting via tweet and thread in order to tease their longer form reporting. It is used to get the word out about crises and disasters, let alone lost pets and people. We’ve seen, and covered it here several times, a spontaneous from the bottom up counter-information warfare operation develop on behalf of Ukraine. By severely hamstringing, if not destroying Twitter, the people that benefit are the same usual suspects we’d list if we were playing: “is the Starlink Snowflake being run by (insert name of authoritarian scumbag here)”.
That’s enough for tonight.
Your daily Patron:
There isn’t a new Patron tweet today, but this should do:
Good morning, friends!
📸 @ng_ukraine pic.twitter.com/DdL1VptKbf— Defense of Ukraine (@DefenceU) November 18, 2022
Here is a new video from Patron’s official TikTok:
@patron__dsns Знайомтеся, мої дорогоцінні качечки! Наче нікого не забув🤭 #песпатрон #патрондснс
The caption machine translates as:
Meet my precious ducklings! As if he didn’t forget anyone🤭 #PatrontheDog #PatronDSNS
Open thread!
Becoming a Normie + Bonus Recipe Thread
Alison Rose
I haven’t actively used Twitter in a few years, but this is the main reason I’m worried about it being destroyed. There really isn’t a substitute for the breaking news/urgent situation communications you can do there. For Northern California, for example, Twitter was the best way to get info out and find help during fires and outages and such. It’s the fastest way to spread crucial info as widely as possible. Facebook can’t do that, nor Instagram, and not really TikTok either since its algorithms are a whole big mysterious mess. Of course, part of the problem is that Muskrat doesn’t give one single solitary fuck about any other human on Earth.
I do wonder if the people who originally created Twitter could just, like…make another one. Give it a different name, switch up the look a bit, and be like, yay Twitter clone. I realize that would take months if not years, but…hope springs eternal.
Or maybe Oprah or someone could now buy Twitter from Muskrat, so long as she promised not to let Dr Phil have anything to do with it.
Stuff sucks. But still, thank you as always, Adam.
Carlo Graziani
The Khomeini arson. I’m just speechless.
Do we know whether any parts of the security/police/military are turning on the government?
dmsilev
re: Twitter, the Post had quite the headline today: “Musk summons engineers to Twitter HQ as millions await platform’s collapse”. I can’t imagine that he’s doing it deliberately; the side effects of puncturing the “Elon Musk, Supergenius” myth will hammer his wealth hard, but I can certainly imagine him being manipulated into cratering the service by let’s say the Saudi Royal Family to pick one possibility.
Fake Irishman
@Carlo Graziani: that’s The Question, right?
The RG is pretty hard core, but as a certain retired Romanian dictator found out, you’re in control right up until you aren’t.
This situation is different of course, but cascades are a thing.
counterfactual
On Musk, the Occam’s razor hypothesis is “excessively online -> radicalization -> relationship breakup -> full MAGA.” It’s the scale that’s different, instead of emptying the 401K to give to Donald Trump’s legal fund, Musk is managing to burn $44 billion.
dmsilev
Today’s long read, from the Post:
Stealthy Kherson resistance fighters undermined Russian occupying forces
Read, as they say, the whole thing.
Anonymous At Work
Antonivka is located in RU held territory. Why would RU forces bomb some place that deep inside territory they occupy?
kalakal
@Fake Irishman: After their initial crackdown failed it would seem they opted for a keep the lid on, wait it out and wear them out strategy. It takes a lot of nerve to protest against a hard line, ruthless regime, to keep it up for week after takes either immense courage and/or desperation. That hasn’t worked. I suspect that the regime didn’t use maximum force from the get go as they have in the past as they felt deep down it wouldn’t work this time. Ceausescu, Gaddafi, Mussolini etc are lessons most authoritarians are well aware of.
Anonymous At Work
@Anonymous At Work: Nevermind, found it. There’s another Antonivka just NE of Kherson city.
Anoniminous
Musk has destroyed Twitter. With the loss of the people who kept the place running – the operational and technical support staff – Twitter is not only one fault from collapse There’s not enough left for somebody else to come in and run the place. The loss of technical know-how is too broad and deep.
bbleh
@Carlo Graziani: (cc @Fake Irishman:) My very thoughts x2. And I know nothing about the relevant internal Iranian politics, eg the composition and relationships of the various security forces (broadly defined), or the nature and strength of the links between and among them and with the religious-national leadership, or even what other players there are and how they fit in. But the simple fact that things have come to this is evidence of major breakdowns, and … wow.
Adding: mindful also that Iran has a large, strong, educated middle class who are very proud of their history (spanning thousands of years of civilization, ahem), so this ain’t just some bunch of yokels doing vandalism with Home Depot tiki torches.
Another Scott
@Anonymous At Work:
(Stripping some hashtag links)
(via searching for Antonivka on Twitter)
HTH.
[eta:] Saw your update. Dunno which is which, but I do just now recall that there are several instances of quasi-duplicate place names occurring earlier.
Cheers,
Scott.
Carlo Graziani
@Fake Irishman: Yeah. Political scientists have a sort of “standard model” of measuring chances of success of a popular revolt against a government that is sufficiently ruthless and determined to suppress challenges to its authority. Basically everything hinges on a substantial fraction of the security, military, police going over to the rebellion. Failing that, it’s just a matter of having enough personnel to power-wash the blood off the streets after the inevitable massacre.
The possibly hopeful thing is that when a protest movement becomes as widespread as the current one, there are an awful lot of police officers, soldiers, intelligence officers, even RG personnel, who have family members and friends in danger, or who they might be ordered to fire upon. At a minimum, a lot of them must be wondering what they would do if they received such orders. Also, we can assume that a movement doesn’t get this large if it doesn’t represent the views of a large cross-section of the population, and that set necessarily intersects the set of government employees, including those working security, military, police, etc.
I wish we had something better than guesswork, though.
Andrya
I would appreciate comments by Adam or anyone: it seems to me that Musk is a significant national security problem for the United States. The SpaceX “Falcon Heavy” is the only launch vehicle capable of lifting the heaviest payloads. And SpaceX is privately held. If Musk is having a mental breakdown/drug breakdown/foaming at the mouth ego trip/loss of contact with reality, can SpaceX be relied on to carry out contracts with the US DOD? What if he got a major mad on towards President Biden, the US government generally, or Gen. Austin? I assume that with a publicly traded company, the board of directors would insist that contracts be honored. But a private company, especially if Musk owns a majority of the stock, what then? Of course, there would be huge financial penalties for refusing to provide a launch vehicle that SpaceX was on contract to provide, but there are huge financial losses for Musk’s Twitter tantrum, and that hasn’t stopped him.
As always, thank you Adam for doing this.
Another Scott
Meanwhile, …
Oooh…
Cheers,
Scott.
Another Scott
@Andrya: The US Government always has options, via the Defense Production Act and probably other things.
Cheers,
Scott.
bbleh
@Andrya: I suppose in extremis the gummint could simply nationalize SpaceX, perhaps (or not) with some eminent-domain-type compensation or promise thereof (assuming there is anything left of Consolidated Musk, Ltd. to compensate). But I also assume there are less blunt instruments available, both carrot and stick.
randy khan
The concerns about Twitter failing are getting enough play that this afternoon one of the shows on MLB Network Radio on SiriusXM had a discussion about it, which included a plan to create a company with one of the best baseball reporters to monetize what he now tweets as texts or emails. It was at least 50% a joke, but the extent to which the idea that Twitter is now doomed to fail has gone mainstream is quite stunning.
Martin
@Another Scott: Every country has their Springfield.
Geminid
@Carlo Graziani: I have read little about security force defections in Iran. But in the runup to and aftermath of our midterms I stopped following these events.
When I turned back to Iran News Wire’s Twitter feed this evening I was struck by an apparant increase in the use of molotov cocktails and even explosives by protesters.
Actually, “revolutionaries” might be a better word. The graffiti, “This is not a protest. This is a revolution” has been popular now for weeks.
Iran News Wire posted a short video taken at the funeral of the young boy whose death is mentioned above, with the caption:
Izeh is a city of ~125.000 people in Southwest Iran.
Anoniminous
@Martin:
Fairview (288) and Midway (256) are #1 and #2 in the US.
bbleh
@Anoniminous: but this is only because “Unimaginative” has a negative connotation, “Pedestrian” is confusing, and “Anodyne” is obscure.
Anonymous Expat
I suggest a pithier nickname for Elon; when mocking him, we should henceforth refer to him as the Marssiah.
Steeplejack
@Anonymous Expat:
Somebody sent me “Phony Stark” in an email today.
counterfactual
@Andrya: SpaceX had a press release this week that Gwynn Shotwell is in charge while Elon is preoccupied. She’s second in command, and the one SpaceX exec that Space Force and NASA have as point of contact. So far, SpaceX is proceeding with US government and commercial launches as normal.
cain
@Carlo Graziani:
There is no going back on this one .. the govt is going to be hellbent on atrocities and it’s going to create even more problems. It’s the atrocities that’s going to exacerbate the situation and create a downward spiral. This govt is going down. It’s just a matter of time now.
cain
The Twitter story shows that it is a single point of failure and literally one person can end it.
We need our own spaces and controlled by community. Perhaps we need to have it as a united nations service or something or the mastodon model. I do t know.
Enhanced Voting Techniques
@counterfactual: Yes, the dude is quite the douche the way he demands everyone works 80 hours weeks while spends all day tweeting like a teen.
Andrya
@Another Scott: @bbleh: Thanks. The considerations you mention are reassuring, especially the Defense Production Act, but Musk can afford expensive lawyers, expensive lawyers can cause delay, and national defense cannot always afford delay.
@counterfactual: Also thanks. Your input is also reassuring, but the US government hasn’t annoyed Musk yet.
Bottom line: If Falcon Heavy is the only platform that can launch the heaviest DOD satellites (something I don’t know) then I think we should develop an alternate launch vehicle.
Bill Arnold
@Anonymous Expat:
It is fun to believe that Elon’s name was inspired by Dr. Wernher von Braun’s 1948 science fiction book (published in English several years later), in which the ruler of Mars has the title “Elon”: : Project MARS(Das Marsprojekt) (Dr. Wernher von Braun, English translation by Henry 1 White, Lt. Cdr. USN)
counterfactual
@Andrya: United Launch Alliance’s Vulcan is designed to carry all National Security Space Launch cargoes, but it’s years behind schedule and won’t make a first test flight till next year. There may be one or two more Delta IV vehicles committed to Space Force heavy launches.
[edit] The Delta IV is an existing heavy lift rocket, but the production line has been shut down, and there are only one or two left to launch.
pacem appellant
Mastodon is already filling the vacuum created when Musk destroyed the Birdsite. This is an objectively better solution. It is decentralized, so it deploys and scales better. It also means that authoritarian will have a hard time blocking it or controlling it. FB and others are all privately held and centralized, and thus suffer the same bad actor risks as Birdsite.
Enhanced Voting Techniques
I noticed the SF Chronicle was encouraging all it’s Twitter followers to join Mastodon.
Edmund Dantes
@counterfactual: she is also no saint. She rose to defend Musk and went after dissenters within SpaceX when they complained about him. She also pulled the old “Musk never harassed me and I never saw him do anything sexually harassing in 20 years”
Origuy
@Another Scott: My money is on USA vs Iran on Monday morning, if Twitter lasts the weekend.
Carlo Graziani
@cain: A violent confrontation certainly seems inevitable. There’s are accounts of deliberations within Iran’s Supreme National Security Council (SNSC) reported in today’s Iran Crisis Update at ISW which, while not necessarily first-hand and completely reliable, suggest strong pressure on the government towards a comprehensive, no-holds-barred crackdown. If that happens, it will, at a minimum, be a bloodbath.
I’m not as sanguine as you about the government failing to reestablish control. Admittedly they now have protests in 31 cities (on November 18, again according to the same report) which is daunting. But they are utterly ruthless, organized, armed with weapons spanning the spectrum from low-crowd-control to battlefield, and also highly informed, and quite capable of pulling people out of their beds at 3AM instead of shooting or gassing them while being recorded on a dozen phone videos in broad daylight. And much as we’d like to think of them as stupid, they aren’t stupid.
What may have been restraining them so far — it’s a hope, anyway — is justified fear that they may lose hearts and minds of the people that they need to carry out their violent suppression. I certainly hope that’s true. I don’t feel the kind of certainty that you do,, though. Too many of these things go bad.
bbleh
@Andrya: I don’t disagree that the US needs to have reliable heavy-lift capability, and (obviously) some accommodation with an existing one (eg SpaceX) is cheaper than developing a parallel one, but should such accommodation not be possible, and an urgent national need arise, I don’t think legal maneuvering would get in the way of using available assets and sorting out the niceties later. At the end of the day, the gummint makes the laws, and it has the rifles and the people trained to use them.
Andrya
@counterfactual: Thanks, that’s extremely reassuring. If I were SecDef Austin, I would be leaning on ULA to get their act together pronto. Hopefully he is!
Shalimar
Twitter isn’t dead. The next couple months will be rough. There will be major outages. But Musk has already started hiring replacement workers and his fan cult is more than large enough to staff a company Twitter’s size 100 times over.
The bigger long-term problem is advertisers. They aren’t coming back. There is no way anyone on the alt-right like Musk can make a profit from Twitter.
wombat probabilty cloud
@pacem appellant: I’m feeling hopeful about Mastodon because of the model. If user communities support the instances/servers, it has real promise. The main obstacle for new users seems to be how to choose an instance, but it’s not difficult and one can shift later.
Big R
Adam you say there are two questions. The first one is “who benefits?” For the second you say this:
I have read this so many times and I cannot discern the question. Can you help?
Carlo Graziani
@wombat probabilty cloud: The main obstacle for new users appears to be this.
Shalimar
@wombat probabilty cloud: I’m hopeful about Mastodon too. It is a better model than Twitter for the essential news and anti-government functions that are irreplaceable. Hopefully Mastodon will expand exponentially after Twitter starts going down with regularity.
Anoniminous
@Shalimar:
Musk’s fan club doesn’t have the slightest idea how to keep Twitter’s computer infrastructure up and running. That knowledge has literally left the building.
Chetan Murthy
@dmsilev: “If HIMARS can’t reach you, a partisan will”. Oboy, those people. Those people. Such heroes.
pacem appellant
@Enhanced Voting Techniques: I recently found them independently on Mastodon. I’m now following them for my SF Bay Area news!
pacem appellant
@wombat probabilty cloud: I think there is/was too much emphasis placed on “choosing a server.” Just pick one. You can move later. Most of the established servers have had to close new registration because of the influx of Twitter migrants. So just grab any ol’ server. When things settle down and you make friends on Mastodon, move to their server.
Carlo Graziani
Apropos of my little obsession: I found this tweet in Russian showing very precise damage to a section of rail from a HIMARS strike on The Drive’s War Zone. Google Translate (my new skill :-|) tells me that the caption reads “Ilovaisk. Looks like HIMARS”. Here is Ilovaisk, about 15 miles East of Donetsk, and currently still in HIMARS range of the front line (about 55 km from Nevelske, where there was fighting today).
That’s the line from Taganrog/Rostov-on-Don. The UA is still murdering it.
Shalimar
@Anoniminous: What I have read is there is nothing special about Twitter’s platform and it wouldn’t be hard to duplicate. A lot of Musk’s fans are engineers. It seems like many of them should have the expertise to figure it out. It won’t happen without problems. But I expect it to happen.
Chetan Murthy
@Shalimar:
It’s both true and false. True, because all the pieces are well-understood, and there are existing examples of those pieces put-together. After all, what made Twitter “sticky” was the user base, not the code. False, b/c all technology at this level is …. hard-to-replicate and only a small number of engineers actually know how to do it.
I can give an example: look up the Google system “chubby”. Then go look at its clones: “Zookeeper”, “etcd”, “consul”. Probably others. If you look carefully, you’ll figure out that chubby has capabilities that the clones do not. In fact, they aren’t really clones. What they are is poor carbon-copies. And those missing capabilities are actually important, for some of the systems Google builds. I could go on: there are deep reasons why Bigtable is built on GFS, that most people haven’t understood.
These facts are understood by some subset of engineers, but obviously not by a lot.
There’s a general principle at work here: when a scientific breakthrough occurs, it takes time before high-fidelity replication occurs. And for many of the distributed-systems technologies that have emerged in the past 25 years, that replication really hasn’t happened.
Anoniminous
@Shalimar:
Speaking as someone with 50+ years experience in ICT I suggest finding writers who don’t have their heads up their asses.
Here is a very brief 20,000 foot overview of Twitter’s infrastructure in 2017.
Carlo Graziani
@Chetan Murthy: As I understand matters, the really difficult thing to reproduce about Twitter is not so much the software engineering at all. It’s the accumulated knowledge about how to conduct content moderation relatively non-stupidly. Which is a social/legal/economic/political/cultural system problem, to be embedded as efficiently as possible in a software engineered system.
The teams that eventually learned to do this in such a way as not to terrify advertisers have been destroyed, the personnel retired or fired, and their bespoke knowledge is gone. Musk can spend all of his remaining pile of cash on hiring the best programmers in the world, but he can’t fix what he broke. He doesn’t even understand what he broke, and wouldn’t agree with the diagnosis even if he could shut up long enough to listen to someone explain his fuckup to him. He came to get rid of content moderation, and he certainly doesn’t want to hear that it was the lifeblood of the business.
The advertisers are definitely never coming back, except maybe the medical scammers who advertise on Fox and conservative radio, and the penis enlargers from Pornhub. Twitter, as it has been, is probably defunct.
Chetan Murthy
@Carlo Graziani: That content-moderation is almost certainly nearly-all done by machine learning models — which need to be trained, but also need to be *configured* correctly. That knowledge will go up in smoke with Twitter. But that knowledge is also software in a very real sense.
Even the hardware is special: if you read the link @Anoniminous: posted, it contains descriptions of the network infrastructure Twitter uses, and I can assure you that it is *not* run-of-the-mill stuff.
Again, I’m not saying that the stuff is impossible to replicate: I suspect that Google could take Google+ and restart it, and that would pretty much be good enough. What I’m saying is that it’s not straightforward.
And btw, I will stand pat on the position that Mastodon is not going to be able to support Twitter’s users. Mastodon is intrinsically designed wrong for such scale. Intrinsically. You don’t use “federation” for such scale.
This was my business until I stopped working “in the biz” in 2015, so I know this stuff from the inside.
Carlo Graziani
@Chetan Murthy: I…think we agree?
Certainly the CM ops are carried out by ML, and trained using supervised learning. But as I understand matters, the decisions of what belonged in those training sets — the “configuration” as you call it — was being made by teams of intellectual property and liability lawyers, sociologists, cultural anthropologists, economists, linguists, political scientists, etc., cooperating with software engineers over the course of years, putting out a succession of small and large fires, to arrive iteratively at successive versions of the CM machine, each one of which sucked a little less than the previous one, in the sense that it pissed off fewer people, or at least less important, less visible people, and hence became more attractive to controversy-hating advertisers.
That stuff is not static. It was like a living organism that needs to keep growing and evolving, or it withers. And the people who knew all about its care and feeding are gone. So it’s as if Musk needs a cat, but all he has is a snapshot — a picture of a cat.
Chetan Murthy
@Carlo Graziani: OK, sure, I can agree with that. Thing is, this describes *all* software. There’s a famous example at a big-ass bank, of its core “book” software, that had been written back in the 70s, lost the source in the 80s/90s, eventually the programmers retired and died, and they’d been updating it by patching the binary since the 90s. Lots of cases of what we call “bit rot”, software that nobody understands. But if you have it, and can start it up, it works and solves today’s problem.
In that sense, Twitter’s software, if they could just keep it running, would keep working. And sure, it would slowly become not-fit-for-purpose, but only slowly.
What I’m saying is: “operations” is different from “development”. Both are high-skill specialities. And Lone Skum has pissed-off and driven-away both kinds of employees. But if he could at least have kept his ops folks, he’d be in a *lot* less shit. A lot less shit. B/c the software could be made to work. That’s what SREs (“site reliability engineers”) are for.
HeartlandLiberal
I liked Starlink Snowflake so much I posted a link to this post on Twitter, and created the hashtag #StarlinkSnowflake, with a couple of salient quotes from the post, and ATed @elonmusk. Good job, as always, Adam.
Adam L Silverman
@HeartlandLiberal: Great, I can now look forward to him suing me …
wombat probabilty cloud
@Carlo Graziani: Ha! Just seeing this now (after the luxury of sleep). At least some are using “posts” instead of “toots” now. Hopefully the terminology will morph and/or become more friendly. Meanwhile the tutorials (e.g., from Wired) aren’t bad.
wombat probabilty cloud
@Chetan Murthy: If you’re still following this thread, can you give me a better sense of why a federation won’t scale appropriately? Thx.
Geminid
I think this report in the Times of Israel is based on Agence France-Presse reporting:
Agence France-Presse has been reporting on the events on Iran fairly frequently.
Carlo Graziani
@Chetan Murthy: Certainly ops are different than development. I guess I’m trying to point at a different blind spot of Musk’s: he thinks of the entire problem as solvable by hiring the appropriate technical skillsets. This assumption is simply incorrect, because it misses a crucial ingredient of Twitter’s CM: the deep-bench legal/social studies team that was collaborating with the software engineers to build that system. In a sense, that team was the key component of the CM system. It kept it alive and relevant. Without them, CM loses value as quickly as Twitter users figure out new ways to offend each other in large numbers, which seems to occur at least once or twice per year. In other words, that team was what distinguished the cat from the picture of the cat.
And that’s what Musk will never see. He can never place value on such people. They were the first Twitter employees that he fired as “bloat” — the smuggest of tech-bro terms. Getting him to see why they mattered would be like getting an earthworm to appreciate a pedometer. His constructed self-image of “engineer-physicist” (yes, I know) rules out a spot in his worldview for such people doing anything of value other than, say, menial clerical tasks.
That’s why I believe that real CM is never coming back to Twitter, even if he should succeed in hiring every software engineer and operations person on the West Coast. Hence, real advertising revenue is never coming back, even to the pathetic extent that it ever did. Hence, requiescat in pacem, Twitter.
Eolirin
@Carlo Graziani: He’s wrong that you can just hire enough technical skill to resolve any problem in a codebase much smaller than Twitter if there isn’t significant knowledge transfer from the people who understand it. Twitter just lost everyone who knows how anything works. There will be no knowledge transfer.
It may be more time and cost effective to have a large number of competent engineers start from scratch. And you can’t do that in a time frame to prevent catastrophic collapse of the existing Twitter infrastructure. (Also, given his terms to existing staff good luck getting a large number of competent engineers)
That’s the level of fucking up that just happened. It’s not just the CM.
The expertise of the existing engineers was a vital “soft” skill that is directly analogous to what you’re talking about in outside domain knowledge.
ETA: Given the loss of institutional knowledge for core operating necessity, worrying about the long term consequences of not paying enough attention to what non technical skills are needed for good content moderation is a bit like being concerned about the plumbing in a building that is currently on fire. After the owner ripped out all the fire alarms and had the local fire deparment defunded.
It isn’t going to matter.
Carlo Graziani
@Geminid: I saw that there’s a shitstorm warning for the Iraqi Kurdish provinces, with Iranian threats that may be precursors of military intervention. This seems related.
Pete Mack
Chuck Pfarrer gives good map, but I gave up on him when he reported half a dozen SEAD (anti-radar) attacks in a single day, but the Ukrainian DoD equivalent reported a single radar destroyed. I knew he’d led me astray before like when he claimed the Admiral Makarov was likely hit by a Neptune missile. But that’s an honest mistake. The SEAD report was really not. He’s sexing up his maps for popular consumption.
Chetan Murthy
@Carlo Graziani: OK, now I see how we agree. I agree with you, that medium/long-term, he will not be able to maintain and run Twitter, b/c that involves forms of knowledge he’s hurled away with great force (e.g. about CM, advertiser relationships, etc). And also about the code[1]. But I was speaking specifically to the short-term: where all of that can be assumed as fixed. In that context, short-term problems can degrade the site, but if you have sufficiently experienced and skilled SREs, they can keep the systems running. No, they can’t improve the ML models to reject new kinds of spam, but even if those things started occurring, it would start as some small percentage of the data-flow, and hence, again, wouldn’t bring down the system.
When you remove the operators (or replace them with new hires who don’t actually know the history and foibles of Twitter’s systems) you remove that ability to keep the thing going.
And all of that is a short-term statement. Medium/long-term, I agree with you 100%.
[1] and now an example. IBM AIX 3.2.5 had something called “HFT” (“high function terminal”). A graphics display with the equivalent of multiple screens, like you can get on Linux with ALT-{F1,F2,F3}, etc. Pretty spiffy. In AIX 4.1, it was replaced by the “LFT” (“low function terminal”) which … lacked that capability. Both versions of AIX ran on the same hardware. Why the change? B/c the guy who’d coded the HFT had left IBM (:rofl:). This happens more often than we think to commercial software.
Chetan Murthy
@wombat probabilty cloud: First, a link: https://blog.twitter.com/engineering/en_us/topics/infrastructure/2017/the-infrastructure-behind-twitter-scale
That explains the systems Twitter runs. It’s out-of-date, but if anything, they’re probably more esoteric today. B/c the biggest scaled apps run specialized networks and hardware. Why? Well, this comes back to the way that they’re written. Two things you MUST have: almost the entire working-set (data used during serving traffic) must be in-memory, and because this means massive traffic *between* servers in the application, you *must* use a full-bisectional-bandwidth network.
Typically a “federated” system like Mastodon has servers running all over the place, connected by the Internet. The Internet is not a FBB network — not by any stretch of the imagination. At Twitter alone, the network became the bottleneck at least twice in my memory — that’s why they use specialized networks now. This is why social network apps run multiple copies of their app, each in a single large datacenter.
There’s a general term for this: “the datacenter as a computer” — and Google pioneered it. Basically, all the machines in a single datacenter (>100k machines) are all run as one big computer, in the sense that it is assumed that they’ll all stay up and that a program running on machine #A can access memory on machine #B much, much, much faster than it can get data off SSD/HDD.
All of that means that as Mastodon scales up in traffic and # of users and nodes, it will start to slow down.
Of course, there’s the other issue, which is that a system designed for federated use will inevitably have friction between its components, which (by design) don’t fully-trust each other. This also will make things slower and hurt scaling.
Hope this helps.
Chetan Murthy
@Chetan Murthy: I felt I should add that all the big players with large-scale apps and clouds have figured this stuff out. You can look at the Google “jupiter network” paper, the MSFT “VL2” paper, and probably others, for good examples of this learning. It’s completely revolutionized the design and implementation of networks for datacenters.
I haven’t addressed the software aspects of a federated design, but those matter, too. Indeed, that’s much more my area of expertise. But the hardware/networking aspects are so ….. stunning that it isn’t necessary to go into software issues.
Geminid
@Carlo Graziani: The killings of the security officials reported above happened in the western, Kurdish areas of Iran. The Kurds have been long been particular targets of repression by the Islamic Republic. Mahsa Amini, the 23 year woman whose murder ignited the protests, was Kurdish, and demonstrations in the Kurdish area and regime repression have both been intense.
Iran’s threat to invade the border areas is a threat to Kurdish groups there, but Iran has been already been attacking them with drones and missiles on and off for almost two months now. Iraq’s Kurdish Regional Government keeps protesting the attacks, saying that they are being diligent in keeping the Iranianian resistance groups (that are being given refuge) from sending weapons and fighters into Iran.
They probably are being diligent.. The Kurdish Regional Government has to be pragmatic about the plight of their brethren in Iran, and those in Turkey and Syria as well. Since the 1st Gulf War the Iraqi Kurds have enjoyed an unprecedented autonomy. They could lose that if they alienate their neighbors. If Turkey and Iran were to gang up with Iraq’s central government to suppress the three Iraqi Kurdish provinces, the Kurds could lose that war.
Right now the Kurdish Reginal Government has good relations with Turkiye, their most powerful neighbor. There is a lot of trade, and Turkish engineering and construction companies play a substantial role in the Kurdish area’s development. Any active support of armed Turkish/Kurdish resistance groups by the Regional Government would turn Turkiye into an enemy, though.
The Kurdish Regional Government’s relations with Iran are more complex. Iran has influence and a certain amount of power in Iraq. They can complicate the relations between the the Kurdish region and Iraq’s central government, which are already conflicted.
The Iranian threats may be bluster. They can’t fight guerillas by land operations much better than they already are with drones. And the Regional Government might resist an Iranian incursion, not by fighting conventional battles but through asymmetric warfare. The Kurds have as much or more experience as anybody when it comes to asymmetric warfare.
U.S. was involved in at least one instance in the Iran/Kurd fighting. In late September the Iranians launched a number of drones at targets in the Iraqi border region. One drone kept going and flew much of the way to Irbil, the capital of the Kurdish region. Then a U.S. Air Force F-15 shot it down. There is a sizable contingent of U.S. military advisors stationed near Irbil.
I’m not certain, but I think Irbil used to be called Arbela, back in the day.
Traveller
With all this Twitter Delenta est talk and Mr Musk’s prodigious efforts at self destruction, I think it necessary to note that many of his best engineers are simply hostages to Twitter and no matter how abused and unhappy they may be, they are very often H1-B visa holders and cannot leave, cannot protest, cannot do much that would put their entire lives, and that of their families, at risk here in the Untied States.
Mr Musk and Twitter has a captive core group of competency that will, of necessity, struggle mightily to keep the lights and the platform running. This is just something to be remembered…
Chetan Murthy
@Traveller: Are H1-Bs still pretty non-portable? I know at one time they were absolutely non-portable, but I thought these days, you could find a new job, get that new employer to agree to sponsor you too, and it was no big deal to switch jobs ?
Obvs. non-portability is a big problem: both because it means that the worker is stuck and has to take whatever the employer dishes out, but also b/c it allows the employer to have access to a source of lower-cost workers. By making it portable, it puts American and immigrant workers on a more even keel. Which is better for both.
Traveller
I think you are correct that H1-B’s are portable…theoretically. My current understand is that, Yes, you can leave your sponsoring company…but there is only a three (3) week window to have a new sponsor company*
This is a real risk that there is some kind of slip between the lip and the cup and….oops. your life is truly upended. There is therefore a compelling reason to stay in place…I think, but it is not my life so I don’t really know.
*(the above could be wrong and I invite being educated further on this)
davecb
I used to work with an former member of the Revolutionary Guard. I interpret him as saying neither he nor many of his his colleagues were nuts. It was their leadership who were insane. He voted with his feet, and now works on this side of the Atlantic.
Were I Ayatollah Ali Khamenei, I would worry about using the guard and army against anyone but the Kurds. Were I to use them against their own cousins and friends, that would give any sane guard or army recruit a very good reason to turn on me.
Matt McIrvin
Does Twitter fight more authoritarianism than it creates? It and Facebook seemed to be working to convert the United States into a dictatorship for a while there. And apparently Trump’s back on.
It seems to work both ways.