Come for the politics, stay for the snark.

Optimism opens the door to great things.

An almost top 10,000 blog!

Authoritarian republicans are opposed to freedom for the rest of us.

They don’t have outfits that big. nor codpieces that small.

Good lord, these people are nuts.

He wakes up lying, and he lies all day.

At some point, the ability to learn is a factor of character, not IQ.

Radicalized white males who support Trump are pitching a tent in the abyss.

Never entrust democracy to any process that requires Republicans to act in good faith.

Jack Smith: “Why did you start campaigning in the middle of my investigation?!”

Let’s delete this post and never speak of this again.

The only way through is to slog through the muck one step at at time.

Republicans got rid of McCarthy. Democrats chose not to save him.

Too little, too late, ftfnyt. fuck all the way off.

This really is a full service blog.

Disagreements are healthy; personal attacks are not.

There are consequences to being an arrogant, sullen prick.

… riddled with inexplicable and elementary errors of law and fact

Some judge needs to shut this circus down soon.

You don’t get to peddle hatred on saturday and offer condolences on sunday.

You passed on an opportunity to be offended? What are you even doing here?

SCOTUS: It’s not “bribery” unless it comes from the Bribery region of France. Otherwise, it’s merely “sparkling malfeasance”.

We are learning that “working class” means “white” for way too many people.

A sufficient plurality of insane, greedy people can tank any democratic system ever devised, apparently.

You are here: Home / Politics / Information Warfare / Monday Evening Open Thread: Facebook, Going Down

Monday Evening Open Thread: Facebook, Going Down

by | 89 Comments

This post is in: , , ,

The Verge says it’s back online now, so don’t panic…

Just as Facebook’s Antigone Davis was live on CNBC defending the company over a whistleblower’s accusations and its handling of research data suggesting Instagram is harmful to teens, its entire network of services suddenly went offline.

The outage started just before noon ET and took nearly six hours before it was resolved. This is the worst outage for Facebook since a 2019 incident took its site offline for more than 24 hours, as the downtime hit hardest on the small businesses and creators who rely on these services for their income.

After failing all tests for most of the day, a test of ISP DNS servers via DNSchecker.org showed most of them successfully finding a route to Facebook.com at 5:30PM ET. A few minutes later, we were able to start using Facebook and Instagram normally, however, it may take time for the DNS fixes to reach everyone…

Inside Facebook, the outage has broken nearly all of the internal systems employees use to communicate and work. Several employees told The Verge they resorted to talking through their work-provided Outlook email accounts, though employees can’t receive emails from external addresses. Employees who were logged into work tools such as Google Docs and Zoom before the outage can still use those, but any employee who needs to login with their work email was blocked…

And here *I* thought the Pandora Papers dump would cause more disruption sooner than last night’s 60 Minutes expose…

Reader Interactions

89Comments

  2. 2.

    Baud

    I felt a great disturbance in the Net, as if millions of voices suddenly cried out in terror and were suddenly silenced.

  3. 3.

    Gin & Tonic

    Pretty much everyone who can understand the technical explanation of what happened has a horror story in their past of an “oh, shit” moment similar to this. It’s just that they probably didn’t fuck up a company with 60,000 employees and a billion users.

  6. 6.

    jl

    Didn’t watch 60 minutes report, but read summary of her charges. Facebook’s response was such a hash of non-denial denials, misdirection and strawmen, that it greatly increased my confidence in Haugen. Facebook is still trying to conflate the freedom of quirky old Aunt Bea and Uncle Fred to spout whatever as individual users, with Facebook’s freedom to drive traffic with their algorithms to segment users by various demographics and maximize ad revenue. Facebook using secret sauce AI to manipulate its membership doesn’t even seem to rise to a corporation’s right to commercial free speech to me, but any BJ lawyers around, let me know if I’m wrong there.

  7. 7.

    Benw

    When my connection to my work servers go down I’m like dude take the day off and go to the beach

  8. 8.

    OzarkHillbilly

    Is Facebook down right now? This global outage is tough on everyone

    Nice to know I’m nobody.

  10. 10.

    HumboldtBlue

    I haven’t logged into Facebook since 2015 when family members traveled to Ireland and they wanted to taunt me with pics.

  11. 11.

    jl

    Coincidentally, I found this story at Barry Ritholtz’ The Big Picture blog this morning. I’d be interested to hear what people who are interested think of it. Seems to connect Facebook’s ongoing carnival of crime, swindles, dishonesty, and malfeasance to inability to maintain market share, and maybe more importantly, maintain engagement, page views and time, and clicks among the old fogeys who remain. But, its business model requires an ever increasing malignant rate of growth, so even maintenance isn’t enough

    Facebook Is Weaker Than We Knew
    A trove of leaked documents, published by The Wall Street Journal, hints at a company whose best days are behind it.

    Kevin Roose, NYT

    nytimes.com/2021/10/04/technology/facebook-files.html

    Note: edited in attempt to eliminate traces of depraved apostrophe crimes against humanity.

  12. 12.

    Steeplejack

    I use Facebook hardly at all—mainly to passively receive notifications from a handful of people—and I was mildly surprised to find out how far-reaching its collapse was. I hadn’t thought of people using their Facebook identity as a log-in to various sites, not to mention businesses basically operating on line via Facebook.

  15. 15.

    Sure Lurkalot

    @Benw: Ha! One job I had everything went down because a squirrel committed suicide via power pole. Cheapskate boss was out of town and after 2 hours and reports that repairs were going slowly, the head of HR told everyone to go home. 2 hours later, things were repaired and cheapskate boss had HR call everyone to come back. At like 2pm.

  16. 16.

    schrodingers_cat

    BJP spreads its poison through WhatsApp. It is its most diabolical and successful outreach tool. And it has made many formerly smart people into blithering idiots who mindlessly forward BJP propaganda after inhaling it themselves. So I am glad that it is down at least for sometime.

    Facebook India has had a thumb on the scale for the BJP and has helped spread the poison of hate and bigotry.

  17. 17.

    jl

    @jl: Actually, the Facebook AI to segment its audience isn’t secret sauce. It’s easy to find everything from introductory explanations, with step-by-step guides on how to game it, to deep weeds tours through the math.

    If Mr. Facebook doesn’t know what his company is doing it’s because he doesn’t want to. He has many many people who can explain the math to him.

    Note: also edited to conceal traces of apostrophe crimes against humanity.

  18. 18.

    Another Scott

    @jl: No click from me, but isn’t the headline old news?  There were jokes a year or few ago that the young hep cats regarded FB as AOL or something and they were off on TikTok and whatever else now.  I’m sure that just like Gates and MS before the “internet tidal wave”, they’re doing everything they can to get their near monopoly mojo back…

    Cheers,
    Scott.

  19. 19.

    jl

    @Another Scott: I kind of agree. The article goes through the parade of horribles and ties it in to Facebook’s increasing difficulty in making its business model work and stock valuation plausible.

    So, YMMV may vary on what the article does.

    Edit: in particular, increasing need to do everything ever cheaper, so no due diligence in trying to get adequate staffing to do what Facebook says it will do, or is doing, or has tried to do in order to not be a malfeasant lying corporation.

  22. 22.

    Martin

    For those who aren’t technically inclined, the outage is more or less the equivalent of being the only guy who can lock up the bank vault at night, and accidentally locking yourself into the vault with no way to unlock it from the inside.

    It’s one of those tasks that can’t have safeguards apart from extraordinarily detailed checklists that you always have two people implement in very formal ways, especially if you’re the size of Facebook. Someone really fucked up ‘following directions 101’ on this. Imagine guys in missile silos calling out and validating codes and turning keys simultaneously. That kind of shit.

  25. 25.

    Roger Moore

    @Gin & Tonic:

    This kind of thing is why older companies like IBM- and now Microsoft- can be slow moving.  They’ve been bitten by problems like this before, and they’ve survived by adding a procedure to prevent it from happening again.  By the time they follow every procedure to prevent every bad thing they’ve ever encountered, it can slow them down a fair bit.  OTOH, they don’t wind up dealing with this kind of problem nearly as often as the newer companies that haven’t implemented every safeguard.

  27. 27.

    HumboldtBlue

    @Martin:

    Thanks.

    Now can you explain to me how there is a weather delay in a stadium that has a roof? And just what the hell is going on with L,A. weather?

  32. 32.

    Roger Moore

    @jl: ​
     
    The thing that gets me is just how much Facebook has just plain lied to steal people’s money. Their whole advertising analytics is one lie after another. They lie about click through rates, they lied about the importance of switching to video, and so forth. It’s probably easier to find things they didn’t lie about. And every one of those lies has left a wake of companies that failed because they listened. The whole Facebook C-suite could spend the rest of their lives in prison, and it wouldn’t come close to providing justice for everyone they’ve screwed.

  35. 35.

    jl

    @Benw: One thing in the article I linked to has that I haven’t seen before are details of Facebook’s desperate and hilariously sad pitches to try get more young people and kids. Which I found to be funny like a clown.

  36. 36.

    Martin

    @Roger Moore: Except DNS is everything for Facebook. They don’t exist if it’s down or wrong. So you build procedures around it that you never, ever deviate from. It needn’t be slow, but that discipline has to be there. I know one of the guys that does it for my employer and there’s a minimum number of people who need to be present before they update the master DNS. You add speed by adding redundancy of staff so that someone out sick doesn’t force you to either delay to the next day or deviate from procedure.

  38. 38.

    Anoniminous

    @Roger Moore:

    If the computer industry was held to the same standards of professional accountability of, say, the average car mechanic every one of the buggers would be bankrupt.

  41. 41.

    Roger Moore

    @Martin: ​
     
    My point is that companies like IBM and Microsoft have managed to become venerable by the standards of their industry by creating a kind of corporate memory. That has a good side- they don’t screw up important things like their company DNS- and bad sides- they can spend so much time following checklists, they aren’t as nimble as the newcomers. Of course nimbleness can be massively overrated. It does you no good at all to be supremely nimble if it just lets you get in trouble faster.

  42. 42.

    raven

    @schrodingers_cat: From a good buddy on FB

     

    ain’t scared of facebook.

    I’ve been in a war. I’ve k;!!ed people. I k;!!ed a Chinese coloneI, cut off his ear and carried it in my shirt pocket for a month. I’ve been shot in the guts, nearly died on the operating table, and spent 6 months in the hospital. I slept in my clothes in the dirt for 8 months. I’ve been court martialed for going AWOL in Vietnam. I stood in the office of the General commanding the Army 101st Airborne Division and told him I didn’t believe in his war.

    I’ve been busted for marijuana, grown large amounts of marijuana, bought pot in 8 foreign countries, smuggled pot through customs a number of times. I’ve snorted heroin and cocaine, dropped acid, eaten mushrooms and peyote, smoked opium and crack. Been in jail several times and have god knows how many tickets.

    I’ve lived in 3 foreign countries. Been with foreign prostitutes. I’ve been scuba diving on the Great Barrier Reef in Australia, in Mexico, Virgin Islands, Key West. I’ve hitch hiked to Key West, Kentucky, Missouri, and Montana. I lived in a $13 hootch with a dirt floor for two years and bathed in the creek. Then 5 years in a one room log cabin I built with logs I cut from my property.

    I cut a finger off with a circular saw. I’ve had multiple stitches from various screw ups. I would have drowned if my friends had not drug me out of a sucking ‘hole’ in the river. I rolled a car down a hill rolling over three times. I’ve parachuted maybe 20 times out of a plane, helicopter, jet. My last jump I landed tangled up in a tall pine tree.

    So facebook, I’m not real concerned or worried about your rebuke. Scared? Is that the best you’ve got? It’s gonna take more than algorithms to make ME worry. Hell, I had only been out of facebook jail 2 weeks when I got thrown in again. But I am a nice guy, don’t want to hurt anyone. And no, I am not rehabilitated.

  43. 43.

    NotMax

    Suppose someone should take note that it’s the first Monday in October, so SCOTUS is officially back in session.

    Fasten your seat belts, it’s going to be a bumpy term.

  45. 45.

    Roger Moore

    @Anoniminous:

    If the computer industry were held to the same standards as the average car mechanic, they would have developed completely differently.  Computer code can be made to be safe- those cars depend critically on computers, after all- but it requires a degree of care that most computer people have never had to deal with.

    It’s actually  interesting to hear computer people talking about the use of computers in safety-conscious industries.  They just don’t seem to get the background of car and medical device manufacturers, so they don’t understand why they don’t want to run their computer development the same way.

  46. 46.

    PJ

    If only Facebook could be down forever, it would be one step in stemming the cancer that is killing the free world. I don’t think our lawmakers have the guts to do it, but a boy can still dream.

  48. 48.

    schrodingers_cat

    @raven: I am not personally afraid of FB but do worry about their malign influence on democracies. They need to be reigned in. Most of the anti-vax nonsense is coming from a few FB account. They have weaponized stupidity and bigotry.

  49. 49.

    Gin & Tonic

    @Roger Moore: If it’s true what was reported, that FB’s physical access control system (card-key access to doors, for those who don’t deal with this) was reliant on Internet-facing DNS, then that’s just a badly-designed system that no amount of checklists and procedures can cure.

  50. 50.

    dmsilev

    @Roger Moore:

    They just don’t seem to get the background of car and medical device manufacturers, so they don’t understand why they don’t want to run their computer development the same way.

    Tesla, which is basically ‘what if we ran a car company like a computer company’, rolling out a beta version of their self-driving car software to end-users being a case in point. What could possibly go wrong?

  52. 52.

    Tony Gerace

    @Martin: Either …   1) The speculation about a massive database cleanup today are true.   … or … 2) The design and implementation of Facebook is deeply incompetent.   … or … both     I used to work in a group that supported online systems in a hospital.  An unplanned 8-hour outage?  All of our asses would have been fired.

  53. 53.

    raven

    @HumboldtBlue:

    However, there’s a reason why this is possible. SoFi Stadium technically isn’t a “domed” stadium.

    It has a roof, but the sides of the structure are open. In reality, the field is under a canopy more than a dome-style top.

  54. 54.

    raven

    SoFi Stadium is the league’s first indoor-outdoor venue with the stadium bowl sitting under a canopy. It’s more of an awning than a fixed roof so fans can still feel the wind and rain depending on where they’re sitting or standing

  55. 55.

    L85NJGT

    FB had exclusivity with a commercially desired user base early on, but has marketed for shit ever since, and morphed into the online MeTV. That’s on Zuckerberg.

    They just start offering hosting services (aka “cloud”) last year. Good luck playing catch up in that market, and we just landed back on the visionary with no vision.

    FB will end up going the aol & yahoo route, squeezing a few bucks out of a residual user base.

  57. 57.

    Shakti

    Anyone familiar with NC-09 or Charles Graham? I love this ad, btw:

    As a legislator, I don’t play politics. I study, listen, and vote my conscience. Those values are absent in Washington today and it’s tearing us apart.

    I’m running as a Democrat to represent #NC09 in the U.S Congress and finally mend our divided spirits. pic.twitter.com/zE12XZIE9r
    — Charles Graham (@CharlesGrahamNC) October 4, 2021

  58. 58.

    Mary G

    @HumboldtBlue: We had huge thunder and drizzle! It was dripping off the roof, even though the driveway didn’t all get wet. I’m in that stage of drought when 0.01 inch of rain is celebration-worthy.

    Our June gloom weather has lasted all summer. I can’t count the number of times I looked at my phone and it was 73 degrees. The highest temp inside my house was 84 degrees all summer. We’ll probably get hellish weeks of 100+ degree weather with no humidity and Santa Ana winds bearing smoke and fire all autumn to make up for it.

  59. 59.

    Dan B

    @Martin: Seattle has very little lightning, probably an average of four bolts a year and never in summer, until this year.  Where I grew up in Ohio and Arkansas many homes had lightning rods.  It was common to see trees blown apart by lightning (also cars, statues, and building bits in fields – but that was twisters).  I was surprised that apartment buildings in Seattle had no lightning rods.  Much of California seems the same.

  60. 60.

    Anoniminous

    @Roger Moore:

    I found it interesting in 1985 when I was working in SillyCon Valley.  In 2021 the unceasing Amateur Hour is just tiresome.

    People are dying because TechBros can’t act like grown-ups.

  61. 61.

    Jim, Foolish Literalist

    @Shakti: Just saw that. Looks like he’s got a steep climb– R+6 district, I’m not familiar with the incumbent R, Dan Bishop, but I’m glad to see a Dem using 1/6 in a campaign .

  62. 62.

    Dan B

    @Gin & Tonic: I Jeff Tieldrich (sp?) Tweeted:  Instagram is down.  Can I come over and look at your food?

     

    Not technical but droll and to the point.  Andy Borowitz quoted Facebook saying they realized they helped organize the attack on the congressmen they own.

  63. 63.

    West of the Rockies

    Can’t say I feel any schadenfreude over this.  I don’t see FB or Zuckerpig especially suffering in any meaningful way.  It’s more like if professional wrestling got canceled.  Don’t watch it, wouldn’t care.

  65. 65.

    gene108

     

    @schrodingers_cat:

    India is WhatsApp’s largest market by far. I have WhatsApp because all my relatives in India use it. It’s ubiquitous there.

    What’s not to like about free calls, free video calls, and messaging people anywhere in the world?

  66. 66.

    Jim, Foolish Literalist

    I wish him luck, if nothing else it will cost Lee money

    Charlie Sykes @SykesCharlie 10h
    One-time presidential candidate Evan McMullin will mount an independent bid to oust Republican Sen. Mike Lee

  67. 67.

    different-church-lady

    Can you imagine working at FB right now, when your email no longer works & all your internal FB-based tools fail?

    At long last Facebook is learning how perilous it is to rely on Facebook for every goddamned thing.

  68. 68.

    schrodingers_cat

    @gene108: It is the vehicle by which BJP injects its poison directly in the veins of Indians who uncritically forward garbage generated by the infamous IT cell.

    Other than that it is wonderful. And I have used to call and message my family in India and be a fact checking killjoy when they forward me some baloney about Sanskrit scriptures and their cult leader Modi

    I get few if any forwards now.

  69. 69.

    Barbara

    @L85NJGT:  “An online MeTV” sounds about right. I use it occasionally keep track of political groups that use it as a way to communicate events. My fitness studio does this as well for feedback on classes. But I find that most people I know posting on Facebook for personal stuff are just wasting time. It’s mostly inane once you’ve seen enough cute puppy videos.  And I  tuned out completely on my cousins who use it to fight about politics.  Their posts are not all that coherent even when I agree with them.

    Two of my three kids absolutely disdain it. The third (coincidentally the oldest) uses it to blow off steam. I think she should find a better outlet.

  70. 70.

    Bill Arnold

    Got a Facebook recruitment email in August (2021), which included this sentence, bold mine:

    I would love to tell you about … as well as the good that Facebook is doing right now to help the world.

    I might have responded, if that part in bold hadn’t been present. (Though almost certainly with a polite no, happy with my current employer.) The part in bold actually made me angry for a moment.
    Facebook fucked around for many years. Now they are finding out. And Zuckerberg even made a bet on Trump. Very not smart.

  71. 71.

    Enhanced Voting Techniques

    @jl: From what I’ve scene, Facebook’s businesses model is just spamming you with the someone replied to your post comments to suck you into never ending flame wars.

  72. 72.

    Ken

    @dmsilev: Tesla, which is basically ‘what if we ran a car company like a computer company’, rolling out a beta version of their self-driving car software to end-users being a case in point. What could possibly go wrong?

    It depends. Did they chat with their[*] congresspeople and get the necessary fixes to the liability laws pushed through, such that a self-driving car that kills a pedestrian is just one of those “whoops” things, no fault here, too bad so sad?

    [*] English possessive pronouns have so many meanings.  Obviously I mean the congresspeople who represent the Tesla corporation, not the ones owned by the Tesla corporation.

  73. 73.

    Shakti

    @schrodingers_cat: Same. I’ve had no success in getting my relatives onto an alternative platform like Signal.  But weirdly, my OLD contacts keep popping up.

    It seems like the same relatives who used to send chain letters and email forwards used FaceBook as adult tumblr and use WhatsApp to send dumb inflammatory memes. My phone cries out every day for me to delete the memes and videos from the family group. They will not stop.

  75. 75.

    karen marie

    @schrodingers_cat: Excrpt for logging in to games. I ppay a game that requires me to have  FB account, so I do, although it’s a fake name with a burner email address and completely locked down.

    It really irks me but there doesn’t seem to be any way to force game companies to change to something else.

  76. 76.

    Jim, Foolish Literalist

    @Shakti:

    Charles Graham @CharlesGrahamNC 33m
    200K views in one hour. I launched this with 71 followers. The outpouring of support for our campaign each minute has touched my heart. Please take a moment to follow this twitter account. Are you ready to make history?

  83. 83.

    Urza

    @Dan B: Greater Seattle at least has had thunderstorms the last 3 summers.  I was told it hadn’t happened since the 90s before that.

  85. 85.

    Dan B

    @Urza: I couldn’t remember if there were summer thunderstorms last year or the year before.  It seemed like last year and this year were active.  I seem to recall T-storms coming from the east iver the Cascades but this year they originated in the Puget Sound lowlands.

    The heat waves have been very disturbing.  The city is not ready and locals seem to believe they’re an aberration that won’t happen again instead of a sign of a new trend.

  86. 86.

    Starboard Tack

    I once accidentally unplugged a server in the middle of a 6 hour backup. The software was so old it wouldn’t continue from the interruption. My fuck up made the cheapskates upgrade the app to one that took less than an hour. I did not get thanked, but also not fired.

  87. 87.

    Another Scott

    In other news, … TheHill – 1.8% of employees fired for refusing vaccine at Northwell NY health system. They phrase it a little differently, of course…

    To put that number in context, nationally quits have been rising all year and are above 4M a month now.

    The employed civilian labor force is around 153M, so 4/153 = 2.6%.

    An organization that large probably has over 1.8% annual turnover anyway…

    I’m sure this is bad news for Democrats and Biden.

    Cheers,
    Scott.

  88. 88.

    Another Scott

    Many people laughed at my Cousin Shane for continuing to use MySpace and ranking his friends.

    Doesn’t seem so funny anymore does it?

    — Three Year Letterman (@3YearLetterman) October 4, 2021

    Cheers,
    Scott.

Comments are closed.