• Menu
  • Skip to primary navigation
  • Skip to main content
  • Skip to primary sidebar

Before Header

  • About Us
  • Lexicon
  • Contact Us
  • Our Store
  • ↑
  • ↓
  • ←
  • →

Balloon Juice

Come for the politics, stay for the snark.

Let us savor the impending downfall of lawless scoundrels who richly deserve the trouble barreling their way.

Take hopelessness and turn it into resilience.

They are lying in pursuit of an agenda.

Whatever happens next week, the fight doesn’t end.

JFC, are there no editors left at that goddamn rag?

Seems like a complicated subject, have you tried yelling at it?

It’s time for the GOP to dust off that post-2012 autopsy, completely ignore it, and light the party on fire again.

In my day, never was longer.

I’m pretty sure there’s only one Jack Smith.

Since when do we limit our critiques to things we could do better ourselves?

Come for the politics, stay for the snark.

Republicans: slavery is when you own me. freedom is when I own you.

Let’s finish the job.

This really is a full service blog.

Our job is not to persuade republicans but to defeat them.

The willow is too close to the house.

Everybody saw this coming.

rich, arrogant assholes who equate luck with genius

Every one of the “Roberts Six” lied to get on the court.

They were going to turn on one another at some point. It was inevitable.

Teach a man to fish, and he’ll sit in a boat all day drinking beer.

We are aware of all internet traditions.

As long as McCarthy is Speaker, the House will never be in order.

If you tweet it in all caps, that makes it true!

Mobile Menu

  • Winnable House Races
  • Donate with Venmo, Zelle & PayPal
  • Site Feedback
  • War in Ukraine
  • Submit Photos to On the Road
  • Politics
  • On The Road
  • Open Threads
  • Topics
  • Balloon Juice 2023 Pet Calendar (coming soon)
  • COVID-19 Coronavirus
  • Authors
  • About Us
  • Contact Us
  • Lexicon
  • Our Store
  • Politics
  • Open Threads
  • War in Ukraine
  • Garden Chats
  • On The Road
  • 2021-22 Fundraising!
You are here: Home / Past Elections / Election 2012 / More Aquatic Creatures

More Aquatic Creatures

by $8 blue check mistermix|  November 15, 20129:35 am| 92 Comments

This post is in: Election 2012, Science & Technology

FacebookTweetEmail

Here’s an interesting Ars Technica piece on the Obama campaign’s technology initiatives. The main one was called “Narwhal” (I’m guessing there are some Redditors on the Obama team, and that it baconed at midnight.)

One telling comparison: the Obama campaign says that at the peak of their operation, they used a good percentage of the capacity at Amazon’s east coast data center, with up to 500 AWS instances (sort-of like a small server) doing the work. The Romney campaign’s Orca system had a single application server hosted at their own site, behind a Comcast business Internet connection, and at one point on election day, Comcast shut them down because they thought there was a denial-of-service attack in progress.

It’s common for winning campaigns to look like, well, winners, and the losing campaigns to look incompetent in hindsight, but this is a pretty clear-cut case of gross incompetence on the part of the Romney campaign.

FacebookTweetEmail
Previous Post: « Douches Wild
Next Post: Hey, Bro, Your Food Sucks »

Reader Interactions

92Comments

  1. 1.

    LGRooney

    November 15, 2012 at 9:38 am

    Harvard, I can haz MBA too? I got out of my paper bag this morning!

  2. 2.

    r€nato

    November 15, 2012 at 9:39 am

    Zac Moffatt says it wasn’t a failure at all.

  3. 3.

    Cassidy

    November 15, 2012 at 9:40 am

    UNLIMITED CORPORATE CASH! ARGLE BARGLE!

    You’d have thought they’d spend some of that money on a gotdamn clue.

  4. 4.

    Dork

    November 15, 2012 at 9:40 am

    The Romney campaign’s Orca system

    That system was killer. The plan was to whale on Obama’s leads in early voting. There was an ocean of votes available, and so many minnows in the swing states that could have voted to seal the deal for Mr. MR.

  5. 5.

    rlrr

    November 15, 2012 at 9:40 am

    @r€nato:

    Just like the Romney campaign wasn’t a failure at all…

  6. 6.

    dmsilev

    November 15, 2012 at 9:41 am

    The most telling comparison:

    Romney tech guy: “Orca was getting 1200 entries per minute. We couldn’t handle the load!”
    Obama tech guy: “We peaked at 10,000 entries per second. And stayed up.”

    Also, the comments on those Ars articles are hilarious, due to the presence of a right-wing blogger who insists that Nefarious Black Guys sabotaged Romney’s software.

  7. 7.

    beltane

    November 15, 2012 at 9:44 am

    @dmsilev: I love how the RW will simultaneously claim that AAs are stupid and lazy and evil geniuses who hatch nefarious plots against the unsuspecting White Man. It must be the corroboratory to the lazy Mexicans stealing everyone’s jobs.

  8. 8.

    Deb T

    November 15, 2012 at 9:46 am

    ” (I’m guessing there are some Redditors on the Obama team, and that it baconed at midnight.)”

    I’m not sure what this means – something to do with a Narwhal and an Orca?

  9. 9.

    markus o'farkus

    November 15, 2012 at 9:47 am

    What does “good percentage” mean? AWS East can host a lot more than 500 instances.

  10. 10.

    WarMunchkin

    November 15, 2012 at 9:48 am

    @Deb T: It’s a reddit thing.

    http://knowyourmeme.com/memes/the-narwhal-bacons-at-midnight

  11. 11.

    r€nato

    November 15, 2012 at 9:51 am

    @dmsilev: ah, the culture of personal responsibility.

  12. 12.

    Hillary Rettig

    November 15, 2012 at 9:52 am

    The most striking thing to me about Romney’s IT screwup is how the process he used showed such contempt for his workers. Not letting them practice the app ahead of time, not even showing it ahead of time, having incompetent training (which appears to have been more like marketing), and expecting them to print out 50-80 page lists of voters in their district on their own home printers *the night before the election.*

    Plus, you know, the cab thing.

    It’s almost as if Romney holds workers in contempt.

    Who would have thought.

  13. 13.

    dmsilev

    November 15, 2012 at 9:53 am

    @r€nato: Yeah, pretty much that.

    She also blamed the fiasco on ‘Al Gore’s developers’; I’m not sure exactly what that was supposed to mean, but I guess if Al Gore invented the Internet then perhaps he owns all developers.

    Or something.

  14. 14.

    mistermix

    November 15, 2012 at 9:53 am

    @markus o’farkus: I don’t know, that was a claim made by the Obama guy. I’m skeptical, but it’s clear that they used a hell of a lot more computing resources than the Romney campaign.

  15. 15.

    Hillary Rettig

    November 15, 2012 at 9:54 am

    Also,

    top-level down control

    versus

    distributed control

    The Internet gives me hope.

  16. 16.

    liberal

    November 15, 2012 at 9:56 am

    I like the angle that the folks who set up Orca where basically looking for a payday instead of working really hard with the goal of setting up something that would function. And that Romney et al. can’t understand the difference.

    Capitalism, bitchez!

  17. 17.

    NotMax

    November 15, 2012 at 9:56 am

    If these same folks had been put in charge of the Marshall Plan, the first shipments of (spoiled) foodstuffs would just now be coming ashore in Europe – delivered, natch, to the wrong port.

  18. 18.

    Pengie

    November 15, 2012 at 9:58 am

    @markus o’farkus: I think if you’re using enough compute resources to even show up as a blip on the radar at AWS, it’s a pretty hefty app and probably a bit bragworthy.

  19. 19.

    MobiusKlein

    November 15, 2012 at 10:00 am

    @mistermix: 500 AWS servers is most assuredly not a good % of AWS east servers.
    Unless they mean 1%.

    But yes, fluffery for the media, exaggeration for emphasis, etc.

    In all likelihood, each of the 500 servers had less effective power than the laptop you are posting on. They run as a slice of a larger system, sharing resources with other projects.

  20. 20.

    mistermix

    November 15, 2012 at 10:05 am

    @MobiusKlein: They also did a lot of RDS, which is their on-demand relational database system. I would not be surprised if they used some non-trivial percentage of the Amazon RDS deployment in that data center because their database was large and RDS is not as widely used as AWS or EC2 or whatever buzzword Amazon uses for cloud servers.

  21. 21.

    Bill Arnold

    November 15, 2012 at 10:06 am

    @mistermix:
    It is possible that the analytics team was sometimes also concurrently using very large clusters, perhaps several of them, and that the reporter inferred too much from some remark about AWS usage.

  22. 22.

    Higgs Boson's Mate

    November 15, 2012 at 10:09 am

    Contempt for technology because, ya’ know, technology is based on heathen science. Contempt for the electorate because they’re just a flock of sheep waiting to be shorn. Contempt for the opponent because who’s going to vote for a blah guy in a down economy.

    What could go wrong?

  23. 23.

    castellan

    November 15, 2012 at 10:09 am

    @MobiusKlein: It’s possible that they reserved dedicated AMIs to do the work. You can actually grab a whole dedicated machine, if that’s what you need. It requires just a little bit more setup, and costs more, but if that’s what they did, then they might really have been a bigger blip on Amazon’s RADAR.

    Of course, given their Web 2.0 savvy and open-source-loving developers and admins, I wouldn’t be surprised if they were running smaller, shared instances, but even there they might have been given some preferential treatment. If you’ve got the clout (and it doesn’t really take TOO much to have it) you can usually get some Amazon engineers and managers to help you out.

    I’d be very interested to read more about the whole project, if more details emerge.

  24. 24.

    Alex

    November 15, 2012 at 10:13 am

    Here’s another article on Obama’s team – http://www.motherjones.com/politics/2012/11/inside-obama-campaign-tech-operation

    No aquatic animals, this one concentrates more on advertising and fund-raising emails.

  25. 25.

    efroh

    November 15, 2012 at 10:14 am

    Comcast shut them down because they thought there was a denial-of-service attack in progress

    L-O-Fucking-L, absolutely hilarious.

  26. 26.

    Capri

    November 15, 2012 at 10:21 am

    The entire GOP get out the vote strategy has been to pay Ralph Reed to turn out the evangelicals. Throw in a little suppression on the other side and there’s your election. It’s the “boots on the ground” arm of the Southern Strategy, and it worked in the past. If the country’s demographics in 2012 were the same as 2002 it would have worked this time. Ignoring the country’s demographic shifts was their major mistake.

    They probably didn’t think they had to do anything more, and, being in a bubble and all, had no idea how quaint their computing power was compared to the Democrats.
    Infrastructure, how does that stuff work?

  27. 27.

    The Moar You Know

    November 15, 2012 at 10:22 am

    The Romney campaign’s Orca system had a single application server hosted at their own site, behind a Comcast business Internet connection

    The mind reels. This is a setup for a small testbed, something you’d use for a couple of hundred users tops.

    Romney had better pray that his donors don’t start asking questions, and that they take his word for it that all the illegal black muslims threw the election. I think they will start asking questions. Does anyone know if you can sue a political campaign that you’ve given money to for incompetence? I think this might come up in the next few weeks.

  28. 28.

    MobiusKlein

    November 15, 2012 at 10:22 am

    @castellan: Yes.

    The general gist, rather than the specific % of AWS used, is the true take-home.

    The whole point of a system like them is how it lets you go from 5 machines to 500 is one day, and back down to 5, as demand dictates.

    And 500 machines for one day, even the high cost ones, won’t set you back many dollars. The AWS bill for that is way smaller than the salaries of the folks creating the system.

    Now if we can get J.C. to run ballon-juice that way, they could actually support the traffic on Debate Night.

  29. 29.

    MattF

    November 15, 2012 at 10:23 am

    Romney’s operation was pretty much bound to fail. They were under terrific pressure because they had to deploy in just a few months– which basically guaranteed that any errors would be fatal. And errors are pretty much inevitable. By comparison, the Obama operation had over a year to do it all, starting from scratch and they just about made it. It’s just another case the old saying, that you can have any two of fast, cheap, and right.

  30. 30.

    Mike in NC

    November 15, 2012 at 10:25 am

    It’s common for winning campaigns to look like, well, winners, and the losing campaigns to look incompetent in hindsight, but this is a pretty clear-cut case of gross incompetence on the part of the Romney campaign.

    Give this much credit to Rmoney: he apparently didn’t spend one thin dime of his own vast unearned personal fortune to finance his campaign, but succeeded in bilking a handful of corrupt billionaires to foot the bill. The one thing he’s good at is running a scam.

  31. 31.

    eohippus

    November 15, 2012 at 10:26 am

    I saw an interview with a Romney spokesperson and the reason Team Rmoney named their system ‘Orca’ was because orcas are the main predators of narwhals and they were convinced that their orca was going to kill Obama’s narwhal. Just like they were convinced that they were going to win.

  32. 32.

    Petorado

    November 15, 2012 at 10:26 am

    The awesome thing about Romney’s computer glitch was that in the end someone made a profit and that the investors were happy. Customers? Product? Screw ’em. That’s pretty much the Romney worldview come to bite him in the ass.

  33. 33.

    castellan

    November 15, 2012 at 10:29 am

    @MobiusKlein: I, too, dream of a BJ upgrade to that kind of model. JC has two years before we’re likely to see significant load again, so here’s hoping!

  34. 34.

    The Moar You Know

    November 15, 2012 at 10:31 am

    “500 instances” was just backup for Hurricane Sandy. I’m sure they used a lot more. The Ars article is quite good.

  35. 35.

    Amir Khalid

    November 15, 2012 at 10:33 am

    These details about the running of the Romney campaign org all point to the same thing, don’t they: it just wasn’t as competent as the Obama campaign org. It says a lot about the calibre of leadership among Mitt’s senior people, and by extension a lot about Mitt’s own calibre as a leader. This information would have been highly germane to a dispassionate evaluation of Mitt the candidate. I keep saying it: the media should cover the campaign org as much as they cover the candidate. It says so much about whether the candidate is the one you want operating the machinery of government.

  36. 36.

    the Conster

    November 15, 2012 at 10:35 am

    @Alex:

    Comparing this to Romney’s lyme disease mailer FAIL and the slightly skeptical tone in this Weekly Standard article whereby Romney’s “micro-targeting” is compared to Obama’s, is a total hoot. It’s 11 dimensional chess v. Chutes and Ladders, and they’re never going to catch up.

  37. 37.

    Dave S.

    November 15, 2012 at 10:36 am

    @liberal: Deriving maximal profit from minimal work is the Bain model. By all rights, Romney should be proud his lessons were so taken to heart.

  38. 38.

    maya

    November 15, 2012 at 10:36 am

    I don’t know. This whole thing sounds rather fishy to me. Yet, Republicans do seem to have a problem understanding where all those tubes go. The ghost of Ted what-was-his-name from Palinstine must be laughing.

  39. 39.

    MattF

    November 15, 2012 at 10:37 am

    @Amir Khalid: This was true in 2008 as well. Obama’s operation just outclassed Clinton’s and then went on to crush McCain’s– and now Romney’s operation has the signature tire tracks imprinted over its torso. That Obama fellow may be a Kenyan socialist mooooslem usurper, but he does seem to know how to run things.

  40. 40.

    max

    November 15, 2012 at 10:39 am

    but this is a pretty clear-cut case of gross incompetence on the part of the Romney campaign.

    Looking at the overall picture, I’ve never seen much evidence that Romney was a good administrator. I’ve seen plenty of evidence he’s an ok to good salesman, particularly at the kind of vertical takeover & loot operations that Bain seemed to specialize in. Bain has never appeared to be an organization with lots of management types with monster organization skills so much as a lot of lawyers and tax guys set up to do a remora on existing company structures.

    Given that one of Bain’s goals was always to minimize labor costs, it looks like they skimped on the manpower, probably figuring that their voters would turn out anyways. (Is it any wonder that Republicans are terrible at running wars? Their priority is always saving 10 bucks by recycling the paperclips and crap like that.)

    Petorado: The awesome thing about Romney’s computer glitch was that in the end someone made a profit and that the investors were happy.

    Well, some people made a profit. It doesn’t look to me like the investors were happy at all.

    max
    [‘True to form.’]

  41. 41.

    Feudalism Now!

    November 15, 2012 at 10:42 am

    A narwhal is a sea unicorn. The project art probably had it farting rainbows.
    Whenever I hear ORCA all I can think about is Usual Suspects when Verbal is spinning his grift to Detective Kujan – “That guy is tense. Tension is a killer. I used to be in a barbershop quartet in Skokie, Illinois. The baritone was this guy named Kip Diskin, big fat guy, I mean, like, orca fat. He was so stressed in the morning… “

  42. 42.

    PreservedKillick

    November 15, 2012 at 10:43 am

    @MobiusKlein:

    500 AWS servers is most assuredly not a good % of AWS east servers.
    Unless they mean 1%.

    Aye, that’s nothing. We used 500 servers in that zone a few years ago and Amazon didn’t even blink, nor did they blink when we moved those 500 servers across zones (which meant we had 1000 total across two zones, for a little while.)

    The thing about the Romney effort is that it smells like it was designed and operated by a bunch of management consultants who had no idea what they were doing. Surprise surprise.

    Also, too, microsoft? Seriously? *Snort*.

  43. 43.

    eohippus

    November 15, 2012 at 10:44 am

    @MattF: “he does seem to know how to run things.” Why, it’s almost as if community organizers know how to organize things. Hoocooda, etc.?

  44. 44.

    Bubblegum Tate

    November 15, 2012 at 10:45 am

    @efroh:

    L-O-Fucking-L, absolutely hilarious.

    I really enjoyed that part, too. Even technology sees them as malignant!

    Also, I heartily agree with the notions mentioned upthread: Romney holding workers in contempt and the workers just trying to get paid instead of busting ass to make something that actually functions.

  45. 45.

    Grumpy Code Monkey

    November 15, 2012 at 10:46 am

    Here, as I understand it, were the problems with Orca:

    1. Insufficient testing at load
    2. Insufficient user testing
    3. Insufficient bandwidth
    4. Unbalanced architecture
    5. B0rked login credentials
    6. Very aggressive schedule

    To me, this sounds like a typical software deployment; it’s just that the stakes usually aren’t so high, and you get more than one day to work out the problems.

    The problem wasn’t that the Orca development team wasn’t competent (project management is another story); it’s just asking an awful lot of people to put something that big together in that short a time and expect it to work out of the box with no problems.

    The Houdini system used by the Obama campaign in 2008 also fell down under load, for many of the same reasons. The main reason Narhwal worked as well as it did? They had much longer to work on it (538 days as opposed to 180ish, go figure).

    @beltane:

    She’s not a right-winger; she’s just asking questions (like why were African-Americans or Al Gore supporters allowed to work on Orca, because everyone knows Democratic supporters would do their damndest to sabotage it, because that’s just how those Democrat/Libertarian/anti-pluralist/Marxist tekkies [sic] roll).

    As for outsourcing Orca development, Moffat made a good point; his company simply has no expertise in designing or building those kinds of systems. And if it was outsourced, it was likely done offshore, meaning the development team probably weren’t US citizens and didn’t have a dog in the fight one way or the other.

  46. 46.

    PreservedKillick

    November 15, 2012 at 10:47 am

    Comcast shut them down because they thought there was a denial-of-service attack in progress

    Comcast was right. Exactly so, Comcast. They were a denial of service attack in progress. Fortunately, they failed.

  47. 47.

    FoxinSocks

    November 15, 2012 at 10:47 am

    I was a lowly Data Coordinator, and I know the Obama campaign was far from perfect, but we were doing election dry runs for several weeks leading up to the big day. In addition, we practiced switching states at a moment’s notice (which we had to do on Election Day – our local group called into four states that night) and we had printed back-up lists in case we lost power or the servers became overloaded.

    Though we had some minor problems the Thursday before the election, everything ran smoothly after that point.

    The campaign also fostered an environment of empowerment and involvement. We had weekly strategy calls, not just with my local team, but with the whole state team so we could understand what they were doing and why. Once you were made a Coordinator (Phonebank, Data, etc.) you could pretty much sign up who you wanted as a fellow coordinator, the idea being that since you were the expert in that area and you’d be working closely with that person, you were the best judge of who else could fulfill that role.

    For example, I just walked in to my local office one day and asked to do data entry. The data coordinator pulled me aside, trained me, then after a half an hour, liked me and my work enough that she asked me to become a data coordinator as well. After that, they’d periodically ask me who I thought we should add to the data team. They were keenly interested in not just ability, but who I liked to work with.

    One other note: I’m a single gal and our office was surrounded by this big, empty, not well-lit parking lot. Our team leads always made sure someone watched me go to my car. I told them I didn’t need that, but they did that for everyone. We watched out for one anoher.

  48. 48.

    Zifnab25

    November 15, 2012 at 10:47 am

    The Romney campaign’s Orca system had a single application server hosted at their own site, behind a Comcast business Internet connection, and at one point on election day, Comcast shut them down because they thought there was a denial-of-service attack in progress.

    Haha! Looks like I’m not the only person who’s been royally fucked by Comcast. Eat a dick, Romney!

  49. 49.

    Petorado

    November 15, 2012 at 10:48 am

    @max: not those that invested in the system, the shareholders of the company that sold the system.

    And I’m sure the tech guys used every cent of the campaign’s voucher they were given to go solve their problem out on the free market.

  50. 50.

    Jennifer

    November 15, 2012 at 10:48 am

    The Rmoney campaign was relying on COMCAST? For something IMPORTANT?

    So much for Rmoney’s vaunted “business experience.” I thought businessmen were cautious and did their research. Wonder how they failed to notice that Comcast was the most hated company in America for at least 3 years running (and only dropped to the #2 spot last year) or wonder why that might be.

    Knowing that they were relying on Comcast for something as vitally important as their internet operations is actually funnier than learning that they were stunned by losing because they’d been getting all their information from Fox News.

    Comcast? Seriously? That’s HILARIOUS!

  51. 51.

    PreservedKillick

    November 15, 2012 at 10:51 am

    @Jennifer:

    The Rmoney campaign was relying on COMCAST? For something IMPORTANT?

    We run our office on Comcast business class. It’s…ok…

    But for mission critical stuff? No way you’d do that without a backup connection. A real backup connection, or more likely that connection would be primary and Comcast would be your “oh shit” connection.

  52. 52.

    MikeJ

    November 15, 2012 at 10:53 am

    @MattF:

    Romney’s operation was pretty much bound to fail. They were under terrific pressure because they had to deploy in just a few months

    Romney had six or seven years to get it right.

  53. 53.

    Jennifer

    November 15, 2012 at 10:55 am

    @PreservedKillick: Comcast is only “ok” as long as nothing goes wrong.

    Once it does, good luck getting it fixed. Their “customer service” reps are only empowered to send a guy out to check out the line on your property. If it’s upstream from your property, no one will look into it. Not to mention every time you call them, it’s a 20 minute wait to talk to “customer service” and it’s a full day of waiting around to get someone to come out and verify that the problem isn’t on your property. Which still doesn’t fix it.

    As you might guess, I fired them, after months of problems and finally having to diagnose the problem for them. Even then, it was over a week before they could be bothered to fix it – even though it was affecting other customers in the area.

  54. 54.

    LanceThruster

    November 15, 2012 at 11:07 am

    It’s common for winning campaigns to look like, well, winners, and the losing campaigns to look incompetent in hindsight, but this is a pretty clear-cut case of gross incompetence on the part of the Romney campaign.

    This is the main reason that it’s such an outrage that Team White & Delightsome was not crowned because if history has taught us anything, white privilege is there specifically to ensure the incompetent white guy is not outdone by the competent “other.” You let that get around and the entire fabric of society is liable to unravel with potentially devastating consequences.

    Is THAT what you want to happen?!?

  55. 55.

    Roger Moore

    November 15, 2012 at 11:07 am

    @Capri:

    Ignoring the country’s demographic shifts was their major mistake.

    They are not ignoring the country’s demographic shift; it’s a major motivation for their political view. They know damn well that they can’t win if those people show up and vote, which is why they’ve been putting so much effort into voter suppression. They just weren’t as good at suppressing the minority vote as they expected and needed to be to win the election.

  56. 56.

    RSA

    November 15, 2012 at 11:11 am

    @Grumpy Code Monkey:

    1. Insufficient testing at load
    2. Insufficient user testing
    3. Insufficient bandwidth
    4. Unbalanced architecture
    5. B0rked login credentials
    6. Very aggressive schedule

    I’d expand number 2. (I’ve debated putting the time into writing a detailed blog post about it.) I see a lot of user interfaces developed by novices in the classes I teach, and the Orca documentation, with screen shots, shows a user interface that was designed and implemented by people who apparently had never built a mobile application before. There are at least a dozen problems that are obvious to someone with experience.

  57. 57.

    shortstop

    November 15, 2012 at 11:15 am

    I seriously cannot get enough of these stories, partly for the schadenfreude and partly because I learn stuff in every one I read.

  58. 58.

    Applejinx

    November 15, 2012 at 11:15 am

    Bloody hell, that is unbelievable.

    I was doing data entry on a little Linux box running eight terminals. When I realized we were going to a web page to do it, and saw how slick (well- mostly) the setup was, I knew serious techies had hammered together the system from top to bottom.

    It choked now and then. I told people, it’s not our computer or terminals at all. It’s the server, and that server is getting hammered. Didn’t know it was that big a setup, but fact remains, it was still getting hammered, and on the whole it held up. I wouldn’t doubt the 1500 entries a second, one bit. We were getting slowed down because our terms were small and we had to side-scroll constantly to enter the data- it was laid out so it wouldn’t fit on our little terminal screens.

    I too just walked in and asked to work. Ended up driving big chunks of the southern NH GOTV effort (folders with ‘turfs’) to another office in Bellows Falls, VT. The organization was REALLY well set up for communities working together to help get big things done.

    My worry was that Republicans would get in and ratfuck, but I didn’t see any, I just worried. Apparently the Republicans were so worried they would be ratfucked that they hardened their whole operation against it and don’t trust their own people. Nor should they, after that big a rip-off. Orca *pfui*

    Got to say big government and unglamorous busywork win this round. No logos on the Obama terminals, no glitz on those web interfaces. They looked and acted like the tiresome government issue computer systems you’d get in some big welfare office that had to turn over a heavy workload all day every day, to do what it does. Unglamorous, but boy did it ever work.

  59. 59.

    LanceThruster

    November 15, 2012 at 11:16 am

    @FoxinSocks:

    No wonder the “government is broken” types hate competence so much. Efficiency, teamwork, shared respect and responsibility damages their ability to promote their prevailing meme.

  60. 60.

    r€nato

    November 15, 2012 at 11:18 am

    @MattF: bah! He’s just a community organizer!

  61. 61.

    Roger Moore

    November 15, 2012 at 11:19 am

    @Grumpy Code Monkey:

    The main reason Narhwal worked as well as it did? They had much longer to work on it (538 days as opposed to 180ish, go figure).

    IOW, they didn’t start thinking about the general election until the nomination was sewn up. That’s called being penny wise and pound foolish. What’s the point of running for the nomination if you aren’t going to have the resources to win the general election? Romney knew for a long time that he was the prohibitive favorite for the nomination, but he wasn’t doing long-range planning for what the campaign was going to do after he won it.

  62. 62.

    MikeJ

    November 15, 2012 at 11:30 am

    @RSA: Did seem sort of dense for a touch screen phone. Maybe I just have fat fingers, but I like to keep some distance between inputs.

  63. 63.

    gwangung

    November 15, 2012 at 11:31 am

    @Roger Moore:

    IOW, they didn’t start thinking about the general election until the nomination was sewn up. That’s called being penny wise and pound foolish. What’s the point of running for the nomination if you aren’t going to have the resources to win the general election? Romney knew for a long time that he was the prohibitive favorite for the nomination, but he wasn’t doing long-range planning for what the campaign was going to do after he won it.

    Yeah, that struck me, too. I’m sure there are some legal restrictions to deal with, but these are not unknown problems–fer FSM’s sake, the OBAMA team had these problems in 2008. You’d think a “sharp” businessman would have taken this into account and designed a) a better system and b) multiple backups (because Murphy’s Law is a Law for a reason….).

  64. 64.

    Aet

    November 15, 2012 at 11:42 am

    Having worked with Comcast in the past, I can say that their business-class connections are focused on a narrow-range of consumers: low-information small business buyers. Good for a few dozen office workers doing tasks spread over months. Exactly the opposite that something like a presidential campaign needs.

    Comcast also tends to oversell its network services, then claim a DDoS when those services actually get used.

    The more I look at it, the more Romney’s IT team looks like nepotism: Romney’s campaign hired some major contributor’s startup web design business run by the contributor’s kid.

  65. 65.

    jayackroyd

    November 15, 2012 at 11:43 am

    i left a comment somewhere (PEC, I think), before most of this stuff came out, that I was pretty sure Romney must have contracted this out. And, I didn’t say then, but say now that running an app, or rather, app suite, that goes from zero to one million on day one, and then shuts down at the end of day one is the most challenging scenario you can imagine.

    I make fun of the 6 sigma guys (each sigma represents an order of magnitude of cost) but you’d really have to run a project like this that way–testing the living shit out of it. The internal Obama staff got that.

  66. 66.

    Roger Moore

    November 15, 2012 at 11:49 am

    @gwangung:

    Yeah, that struck me, too. I’m sure there are some legal restrictions to deal with, but these are not unknown problems

    If there are legal restrictions, then how did Obama start so long before the convention? No, I’m pretty sure that the biggest restriction was that Romney didn’t want to put money into stuff for the general election until he had the nomination sewn up, mostly because that probably would have been his personal money that he couldn’t get back if he weren’t nominated.

  67. 67.

    Judas Escargot, Bringer of Loaves and Fish Sandwiches

    November 15, 2012 at 11:49 am

    Comcast shut them down because they thought there was a denial-of-service attack in progress

    More likely Comcast thought they were streaming Netflix in HD.

  68. 68.

    Knight of Nothing

    November 15, 2012 at 11:58 am

    @PreservedKillick: The Software’s Mate says “which it will be ready when it’s ready!”

    @Grumpy Code Monkey: Exactly this! Romney’s skill as a businessman is obviously in sales. And phony sales pitches do not make good software.

    This failure is a microcosm of Romney’s idea of governing and his campaign as a whole: what makes software great is requirements based on close scrutiny of what the users actually need, not some theoretical/ideological BS cooked up by management. What makes software great is metrics and thorough testing, not assumptions. Finally, what makes software great is collaboration and transparency, not competition and secrecy. All of this is a metaphor for good government as well.

  69. 69.

    WereBear

    November 15, 2012 at 12:07 pm

    The part where they planned to launch the app on Election Day: that is where I think it HAD to be a scam from the get-go.

    Nobody works that way! It is completely insane!

  70. 70.

    BenA

    November 15, 2012 at 12:11 pm

    @PreservedKillick: The tools used Microsoft/Comcast aside… one server for 50 states on one connection? It’s absurd.

    I could probably design a reasonable system using Microsoft products that would do the job that Orca was supposed to do.. and in the time frame they were talking about. The actual information that they’re dealing with is pretty trivial actually… but to have one server and only one pipe to deal with it? It’s beyond incompentance. I mean it couldn’t have even been a cost thing. Servers are a relatively cheap commodity in this day and age. The whole thing reeks of nepotism and graft.

    Of course one of my former bosses was a right wing nut job with no respect for his employees… and with a straight face that an employee should be a good programmer because she “typed fast” and that we should give her a break. This was a guy who worked in IT. I’m thinking that’s the sort of management you had working there.

    “We’ll let Fred run the project… he knows Excel.”

  71. 71.

    Frank in midtown

    November 15, 2012 at 12:21 pm

    @LanceThruster: re 54: Schumpeter nods approvingly.

  72. 72.

    f space that

    November 15, 2012 at 12:22 pm

    Hello, IT. Have you tried turning it off and on again?

  73. 73.

    MikeJ

    November 15, 2012 at 12:29 pm

    @BenA: It’s not as if they had a huge universe of users, only 37k. And I did read somewhere (one of the ars articles, I think)that the db at least was a 10 machine cluster.

    Frankly I would have done it as an actual phone app instead of a web app. Download the voter lists at the beginning of the day (or before) and do all the search, etc locally. All you’ve got coming back to the back end is a dribble of UIDs and a “voted” boolean. The app doesn’t even need to hear back from the server to continue functioning.

  74. 74.

    300baud

    November 15, 2012 at 12:32 pm

    @Grumpy Code Monkey:

    The main reason Narhwal worked as well as it did? They had much longer to work on it (538 days as opposed to 180ish, go figure).

    I see what you’re saying, but I disagree.

    First, I have no fucking idea why the Republican party didn’t start building a GOTV system in, say, 2009. Or that Romney didn’t. Not building one until late in the game is either failure to plan or planning to fail. Both of which are depressingly common.

    But mainly, if you’ve got 180 days and a hard deadline, then that’s what you work with. You cut scope to something minimal and get that out the door as soon as possible. (In their shoes, I would have shipped something to at least two beta field offices in 90 days or less.) Then if you have enough time, you add more features. Given that they knew they had a tight schedule, waiting until the day of for first use was idiocy. Like the Obama campaign, they should have been doing trial runs for weeks at least.

    And the hosting the app in the Romney offices on a single cable provider line on which you’ve done no serious testing? That isn’t a time constraint: that’s outright incompetence.

    That said, this is all par-for-the-course corporate incompetence. A large fraction of software projects end up this way, because quality and competence take a back seat to salesmanship and internal politics. The methods for doing better have been available for years, but few really follow them. Because doing better mostly doesn’t matter. Until one of your competitors actually does it right, of course. Then, suddenly, it matters.

    We’ll see if anybody learns a lesson, though. That the Romney people can still deny the fail is a sign that this may happen in 2016 too.

  75. 75.

    ericblair

    November 15, 2012 at 12:34 pm

    @BenA:

    I could probably design a reasonable system using Microsoft products that would do the job that Orca was supposed to do.. and in the time frame they were talking about. The actual information that they’re dealing with is pretty trivial actually… but to have one server and only one pipe to deal with it?

    Maybe, except you’re not really concerned about the software products themselves. You’ll need one of the big cloud providers on board and a message broker backbone to handle the surge volume and get you a basic messaging capability, which you then use to build services to do the actual work and will need developers to handle the end-user clients. This is mainly putting together enterprise level big-ass services and developing business logic, not hacking code.

    It’s obvious that Romney’s team had no fucking clue what they were doing, and they could have had the next five billion years before the Sun goes red giant and they wouldn’t have fixed it. You can’t simply scale up something you’d hack together in your basement to handle tens of thousands of users; it needs a completely different approach to the problem with competent architects, developers, and managers. Sounds like the project got handed off to somebody’s kid’s company that put together some crappy small business apps but really nice Powerpoints, and it all went downhill from there.

  76. 76.

    BenA

    November 15, 2012 at 12:41 pm

    @ericblair:

    Maybe, except you’re not really concerned about the software products themselves. You’ll need one of the big cloud providers on board and a message broker backbone to handle the surge volume and get you a basic messaging capability, which you then use to build services to do the actual work and will need developers to handle the end-user clients. This is mainly putting together enterprise level big-ass services, not hacking code.

    That’s more or less what I was saying.

  77. 77.

    BenA

    November 15, 2012 at 12:45 pm

    @MikeJ:

    Frankly I would have done it as an actual phone app instead of a web app. Download the voter lists at the beginning of the day (or before) and do all the search, etc locally. All you’ve got coming back to the back end is a dribble of UIDs and a “voted” boolean. The app doesn’t even need to hear back from the server to continue functioning.

    This is probably where it got hammered… the very first time everyone logs in and it’s trying to serve 37k voting lists at once…

  78. 78.

    Ruckus

    November 15, 2012 at 12:52 pm

    @Amir Khalid:
    It says so much about whether the candidate is the one you want operating the machinery of government.

    That’s why they want smaller government. Conservatives know they are incompetent at actually getting anything done so they figure everyone else must be worse. That (and the black thing) are why they are pissed they lost. They really are mad at themselves for being such losers.

  79. 79.

    MikeJ

    November 15, 2012 at 12:53 pm

    @BenA: Yet another reason to not have the first time people see the app be on election morning. But since there were no smarts on the phone end, they couldn’t save the whole list down to the phone. It was all browser based, and I don’t think it was clever enough to just keep the whole list locally.

    I could be wrong though, you certainly could do local storage with html5 storage and some good client side hacking, but the impression I got from the few screens I saw was lots of server round trips. Like searching for each voter and marking them off.

  80. 80.

    Ruckus

    November 15, 2012 at 12:59 pm

    @FoxinSocks:
    I told them I didn’t need that, but they did that for everyone. We watched out for one another.

    That is the difference. In a nutshell. The most important nutshell. You want to know the difference in liberals and conservatives? That’s it.

    We watched out for one another.

    It really is that simple.

  81. 81.

    LanceThruster

    November 15, 2012 at 1:33 pm

    @Frank in midtown:

    Schumpeter nods approvingly.

    I’m unworthy but will take the nod.

    Thanks for the name to track

    —

    Schumpeter and capitalism’s demise

    Schumpeter’s most popular book in English is probably Capitalism, Socialism and Democracy. This book opens with a treatment of Karl Marx. While he is sympathetic to Marx’s theory that capitalism will collapse and will be replaced by socialism, Schumpeter concludes that this will not come about in the way Marx predicted. To describe it he borrowed the phrase “creative destruction”, and made it famous by using it to describe a process in which the old ways of doing things are endogenously destroyed and replaced by new ways.

    Schumpeter’s theory is that the success of capitalism will lead to a form of corporatism and a fostering of values hostile to capitalism, especially among intellectuals. The intellectual and social climate needed to allow entrepreneurship to thrive will not exist in advanced capitalism; it will be replaced by socialism in some form. There will not be a revolution, but merely a trend in parliaments to elect social democratic parties of one stripe or another. He argued that capitalism’s collapse from within will come about as democratic majorities vote for restrictions upon entrepreneurship that will burden and destroy the capitalist structure, but also emphasizes non-political, evolutionary processes in society where “liberal capitalism” was evolving into democratic socialism because of the growth of workers’ self-management, industrial democracy and regulatory institutions.[17] Schumpeter emphasizes throughout this book that he is analyzing trends, not engaging in political advocacy. In his vision, the intellectual class will play an important role in capitalism’s demise. The term “intellectuals” denotes a class of persons in a position to develop critiques of societal matters for which they are not directly responsible and able to stand up for the interests of strata to which they themselves do not belong. One of the great advantages of capitalism, he argues, is that as compared with pre-capitalist periods, when education was a privilege of the few, more and more people acquire (higher) education. The availability of fulfilling work is, however, limited, and this lack, coupled with the experience of unemployment, produces discontent. The intellectual class is then able to organize protest and develop critical ideas.

    Schumpeter and democratic theory

    In the same book, Schumpeter expounded a theory of democracy which sought to challenge what he called the “classical doctrine”. He disputed the idea that democracy was a process by which the electorate identified the common good, and politicians carried this out for them. He argued this was unrealistic, and that people’s ignorance and superficiality meant that in fact they were largely manipulated by politicians, who set the agenda. This made a ‘rule by the people’ concept both unlikely and undesirable. Instead he advocated a minimalist model, much influenced by Max Weber, whereby democracy is the mechanism for competition between leaders, much like a market structure. Although periodic votes by the general public legitimize governments and keep them accountable, the policy program is very much seen as their own and not that of the people, and the participatory role for individuals is usually severely limited.

  82. 82.

    Barry

    November 15, 2012 at 1:43 pm

    @Hillary Rettig: I guess that the lesson is that swindlers don’t build, they swindle.

  83. 83.

    blingee

    November 15, 2012 at 1:49 pm

    I love this story because it says so much about what an incompetent leader and organizer Rawmoney was and what a great leader and organizer Obama is.

    It all starts with them after all even though they probably never had much to do with any of the planning and execution. The best CEO’s and Leaders are not necessarily the smartest or the most organized. However, they are good at surrounding themselves with people who are. Clearly Rawmoney failed that test miserably and Obama, as always, passed with flying colors.

  84. 84.

    Barry

    November 15, 2012 at 1:54 pm

    @MattF: “Romney’s operation was pretty much bound to fail. They were under terrific pressure because they had to deploy in just a few months—which basically guaranteed that any errors would be fatal. And errors are pretty much inevitable. By comparison, the Obama operation had over a year to do it all, starting from scratch and they just about made it. It’s just another case the old saying, that you can have any two of fast, cheap, and right.”

    It’s too bad there wasn’t some sort of ‘national committee’ for the Republican party – call it the ‘RNC’. That could run things year over year, and build stuff lasting longer than a few months.

    It’s too bad that Romney didn’t have business contacts, or something, who might hook him up with some marketing people.

    It’s too bad that Romney wasn’t personally rich, so that he could throw a few million $ into building this sort of thing, starting last year. Or that some people weren’t willing to build it, with a handshake deal that Romney would pay them back once he had won the nomination.

    I guess that a poor, small-town guy like Romney was helpless.

  85. 85.

    Taylor

    November 15, 2012 at 2:10 pm

    I feel bad for Microsoft. Their Azure platform is actually quite good, a lot easier to get into (if you’re willing to use .NET) than AWS, and would have avoided some of the network issues with Orca. It’s not their fault that Romney IT team were a bunch of jackasses. Using Comcast as your ISP for a national election day? I can only guess that Comcast donated the network resources, otherwise I wouldn’t trust them to be the ISP for a bake sale.

    There is no way a trained IT guy could have made the stupid mistakes that Team Romney made. As others have said, this sounds like a project run by management consultants, with all communication going from the top down. I mean, beta testing on election day? Jesus.

  86. 86.

    Svensker

    November 15, 2012 at 2:11 pm

    @blingee:

    This. Also why I love this story. Encapsulates the difference between the two parties in a virtual nutshell.

  87. 87.

    El Cid

    November 15, 2012 at 2:26 pm

    Yes, but the economy of the richest, most powerful nation-state to have existed yet, with all its interactive dependence on regulation, authorization, funding, subsidies, infrastructure, and personnel, is surely a lot easier to run than a campaign GOTV database.

  88. 88.

    LanceThruster

    November 15, 2012 at 2:39 pm

    @Frank in midtown:

    Ahhh…with a reread I see Schumpeter nodding to my question.

  89. 89.

    BruceJ

    November 15, 2012 at 4:45 pm

    !@MattF:
    Yeah, it’s as if he had experience with organizing things, like, you know, groups opf people to accomplish things.

    Naaah, unpossible, he was just some community organizer, never worked a day in his life…

  90. 90.

    Deb T

    November 15, 2012 at 5:39 pm

    @WarMunchkin:
    Thanks. I feel enlightened.

  91. 91.

    Barry

    November 16, 2012 at 8:44 am

    @BenA: “This is probably where it got hammered… the very first time everyone logs in and it’s trying to serve 37k voting lists at once…”

    ‘where it would have been hammered’, because the way that they actually did it was to e-mail all of the volunteers looooong PDF files, and asked them to print them out – Monday afternoon/evening. Dumb on top of dumb.

    The way that this system could have worked (downloading to volunteers’ phones) would be to let them login in the week before, and to download the list. Then all that would happen on Monday is updates + some downloads for the last minute people.

    In the end, though, one of the many layers of stupidity would have killed them anyway.

  92. 92.

    Guest

    November 18, 2012 at 7:30 am

    @Capri: you have absolutely no idea what you are talking about. None.

Comments are closed.

Primary Sidebar

Political Action

Postcard Writing Information

Recent Comments

  • Tony Jay on Late Night Open Thread: Rude Mechanicals (Sep 27, 2023 @ 5:28am)
  • lowtechcyclist on Late Night Open Thread: Rude Mechanicals (Sep 27, 2023 @ 5:16am)
  • HumboldtBlue on Late Night Open Thread: Rude Mechanicals (Sep 27, 2023 @ 4:14am)
  • Baud on Late Night Open Thread: Rude Mechanicals (Sep 27, 2023 @ 4:07am)
  • sab on Late Night Open Thread: Rude Mechanicals (Sep 27, 2023 @ 3:58am)

🎈Keep Balloon Juice Ad Free

Become a Balloon Juice Patreon
Donate with Venmo, Zelle or PayPal

Balloon Juice Posts

View by Topic
View by Author
View by Month & Year
View by Past Author

Featuring

Medium Cool
Artists in Our Midst
Authors in Our Midst
We All Need A Little Kindness
What Has Biden Done for You Lately?

Balloon Juice Meetups!

All Meetups
Talk of Meetups – Meetup Planning

Fundraising 2023-24

Wis*Dems Supreme Court + SD-8

Calling All Jackals

Site Feedback
Nominate a Rotating Tag
Submit Photos to On the Road
Balloon Juice Mailing List Signup
Balloon Juice Anniversary (All Links)
Balloon Juice Anniversary (All Posts)

Twitter / Spoutible

Balloon Juice (Spoutible)
WaterGirl (Spoutible)
TaMara (Spoutible)
John Cole
DougJ (aka NYT Pitchbot)
Betty Cracker
Tom Levenson
TaMara
David Anderson
Major Major Major Major
ActualCitizensUnited

Join the Fight!

Join the Fight Signup Form
All Join the Fight Posts

Balloon Juice for Ukraine

Donate

Cole & Friends Learn Español

Introductory Post
Cole & Friends Learn Español

Site Footer

Come for the politics, stay for the snark.

  • Facebook
  • RSS
  • Twitter
  • YouTube
  • Comment Policy
  • Our Authors
  • Blogroll
  • Our Artists
  • Privacy Policy

Copyright © 2023 Dev Balloon Juice · All Rights Reserved · Powered by BizBudding Inc

Share this ArticleLike this article? Email it to a friend!

Email sent!