Here’s an interesting Ars Technica piece on the Obama campaign’s technology initiatives. The main one was called “Narwhal” (I’m guessing there are some Redditors on the Obama team, and that it baconed at midnight.)
One telling comparison: the Obama campaign says that at the peak of their operation, they used a good percentage of the capacity at Amazon’s east coast data center, with up to 500 AWS instances (sort-of like a small server) doing the work. The Romney campaign’s Orca system had a single application server hosted at their own site, behind a Comcast business Internet connection, and at one point on election day, Comcast shut them down because they thought there was a denial-of-service attack in progress.
It’s common for winning campaigns to look like, well, winners, and the losing campaigns to look incompetent in hindsight, but this is a pretty clear-cut case of gross incompetence on the part of the Romney campaign.
LGRooney
Harvard, I can haz MBA too? I got out of my paper bag this morning!
r€nato
Zac Moffatt says it wasn’t a failure at all.
Cassidy
UNLIMITED CORPORATE CASH! ARGLE BARGLE!
You’d have thought they’d spend some of that money on a gotdamn clue.
Dork
That system was killer. The plan was to whale on Obama’s leads in early voting. There was an ocean of votes available, and so many minnows in the swing states that could have voted to seal the deal for Mr. MR.
rlrr
@r€nato:
Just like the Romney campaign wasn’t a failure at all…
dmsilev
The most telling comparison:
Romney tech guy: “Orca was getting 1200 entries per minute. We couldn’t handle the load!”
Obama tech guy: “We peaked at 10,000 entries per second. And stayed up.”
Also, the comments on those Ars articles are hilarious, due to the presence of a right-wing blogger who insists that Nefarious Black Guys sabotaged Romney’s software.
beltane
@dmsilev: I love how the RW will simultaneously claim that AAs are stupid and lazy and evil geniuses who hatch nefarious plots against the unsuspecting White Man. It must be the corroboratory to the lazy Mexicans stealing everyone’s jobs.
Deb T
” (I’m guessing there are some Redditors on the Obama team, and that it baconed at midnight.)”
I’m not sure what this means – something to do with a Narwhal and an Orca?
markus o'farkus
What does “good percentage” mean? AWS East can host a lot more than 500 instances.
WarMunchkin
@Deb T: It’s a reddit thing.
http://knowyourmeme.com/memes/the-narwhal-bacons-at-midnight
r€nato
@dmsilev: ah, the culture of personal responsibility.
Hillary Rettig
The most striking thing to me about Romney’s IT screwup is how the process he used showed such contempt for his workers. Not letting them practice the app ahead of time, not even showing it ahead of time, having incompetent training (which appears to have been more like marketing), and expecting them to print out 50-80 page lists of voters in their district on their own home printers *the night before the election.*
Plus, you know, the cab thing.
It’s almost as if Romney holds workers in contempt.
Who would have thought.
dmsilev
@r€nato: Yeah, pretty much that.
She also blamed the fiasco on ‘Al Gore’s developers’; I’m not sure exactly what that was supposed to mean, but I guess if Al Gore invented the Internet then perhaps he owns all developers.
Or something.
mistermix
@markus o’farkus: I don’t know, that was a claim made by the Obama guy. I’m skeptical, but it’s clear that they used a hell of a lot more computing resources than the Romney campaign.
Hillary Rettig
Also,
top-level down control
versus
distributed control
The Internet gives me hope.
liberal
I like the angle that the folks who set up Orca where basically looking for a payday instead of working really hard with the goal of setting up something that would function. And that Romney et al. can’t understand the difference.
Capitalism, bitchez!
NotMax
If these same folks had been put in charge of the Marshall Plan, the first shipments of (spoiled) foodstuffs would just now be coming ashore in Europe – delivered, natch, to the wrong port.
Pengie
@markus o’farkus: I think if you’re using enough compute resources to even show up as a blip on the radar at AWS, it’s a pretty hefty app and probably a bit bragworthy.
MobiusKlein
@mistermix: 500 AWS servers is most assuredly not a good % of AWS east servers.
Unless they mean 1%.
But yes, fluffery for the media, exaggeration for emphasis, etc.
In all likelihood, each of the 500 servers had less effective power than the laptop you are posting on. They run as a slice of a larger system, sharing resources with other projects.
mistermix
@MobiusKlein: They also did a lot of RDS, which is their on-demand relational database system. I would not be surprised if they used some non-trivial percentage of the Amazon RDS deployment in that data center because their database was large and RDS is not as widely used as AWS or EC2 or whatever buzzword Amazon uses for cloud servers.
Bill Arnold
@mistermix:
It is possible that the analytics team was sometimes also concurrently using very large clusters, perhaps several of them, and that the reporter inferred too much from some remark about AWS usage.
Higgs Boson's Mate
Contempt for technology because, ya’ know, technology is based on heathen science. Contempt for the electorate because they’re just a flock of sheep waiting to be shorn. Contempt for the opponent because who’s going to vote for a blah guy in a down economy.
What could go wrong?
castellan
@MobiusKlein: It’s possible that they reserved dedicated AMIs to do the work. You can actually grab a whole dedicated machine, if that’s what you need. It requires just a little bit more setup, and costs more, but if that’s what they did, then they might really have been a bigger blip on Amazon’s RADAR.
Of course, given their Web 2.0 savvy and open-source-loving developers and admins, I wouldn’t be surprised if they were running smaller, shared instances, but even there they might have been given some preferential treatment. If you’ve got the clout (and it doesn’t really take TOO much to have it) you can usually get some Amazon engineers and managers to help you out.
I’d be very interested to read more about the whole project, if more details emerge.
Alex
Here’s another article on Obama’s team – http://www.motherjones.com/politics/2012/11/inside-obama-campaign-tech-operation
No aquatic animals, this one concentrates more on advertising and fund-raising emails.
efroh
L-O-Fucking-L, absolutely hilarious.
Capri
The entire GOP get out the vote strategy has been to pay Ralph Reed to turn out the evangelicals. Throw in a little suppression on the other side and there’s your election. It’s the “boots on the ground” arm of the Southern Strategy, and it worked in the past. If the country’s demographics in 2012 were the same as 2002 it would have worked this time. Ignoring the country’s demographic shifts was their major mistake.
They probably didn’t think they had to do anything more, and, being in a bubble and all, had no idea how quaint their computing power was compared to the Democrats.
Infrastructure, how does that stuff work?
The Moar You Know
The mind reels. This is a setup for a small testbed, something you’d use for a couple of hundred users tops.
Romney had better pray that his donors don’t start asking questions, and that they take his word for it that all the illegal black muslims threw the election. I think they will start asking questions. Does anyone know if you can sue a political campaign that you’ve given money to for incompetence? I think this might come up in the next few weeks.
MobiusKlein
@castellan: Yes.
The general gist, rather than the specific % of AWS used, is the true take-home.
The whole point of a system like them is how it lets you go from 5 machines to 500 is one day, and back down to 5, as demand dictates.
And 500 machines for one day, even the high cost ones, won’t set you back many dollars. The AWS bill for that is way smaller than the salaries of the folks creating the system.
Now if we can get J.C. to run ballon-juice that way, they could actually support the traffic on Debate Night.
MattF
Romney’s operation was pretty much bound to fail. They were under terrific pressure because they had to deploy in just a few months– which basically guaranteed that any errors would be fatal. And errors are pretty much inevitable. By comparison, the Obama operation had over a year to do it all, starting from scratch and they just about made it. It’s just another case the old saying, that you can have any two of fast, cheap, and right.
Mike in NC
Give this much credit to Rmoney: he apparently didn’t spend one thin dime of his own vast unearned personal fortune to finance his campaign, but succeeded in bilking a handful of corrupt billionaires to foot the bill. The one thing he’s good at is running a scam.
eohippus
I saw an interview with a Romney spokesperson and the reason Team Rmoney named their system ‘Orca’ was because orcas are the main predators of narwhals and they were convinced that their orca was going to kill Obama’s narwhal. Just like they were convinced that they were going to win.
Petorado
The awesome thing about Romney’s computer glitch was that in the end someone made a profit and that the investors were happy. Customers? Product? Screw ’em. That’s pretty much the Romney worldview come to bite him in the ass.
castellan
@MobiusKlein: I, too, dream of a BJ upgrade to that kind of model. JC has two years before we’re likely to see significant load again, so here’s hoping!
The Moar You Know
“500 instances” was just backup for Hurricane Sandy. I’m sure they used a lot more. The Ars article is quite good.
Amir Khalid
These details about the running of the Romney campaign org all point to the same thing, don’t they: it just wasn’t as competent as the Obama campaign org. It says a lot about the calibre of leadership among Mitt’s senior people, and by extension a lot about Mitt’s own calibre as a leader. This information would have been highly germane to a dispassionate evaluation of Mitt the candidate. I keep saying it: the media should cover the campaign org as much as they cover the candidate. It says so much about whether the candidate is the one you want operating the machinery of government.
the Conster
@Alex:
Comparing this to Romney’s lyme disease mailer FAIL and the slightly skeptical tone in this Weekly Standard article whereby Romney’s “micro-targeting” is compared to Obama’s, is a total hoot. It’s 11 dimensional chess v. Chutes and Ladders, and they’re never going to catch up.
Dave S.
@liberal: Deriving maximal profit from minimal work is the Bain model. By all rights, Romney should be proud his lessons were so taken to heart.
maya
I don’t know. This whole thing sounds rather fishy to me. Yet, Republicans do seem to have a problem understanding where all those tubes go. The ghost of Ted what-was-his-name from Palinstine must be laughing.
MattF
@Amir Khalid: This was true in 2008 as well. Obama’s operation just outclassed Clinton’s and then went on to crush McCain’s– and now Romney’s operation has the signature tire tracks imprinted over its torso. That Obama fellow may be a Kenyan socialist mooooslem usurper, but he does seem to know how to run things.
max
but this is a pretty clear-cut case of gross incompetence on the part of the Romney campaign.
Looking at the overall picture, I’ve never seen much evidence that Romney was a good administrator. I’ve seen plenty of evidence he’s an ok to good salesman, particularly at the kind of vertical takeover & loot operations that Bain seemed to specialize in. Bain has never appeared to be an organization with lots of management types with monster organization skills so much as a lot of lawyers and tax guys set up to do a remora on existing company structures.
Given that one of Bain’s goals was always to minimize labor costs, it looks like they skimped on the manpower, probably figuring that their voters would turn out anyways. (Is it any wonder that Republicans are terrible at running wars? Their priority is always saving 10 bucks by recycling the paperclips and crap like that.)
Petorado: The awesome thing about Romney’s computer glitch was that in the end someone made a profit and that the investors were happy.
Well, some people made a profit. It doesn’t look to me like the investors were happy at all.
max
[‘True to form.’]
Feudalism Now!
A narwhal is a sea unicorn. The project art probably had it farting rainbows.
Whenever I hear ORCA all I can think about is Usual Suspects when Verbal is spinning his grift to Detective Kujan – “That guy is tense. Tension is a killer. I used to be in a barbershop quartet in Skokie, Illinois. The baritone was this guy named Kip Diskin, big fat guy, I mean, like, orca fat. He was so stressed in the morning… “
PreservedKillick
@MobiusKlein:
Aye, that’s nothing. We used 500 servers in that zone a few years ago and Amazon didn’t even blink, nor did they blink when we moved those 500 servers across zones (which meant we had 1000 total across two zones, for a little while.)
The thing about the Romney effort is that it smells like it was designed and operated by a bunch of management consultants who had no idea what they were doing. Surprise surprise.
Also, too, microsoft? Seriously? *Snort*.
eohippus
@MattF: “he does seem to know how to run things.” Why, it’s almost as if community organizers know how to organize things. Hoocooda, etc.?
Bubblegum Tate
@efroh:
I really enjoyed that part, too. Even technology sees them as malignant!
Also, I heartily agree with the notions mentioned upthread: Romney holding workers in contempt and the workers just trying to get paid instead of busting ass to make something that actually functions.
Grumpy Code Monkey
Here, as I understand it, were the problems with Orca:
1. Insufficient testing at load
2. Insufficient user testing
3. Insufficient bandwidth
4. Unbalanced architecture
5. B0rked login credentials
6. Very aggressive schedule
To me, this sounds like a typical software deployment; it’s just that the stakes usually aren’t so high, and you get more than one day to work out the problems.
The problem wasn’t that the Orca development team wasn’t competent (project management is another story); it’s just asking an awful lot of people to put something that big together in that short a time and expect it to work out of the box with no problems.
The Houdini system used by the Obama campaign in 2008 also fell down under load, for many of the same reasons. The main reason Narhwal worked as well as it did? They had much longer to work on it (538 days as opposed to 180ish, go figure).
@beltane:
She’s not a right-winger; she’s just asking questions (like why were African-Americans or Al Gore supporters allowed to work on Orca, because everyone knows Democratic supporters would do their damndest to sabotage it, because that’s just how those Democrat/Libertarian/anti-pluralist/Marxist tekkies [sic] roll).
As for outsourcing Orca development, Moffat made a good point; his company simply has no expertise in designing or building those kinds of systems. And if it was outsourced, it was likely done offshore, meaning the development team probably weren’t US citizens and didn’t have a dog in the fight one way or the other.
PreservedKillick
Comcast was right. Exactly so, Comcast. They were a denial of service attack in progress. Fortunately, they failed.
FoxinSocks
I was a lowly Data Coordinator, and I know the Obama campaign was far from perfect, but we were doing election dry runs for several weeks leading up to the big day. In addition, we practiced switching states at a moment’s notice (which we had to do on Election Day – our local group called into four states that night) and we had printed back-up lists in case we lost power or the servers became overloaded.
Though we had some minor problems the Thursday before the election, everything ran smoothly after that point.
The campaign also fostered an environment of empowerment and involvement. We had weekly strategy calls, not just with my local team, but with the whole state team so we could understand what they were doing and why. Once you were made a Coordinator (Phonebank, Data, etc.) you could pretty much sign up who you wanted as a fellow coordinator, the idea being that since you were the expert in that area and you’d be working closely with that person, you were the best judge of who else could fulfill that role.
For example, I just walked in to my local office one day and asked to do data entry. The data coordinator pulled me aside, trained me, then after a half an hour, liked me and my work enough that she asked me to become a data coordinator as well. After that, they’d periodically ask me who I thought we should add to the data team. They were keenly interested in not just ability, but who I liked to work with.
One other note: I’m a single gal and our office was surrounded by this big, empty, not well-lit parking lot. Our team leads always made sure someone watched me go to my car. I told them I didn’t need that, but they did that for everyone. We watched out for one anoher.
Zifnab25
Haha! Looks like I’m not the only person who’s been royally fucked by Comcast. Eat a dick, Romney!
Petorado
@max: not those that invested in the system, the shareholders of the company that sold the system.
And I’m sure the tech guys used every cent of the campaign’s voucher they were given to go solve their problem out on the free market.
Jennifer
The Rmoney campaign was relying on COMCAST? For something IMPORTANT?
So much for Rmoney’s vaunted “business experience.” I thought businessmen were cautious and did their research. Wonder how they failed to notice that Comcast was the most hated company in America for at least 3 years running (and only dropped to the #2 spot last year) or wonder why that might be.
Knowing that they were relying on Comcast for something as vitally important as their internet operations is actually funnier than learning that they were stunned by losing because they’d been getting all their information from Fox News.
Comcast? Seriously? That’s HILARIOUS!
PreservedKillick
@Jennifer:
We run our office on Comcast business class. It’s…ok…
But for mission critical stuff? No way you’d do that without a backup connection. A real backup connection, or more likely that connection would be primary and Comcast would be your “oh shit” connection.
MikeJ
@MattF:
Romney had six or seven years to get it right.
Jennifer
@PreservedKillick: Comcast is only “ok” as long as nothing goes wrong.
Once it does, good luck getting it fixed. Their “customer service” reps are only empowered to send a guy out to check out the line on your property. If it’s upstream from your property, no one will look into it. Not to mention every time you call them, it’s a 20 minute wait to talk to “customer service” and it’s a full day of waiting around to get someone to come out and verify that the problem isn’t on your property. Which still doesn’t fix it.
As you might guess, I fired them, after months of problems and finally having to diagnose the problem for them. Even then, it was over a week before they could be bothered to fix it – even though it was affecting other customers in the area.
LanceThruster
This is the main reason that it’s such an outrage that Team White & Delightsome was not crowned because if history has taught us anything, white privilege is there specifically to ensure the incompetent white guy is not outdone by the competent “other.” You let that get around and the entire fabric of society is liable to unravel with potentially devastating consequences.
Is THAT what you want to happen?!?
Roger Moore
@Capri:
They are not ignoring the country’s demographic shift; it’s a major motivation for their political view. They know damn well that they can’t win if those people show up and vote, which is why they’ve been putting so much effort into voter suppression. They just weren’t as good at suppressing the minority vote as they expected and needed to be to win the election.
RSA
@Grumpy Code Monkey:
I’d expand number 2. (I’ve debated putting the time into writing a detailed blog post about it.) I see a lot of user interfaces developed by novices in the classes I teach, and the Orca documentation, with screen shots, shows a user interface that was designed and implemented by people who apparently had never built a mobile application before. There are at least a dozen problems that are obvious to someone with experience.
shortstop
I seriously cannot get enough of these stories, partly for the schadenfreude and partly because I learn stuff in every one I read.
Applejinx
Bloody hell, that is unbelievable.
I was doing data entry on a little Linux box running eight terminals. When I realized we were going to a web page to do it, and saw how slick (well- mostly) the setup was, I knew serious techies had hammered together the system from top to bottom.
It choked now and then. I told people, it’s not our computer or terminals at all. It’s the server, and that server is getting hammered. Didn’t know it was that big a setup, but fact remains, it was still getting hammered, and on the whole it held up. I wouldn’t doubt the 1500 entries a second, one bit. We were getting slowed down because our terms were small and we had to side-scroll constantly to enter the data- it was laid out so it wouldn’t fit on our little terminal screens.
I too just walked in and asked to work. Ended up driving big chunks of the southern NH GOTV effort (folders with ‘turfs’) to another office in Bellows Falls, VT. The organization was REALLY well set up for communities working together to help get big things done.
My worry was that Republicans would get in and ratfuck, but I didn’t see any, I just worried. Apparently the Republicans were so worried they would be ratfucked that they hardened their whole operation against it and don’t trust their own people. Nor should they, after that big a rip-off. Orca *pfui*
Got to say big government and unglamorous busywork win this round. No logos on the Obama terminals, no glitz on those web interfaces. They looked and acted like the tiresome government issue computer systems you’d get in some big welfare office that had to turn over a heavy workload all day every day, to do what it does. Unglamorous, but boy did it ever work.
LanceThruster
@FoxinSocks:
No wonder the “government is broken” types hate competence so much. Efficiency, teamwork, shared respect and responsibility damages their ability to promote their prevailing meme.
r€nato
@MattF: bah! He’s just a community organizer!
Roger Moore
@Grumpy Code Monkey:
IOW, they didn’t start thinking about the general election until the nomination was sewn up. That’s called being penny wise and pound foolish. What’s the point of running for the nomination if you aren’t going to have the resources to win the general election? Romney knew for a long time that he was the prohibitive favorite for the nomination, but he wasn’t doing long-range planning for what the campaign was going to do after he won it.
MikeJ
@RSA: Did seem sort of dense for a touch screen phone. Maybe I just have fat fingers, but I like to keep some distance between inputs.
gwangung
@Roger Moore:
Yeah, that struck me, too. I’m sure there are some legal restrictions to deal with, but these are not unknown problems–fer FSM’s sake, the OBAMA team had these problems in 2008. You’d think a “sharp” businessman would have taken this into account and designed a) a better system and b) multiple backups (because Murphy’s Law is a Law for a reason….).
Aet
Having worked with Comcast in the past, I can say that their business-class connections are focused on a narrow-range of consumers: low-information small business buyers. Good for a few dozen office workers doing tasks spread over months. Exactly the opposite that something like a presidential campaign needs.
Comcast also tends to oversell its network services, then claim a DDoS when those services actually get used.
The more I look at it, the more Romney’s IT team looks like nepotism: Romney’s campaign hired some major contributor’s startup web design business run by the contributor’s kid.
jayackroyd
i left a comment somewhere (PEC, I think), before most of this stuff came out, that I was pretty sure Romney must have contracted this out. And, I didn’t say then, but say now that running an app, or rather, app suite, that goes from zero to one million on day one, and then shuts down at the end of day one is the most challenging scenario you can imagine.
I make fun of the 6 sigma guys (each sigma represents an order of magnitude of cost) but you’d really have to run a project like this that way–testing the living shit out of it. The internal Obama staff got that.
Roger Moore
@gwangung:
If there are legal restrictions, then how did Obama start so long before the convention? No, I’m pretty sure that the biggest restriction was that Romney didn’t want to put money into stuff for the general election until he had the nomination sewn up, mostly because that probably would have been his personal money that he couldn’t get back if he weren’t nominated.
Judas Escargot, Bringer of Loaves and Fish Sandwiches
More likely Comcast thought they were streaming Netflix in HD.
Knight of Nothing
@PreservedKillick: The Software’s Mate says “which it will be ready when it’s ready!”
@Grumpy Code Monkey: Exactly this! Romney’s skill as a businessman is obviously in sales. And phony sales pitches do not make good software.
This failure is a microcosm of Romney’s idea of governing and his campaign as a whole: what makes software great is requirements based on close scrutiny of what the users actually need, not some theoretical/ideological BS cooked up by management. What makes software great is metrics and thorough testing, not assumptions. Finally, what makes software great is collaboration and transparency, not competition and secrecy. All of this is a metaphor for good government as well.
WereBear
The part where they planned to launch the app on Election Day: that is where I think it HAD to be a scam from the get-go.
Nobody works that way! It is completely insane!
BenA
@PreservedKillick: The tools used Microsoft/Comcast aside… one server for 50 states on one connection? It’s absurd.
I could probably design a reasonable system using Microsoft products that would do the job that Orca was supposed to do.. and in the time frame they were talking about. The actual information that they’re dealing with is pretty trivial actually… but to have one server and only one pipe to deal with it? It’s beyond incompentance. I mean it couldn’t have even been a cost thing. Servers are a relatively cheap commodity in this day and age. The whole thing reeks of nepotism and graft.
Of course one of my former bosses was a right wing nut job with no respect for his employees… and with a straight face that an employee should be a good programmer because she “typed fast” and that we should give her a break. This was a guy who worked in IT. I’m thinking that’s the sort of management you had working there.
“We’ll let Fred run the project… he knows Excel.”
Frank in midtown
@LanceThruster: re 54: Schumpeter nods approvingly.
f space that
Hello, IT. Have you tried turning it off and on again?
MikeJ
@BenA: It’s not as if they had a huge universe of users, only 37k. And I did read somewhere (one of the ars articles, I think)that the db at least was a 10 machine cluster.
Frankly I would have done it as an actual phone app instead of a web app. Download the voter lists at the beginning of the day (or before) and do all the search, etc locally. All you’ve got coming back to the back end is a dribble of UIDs and a “voted” boolean. The app doesn’t even need to hear back from the server to continue functioning.
300baud
@Grumpy Code Monkey:
I see what you’re saying, but I disagree.
First, I have no fucking idea why the Republican party didn’t start building a GOTV system in, say, 2009. Or that Romney didn’t. Not building one until late in the game is either failure to plan or planning to fail. Both of which are depressingly common.
But mainly, if you’ve got 180 days and a hard deadline, then that’s what you work with. You cut scope to something minimal and get that out the door as soon as possible. (In their shoes, I would have shipped something to at least two beta field offices in 90 days or less.) Then if you have enough time, you add more features. Given that they knew they had a tight schedule, waiting until the day of for first use was idiocy. Like the Obama campaign, they should have been doing trial runs for weeks at least.
And the hosting the app in the Romney offices on a single cable provider line on which you’ve done no serious testing? That isn’t a time constraint: that’s outright incompetence.
That said, this is all par-for-the-course corporate incompetence. A large fraction of software projects end up this way, because quality and competence take a back seat to salesmanship and internal politics. The methods for doing better have been available for years, but few really follow them. Because doing better mostly doesn’t matter. Until one of your competitors actually does it right, of course. Then, suddenly, it matters.
We’ll see if anybody learns a lesson, though. That the Romney people can still deny the fail is a sign that this may happen in 2016 too.
ericblair
@BenA:
Maybe, except you’re not really concerned about the software products themselves. You’ll need one of the big cloud providers on board and a message broker backbone to handle the surge volume and get you a basic messaging capability, which you then use to build services to do the actual work and will need developers to handle the end-user clients. This is mainly putting together enterprise level big-ass services and developing business logic, not hacking code.
It’s obvious that Romney’s team had no fucking clue what they were doing, and they could have had the next five billion years before the Sun goes red giant and they wouldn’t have fixed it. You can’t simply scale up something you’d hack together in your basement to handle tens of thousands of users; it needs a completely different approach to the problem with competent architects, developers, and managers. Sounds like the project got handed off to somebody’s kid’s company that put together some crappy small business apps but really nice Powerpoints, and it all went downhill from there.
BenA
@ericblair:
That’s more or less what I was saying.
BenA
@MikeJ:
This is probably where it got hammered… the very first time everyone logs in and it’s trying to serve 37k voting lists at once…
Ruckus
@Amir Khalid:
It says so much about whether the candidate is the one you want operating the machinery of government.
That’s why they want smaller government. Conservatives know they are incompetent at actually getting anything done so they figure everyone else must be worse. That (and the black thing) are why they are pissed they lost. They really are mad at themselves for being such losers.
MikeJ
@BenA: Yet another reason to not have the first time people see the app be on election morning. But since there were no smarts on the phone end, they couldn’t save the whole list down to the phone. It was all browser based, and I don’t think it was clever enough to just keep the whole list locally.
I could be wrong though, you certainly could do local storage with html5 storage and some good client side hacking, but the impression I got from the few screens I saw was lots of server round trips. Like searching for each voter and marking them off.
Ruckus
@FoxinSocks:
I told them I didn’t need that, but they did that for everyone. We watched out for one another.
That is the difference. In a nutshell. The most important nutshell. You want to know the difference in liberals and conservatives? That’s it.
We watched out for one another.
It really is that simple.
LanceThruster
@Frank in midtown:
I’m unworthy but will take the nod.
Thanks for the name to track
—
Schumpeter and capitalism’s demise
Schumpeter’s most popular book in English is probably Capitalism, Socialism and Democracy. This book opens with a treatment of Karl Marx. While he is sympathetic to Marx’s theory that capitalism will collapse and will be replaced by socialism, Schumpeter concludes that this will not come about in the way Marx predicted. To describe it he borrowed the phrase “creative destruction”, and made it famous by using it to describe a process in which the old ways of doing things are endogenously destroyed and replaced by new ways.
Schumpeter’s theory is that the success of capitalism will lead to a form of corporatism and a fostering of values hostile to capitalism, especially among intellectuals. The intellectual and social climate needed to allow entrepreneurship to thrive will not exist in advanced capitalism; it will be replaced by socialism in some form. There will not be a revolution, but merely a trend in parliaments to elect social democratic parties of one stripe or another. He argued that capitalism’s collapse from within will come about as democratic majorities vote for restrictions upon entrepreneurship that will burden and destroy the capitalist structure, but also emphasizes non-political, evolutionary processes in society where “liberal capitalism” was evolving into democratic socialism because of the growth of workers’ self-management, industrial democracy and regulatory institutions.[17] Schumpeter emphasizes throughout this book that he is analyzing trends, not engaging in political advocacy. In his vision, the intellectual class will play an important role in capitalism’s demise. The term “intellectuals” denotes a class of persons in a position to develop critiques of societal matters for which they are not directly responsible and able to stand up for the interests of strata to which they themselves do not belong. One of the great advantages of capitalism, he argues, is that as compared with pre-capitalist periods, when education was a privilege of the few, more and more people acquire (higher) education. The availability of fulfilling work is, however, limited, and this lack, coupled with the experience of unemployment, produces discontent. The intellectual class is then able to organize protest and develop critical ideas.
Schumpeter and democratic theory
In the same book, Schumpeter expounded a theory of democracy which sought to challenge what he called the “classical doctrine”. He disputed the idea that democracy was a process by which the electorate identified the common good, and politicians carried this out for them. He argued this was unrealistic, and that people’s ignorance and superficiality meant that in fact they were largely manipulated by politicians, who set the agenda. This made a ‘rule by the people’ concept both unlikely and undesirable. Instead he advocated a minimalist model, much influenced by Max Weber, whereby democracy is the mechanism for competition between leaders, much like a market structure. Although periodic votes by the general public legitimize governments and keep them accountable, the policy program is very much seen as their own and not that of the people, and the participatory role for individuals is usually severely limited.
Barry
@Hillary Rettig: I guess that the lesson is that swindlers don’t build, they swindle.
blingee
I love this story because it says so much about what an incompetent leader and organizer Rawmoney was and what a great leader and organizer Obama is.
It all starts with them after all even though they probably never had much to do with any of the planning and execution. The best CEO’s and Leaders are not necessarily the smartest or the most organized. However, they are good at surrounding themselves with people who are. Clearly Rawmoney failed that test miserably and Obama, as always, passed with flying colors.
Barry
@MattF: “Romney’s operation was pretty much bound to fail. They were under terrific pressure because they had to deploy in just a few months—which basically guaranteed that any errors would be fatal. And errors are pretty much inevitable. By comparison, the Obama operation had over a year to do it all, starting from scratch and they just about made it. It’s just another case the old saying, that you can have any two of fast, cheap, and right.”
It’s too bad there wasn’t some sort of ‘national committee’ for the Republican party – call it the ‘RNC’. That could run things year over year, and build stuff lasting longer than a few months.
It’s too bad that Romney didn’t have business contacts, or something, who might hook him up with some marketing people.
It’s too bad that Romney wasn’t personally rich, so that he could throw a few million $ into building this sort of thing, starting last year. Or that some people weren’t willing to build it, with a handshake deal that Romney would pay them back once he had won the nomination.
I guess that a poor, small-town guy like Romney was helpless.
Taylor
I feel bad for Microsoft. Their Azure platform is actually quite good, a lot easier to get into (if you’re willing to use .NET) than AWS, and would have avoided some of the network issues with Orca. It’s not their fault that Romney IT team were a bunch of jackasses. Using Comcast as your ISP for a national election day? I can only guess that Comcast donated the network resources, otherwise I wouldn’t trust them to be the ISP for a bake sale.
There is no way a trained IT guy could have made the stupid mistakes that Team Romney made. As others have said, this sounds like a project run by management consultants, with all communication going from the top down. I mean, beta testing on election day? Jesus.
Svensker
@blingee:
This. Also why I love this story. Encapsulates the difference between the two parties in a virtual nutshell.
El Cid
Yes, but the economy of the richest, most powerful nation-state to have existed yet, with all its interactive dependence on regulation, authorization, funding, subsidies, infrastructure, and personnel, is surely a lot easier to run than a campaign GOTV database.
LanceThruster
@Frank in midtown:
Ahhh…with a reread I see Schumpeter nodding to my question.
BruceJ
[email protected]MattF:
Yeah, it’s as if he had experience with organizing things, like, you know, groups opf people to accomplish things.
Naaah, unpossible, he was just some community organizer, never worked a day in his life…
Deb T
@WarMunchkin:
Thanks. I feel enlightened.
Barry
@BenA: “This is probably where it got hammered… the very first time everyone logs in and it’s trying to serve 37k voting lists at once…”
‘where it would have been hammered’, because the way that they actually did it was to e-mail all of the volunteers looooong PDF files, and asked them to print them out – Monday afternoon/evening. Dumb on top of dumb.
The way that this system could have worked (downloading to volunteers’ phones) would be to let them login in the week before, and to download the list. Then all that would happen on Monday is updates + some downloads for the last minute people.
In the end, though, one of the many layers of stupidity would have killed them anyway.
Guest
@Capri: you have absolutely no idea what you are talking about. None.