Apparently one of the buzzwords the Obama Administration is throwing around is “tech surge”, i.e., adding manpower to healthcare.gov. This guy is right:
“‘Tech surge’ is just a couple buzz words thrown together,” a source familiar with CGI Federal tells The Verge in an email. “Anyone who knows how software development and engineering works will tell you that adding more people to a late project will only make it later. This is because you basically have to bring the new resources up to speed, which takes time away from working on the task at hand.”
Another way of putting it: you can’t make a baby by hiring 9 women to work simultaneously during one month. This guy, however, is wrong on all counts:
Clay Johnson, a former White House Presidential Innovation Fellow, does believe that people are what’s needed to fix Healthcare.gov — but not more people, just new ones. He advocates opening up the source code so independent developers can give suggestions, and then opening up requests for proposals to recruit new talent to fix specific issues.
“It’s like if you take your car to the shop and your car mechanic turns your engine upside down, you don’t pay that same mechanic to fix it,” he says. “Adding different people to the mix is the right thing to do. I don’t believe that we ought to reward this kind of gross incompetence with more money and more resources.”
Unless the software developers working on healthcare.gov are utterly incompetent, switching them out is dumb because they know the code. Management often indulges in the fantasy that code that is “well documented” can be picked up and used by anyone, but the healthcare.gov developers have been studying the code and the requirements for many months. So, for better or worse, switching them out would mean more delay and confusion as the new crew starts from zero. And open source is not a magic bullet. Having a bunch of random people review code they’ve never seen before (and where they don’t know the requirements) might in the long run lead to fixing healthcare.gov, but it won’t happen quickly because they have to get up to speed on the code just like anyone else. But I’m sure hackers looking for security back doors would love to analyze that code to find vulnerabilities.
What could help is hiring a few good, seasoned project managers to go to each team and ask them what they need, and give it to them. Sounds crazy, right? You’d be surprised how often it doesn’t happen.
(I believe the 9 months to make a baby analogy comes from the excellent and still relevant book, The Mythical Man Month.)
anthrosciguy
Really, The Mythical Man Month is ancient. Do these people not know it yet?!
And why is getting different people the right thing to do here, but oh so wrong in the case of the banks, insurance companies, ratings agencies, and brokerages that blew up our economy?
mikej
I bet that if FDR had been able to get health care reform passed he wouldn’t have rolled out a slow website.
OldDave
Just a thumbs-up for “The Mythical Man Month”. It was excellent when I first read it back when the Earth was cooling, it’s still excellent now.
MattF
Things you should never do (in software engineering) Part 1:
http://www.joelonsoftware.com/articles/fog0000000069.html
Sly
But the Very Sagely John McCain very sagely suggested that the President send Air Force One out to Silicon Valley to pick up some “smart people” and bring them back to “Warshington” to fix the problem.
I’m confused. Too bad Ted Stevens isn’t around to help us figure this out.
MikeBoyScout
Mistermix, you’re right but…
Other than the stupid implementation of authentication at the front end, just what are the list of problems needed to be fixed? Seems they’ve got some bugs, but (as we well know) they’ve not needed to roll it back out. Hell, people are still running MS Vista.
Having been part of some major rollout fkups, this doesn’t look as bad as the hullabaloo shtstorm surrounding it.
c u n d gulag
You know what pisses me off the most about this PPACA fustercluck?
It’s that a Canadian company was hired to screw the pooch on this project!!!
Couldn’t a “Made in the USA” company have done just as shitty a job?
I mean, just for the sake of optics, shouldn’t an American company been hired?
Must EVERYthing be out-sourced?
Even fusterclucks?
Which just goes to prove, that companies which are the best at getting through government red-tape to get a job, aren’t necessarily the ones that are best suited for doing the actual job!!!!!!!
Enhanced Voting Techniques
New people or more people, considering there is a tech boom going on were are they going to find all these senior programers they want to hire ASAP?
Poopyman
Not project managers, systems engineers. Although from out here at 50,000 feet it looks like all of the bottleneck is at the front end of the user interface, so I would hope they’re concentrating their efforts there.
I’d like to see somebody rework the requirements from the ground up as part of v2.0, though.
c u n d gulag
@mikej:
And look at how long it took to crack Hammurabi’s code!
killer
I don’t think that opening up the code to be open-sourced would be good. This should be one of the most secure pieces of software around. I’m kinda surprised they don’t just grind it out.
I also think that this kind of work should be done with performance clauses. Bonuses and penalties. No more money to fix problems that are inherent in what the contractor created.
Poopyman
@MikeBoyScout:
Chances are, the rollout fuckups you’re familiar with weren’t under the watchful eye of the Confederate Party and courtier press.
dpm (dread pirate mistermix)
@MikeBoyScout: As of this moment, I don’t know exactly what’s wrong with healthcare.gov, since fixes are ongoing, but in the past what’s been wrong is that it’s slow and/or crashes as someone tries to sign up.
Poopyman
@Sly: Are you serious? This from a guy who still can’t get his VCR to stop blinking “12:00”.
Nina
I really don’t like the idea of open sourcing something that has been a lightning rod for controversy and that will be handling people’s sensitive information.
Thoroughly agree with you. Adding more people to a late project always makes it later, and often lowers the quality.
Just get out of the way of the people actually doing the job, and MAYBE get them a few, targeted resources who are experts in one niche area that the team feels that they’re weak in or that they could use a review on. Get them more testers and a better test bed. But more coders? That’s crazy talk.
R-Jud
@Poopyman: And said during the 2008 campaign that he’d never “done a Google”, IIRC.
Baud
@Sly:
All this needs is a little elbow grease, a monkey wrench, and some gumption.
Sly
@Poopyman:
Of course I’m serious.
azlib
@killer:
Open Source does not mean good security. If anything opening the source code actually improves security. There is also a lot of Open Source components in the implementation. For example the JQuery libraries appear to be used on the front-end.
MikeBoyScout
@dpm (dread pirate mistermix): Sure sounds like authentication, error handling and scalability. Tough nuts, but not failure. Worse thing maybe after it’s fixed is that MBA knownothings will start talking TechSurge solutions going forward. :-( Damn you Obama!
The Pale Scot
O/T sorta,
Tea Galaxy, Voyage to the Center of Delusion
Captain Cruz and the USS Intransigent
Sibelius
@Baud: And some spit, grit and a whole lot of duct tape!*
*Madagascar 2, for those without a 5 and 6 year old
ericblair
Well, Something Needs to Be Done, so let’s Do Something. Parachuting in experts and crowdsourcing sound like good ideas to people who have no clue what the problem is, so we’ll do that.
From listening to the complaints, it looks like the problems are related to interfaces and loading. The interface problems don’t take geniuses, just the right people talking to each other. The loading problem will probably go away after the initial surges. You’ll want to do a program-level review of the issues, prioritize the fixes, and let the developers work in (relative) peace for a few weeks.
Belafon
@OldDave: I believe the word you are looking for is relevant, not excellent. As in, it still applies now as much as it did back then. Which is sad, because you would think we would have learned those lessons by now.
Poopyman
And all of this gets handled separately from the Republican-led DOS attacks, which I strongly suspect are ongoing, and I hope are being countered at least somewhat effectively.
And don’t you think for a minute that there hasn’t been at least some of that going on.
Violet
The thing that made the morning news was the committee hearing some Congressman from NJ (Dem) who called the whole thing a “monkey court”. WHERE IS THE COMITY?
The Pale Scot
Do some of that “Pour encourager les autres” stuff, they’re Canadian, they’ll understand the necessity of fighting the Prussian er… Tea Raver menace
amk
so tech surge then?
And how long before you make sure the horse is really dead?
liberal
Billmon cited reports of some standard interchange file occasionally being nangled. If so, it might not just be the front end.
SP
Verge source: Adding more people just means those familiar with the code have to slow down their work to train the new people.
Clay Johnson: So let’s get rid of the ones familiar with the code!
Kay
@MikeBoyScout:
I’ll try to explain what it’s like, although I am not good at getting this across, my son works in this field and he had no idea what was wrong after my explanation.
In the first week, two weeks, it was “slow”, and that was the problem. Honestly, though I think the slowness just disguised the problems because one would never reach the next steps.
Then they fixed the capacity (or whatever!) and the problem was parts of the thing didn’t work. Two things I saw happen were an email verification never arriving and the dropdown list for the security questions not dropping down. Obviously, this is a problem because one can’t go further without these two pieces working. By about three weeks the dropdown lists worked fine, but it still took two tries to get the email link.
This process was already going to be slow and laborious for older low income people who don’t have internet access and computers. They were always going to need help with it. For one thing, they have never purchased insurance (other than state minimum auto insurance, and they don’t read the contract, they just hit “cheapest”) so the substantive explanations take a long time and the site not working properly just adds to that.
I don’t believe there was ever going to be a “smooth” rollout. It was always going to be a lot of work involving lots of human beings, one on one. Anyone who thought differently was not taking into account what the actual process involves (they’re buying a complicated product that they are not familiar with) – but it would HELP if the site worked smoothly.
Just Some Fuckhead, Thought Leader
OMFG, now everyone is an expert software developer. The world can’t end soon enough. Is that something we can open source? Maybe just get some end of world code off of experts exchange?
Eric S.
Some years back I worked at a company where executive management engaged in the “well documented code” theory. They wanted to bring in contractors to take over a project just as coding was about to start. Only one executive, one from IT, was opposed. As the story went, he suggested their plan was akin to driving your car onto a highway, accelerating up to 60mph, and deciding to change drivers without coming to a stop first.
Shortly after he was shown the door.
Shortly after that contractors showed up.
I’ve heard from friends that 2 years after implementation they are still working on a “gap” project to make the system fully functional.
Another Botsplainer
Open Source does not mean I get to look at YOUR source code you’re running at your place of business. It means that the components of the system has Source code that you can look at. Not necessarily the same thing. In the good old days you could look at IBM’s Source code for VM on microfiche. That was fun.
pluege
“This is because you basically have to bring the new resources up to speed, which takes time away from working on the task at hand.”
While this is true that “ramping” up takes time, its not the essential issue as to why “new” people to a project will further delay the project instead of speeding it up. The main cause of the added delay is that new people always think they have a better way of doing things than what exists and in so doing – “doing it better”, break more than they fix.
The rule of thumb for software development is: twice the time and twice the budget for 2/3 of the functionality. And the reason for that is that there is no such thing as “Software Engineering”: developing new software applications is still, after all these years mostly hackery. If software was developed like bridges are engineered you wouldn’t have all of the problems we see in software.
Eric S.
@MikeBoyScout: “Worse thing maybe after it’s fixed is that MBA knownothings will start talking TechSurge solutions going forward.”
This is a depressing and probably accurate thought. And I was having a good Friday up to now.
Another Botsplainer
Also as Charles Johnson pointed out yesterday, someone got to look at the “Source Code” and freaked out over some comments that were in the code. Just comments, probably left over from a previous job. Everyone knows these contractors take the same shit from the last job and just modify for the new one, right? Everyone’s a programmer now. This is getting really stupid. If the Republicans want to hold hearings to make Healthcare.gov work better, then we should all say “please proceed”.
Gin & Tonic
@pluege: Time for Dijkstra in this discussion?
Robert Sneddon
There’s nothing intrinsically broken about the website as people are getting signed up through it. Anecdotal reports keep coming in saying this and unless they’re lying there are many paths through the existing structure to a successful outcome (i.e processing someone’s status, location, financial situation etc., creating a series of quotes for them and even allowing them to sign up for insurance).
The problem is that there are still a large number of paths that don’t work and, from my limited understanding of the structure, that success rate depends almost entirely on the backend stuff folks don’t see, the connections to the databases of the the insurance companies, financial services, government databases, zipcode lookups etc. In that case the programmers who created the healthcare website front-end are relying on a morass of badly-documented interfaces to internal databases where the folks who built them never considered that thousands of people would want to access them simultaneously in real-time. Until now.
There are load considerations too, the “door-buster” effect but planning for that takes effort too and adds complexity which may not be justified in the long run in terms of cost, staffing, equipment budgeting etc. Just designing a web-based system that can cope with a million hits a day is a lot more expensive than laying out something that will never exceed five thousand hits a day three months after it opens.
RP
A surge is a good idea, just not for the website. They should hire a bunch of people to expand the call center, and encourage everyone to call instead of going to healthcare.gov for the time being. That will give them some breathing room to fix the site.
Rolling out a site like this was always going to be difficult. The admin’s mistakes were (a) not lowering everyone’s expectations and preparing them for the bumpy rollout, and (b) relying so heavily on the website instead of investing more in good old fashioned phone calls.
dr. luba
@Sibelius: Red Green works, too.
Anya
@Just Some Fuckhead, Thought Leader: how can I make sense of SFH making me agree with him on two consecutive days? Please don’t make me like you.
Lol
Highly suggest reading this piece from a woman who did CFPB’s complaint intake website.
Kay
@RP:
I agree pretty much. I also think they over-sold the “like buying a product on Amazon” idea. That’s fine for the people who once had health insurance and are familiar with the whole thing, but if you never had health insurance you don’t know how it works.
There were limited options for people who get employer-provided insurance. They get a packet that says “pick one”, if that. All of these media people are pretending we all go out and purchase health insurance and we don’t.
jrg
True, but it might not be a bad idea to throw out some management and senior engineering. The “sunk cost” fallacy can be a powerful thing… Sometimes people double-down on bad processes and practices because to do otherwise would be an admission of failure, and because they don’t want to accept that some of the work they did was in vain.
In my experience, lower-level folks can get over this pretty easily, but this attitude can be very sticky with higher level decision makers.
Kay
@Lol:
It is good. Thanks. Pretty daunting, though. Only ten per cent of big launches succeed?
I knew some of it was people who have an interest in unfavorable comparisons between government and the private sector. That’s been a standard theme probably since the postal service “launched.”
Trakker
Unless this administration is more incompetent than I think, this Tech Surge is 95% PR, unless the contractors identified specific expertise they need but don’t have. But I find that unlikely. Most likely the contractors have the expertise needed, have a pretty good handle by now what the problems are, has fixed many of them, and will have the website working acceptably (but not flawlessly) in another week or so.
By December this will only an issue for those on the right, but as more people get coverage who never had it before, the administration can begin publicizing the tearful feel-good stories of people whose lives have been vastly improved by Obamacare. By 2015 the tweaks needed to get healthy young people to sign up will be made and by 2016 we’ll see insurance companies begin to exploit the inevitable flaws and loopholes in the system and begin legally cheating clients again. It’s the nature of capitalism…
Gin & Tonic
@Kay: I work in a somewhat similar business. Most of the design and development people here are more or less surprised the thing works at all. Most surprised are those who have worked on a project (we’ve had one here) where CGI, healthcare.gov’s prime contractor, was involved. Our primary priority with that project was taking it in-house and showing CGI the door.
PhoenixRising
@RP:
You’re right, and you’re wrong.
If the problems were confined to the web site, the smart fix would be to hire an unemployed new grad who can talk on the phone per each 50 eligible people in the state being served, and have that team take on getting ‘their’ leads into that state’s database.
Since the problems are not confined to the front end (per my own observation, having tried that idea and been lost by the system because the database itself is brittle), a hand-holding surge would not work to get every eligible citizen enrolled.
But good thinking! Part of the problem really IS the front end, and trying to do this project through an app in the first place really was a Specification of the Damned. This would have been a hell of a mess had it been a project to develop software to kludge all this ancient code in 34 states into an interface that we had 2 weeks to train groups of hundreds of people to use for data entry.
(in addition to 15 years of sales in the software development niche, I also worked for the Census Bureau in 2000 when it was time to train (temporary contract) census takers to use a mainframe…and I would have been happy to help write the specs for this thing if anyone had asked.)
slim shady
What we need is a Tech Czar. Whatever happened to Czars? They were all the rage in the 90’s.
ericblair
@RP:
And that would have been easier if Congress hadn’t denied marketing money for the Obamacare rollout.
low-tech cyclist
Tru dat, but it grates my ears when people are referred to as ‘resources.’ My office has recently gone there, and I really wish they hadn’t.
RP
@PhoenixRising: True, although someone could take down the information for processing later so that the person doesn’t have to sit on the phone.
Violet
@low-tech cyclist: Been there for years–the “Human Resources Department” rather than “Personnel.” The “person” part of it vanished decades ago.
piratedan
@Lol: great insight but again, different animal, they got to be a help desk complaint center, PPACA is supposed to be a registration throughput design. It is neat that the first one works but how much traffic does it push and the integration between systems doesn’t appear to be anywhere near as difficult.
Just sounds as if the tech company that handled the design rushed to publish and didn’t get enough people doing QA, doing the grunt work of verifying links and data capture… i.e. they didn’t vette for stupidity. So now, they’re in mega bug fix mode and still filtering usage failures. These can be fixed, but getting the fixes tested and implemented takes time.Naturally, I’d be flogging the company (behind closed doors in regards to testing the product) and take my flogging from the R’s in public and keep repeating that the fixes are ongoing and will be in place as soon as possible.
mai naem
I think they should do what I do. Hit the machines that the code was created on.
Seriously, I don’t claim to know shit about programming but what can you copy what has been used in California or NY or Kentucky and copy it for each of the states that chose not to set up their own system? You may have 20+ different systems but at least they would be all working.
Just Some Fuckhead, Thought Leader
@Anya: I’m just going through a brief agreeable phase. It will pass.
Gin & Tonic
@piratedan: take my flogging from the R’s in public
Now I see that some Republicans want to start issuing subpoenas. Because assembling documents and preparing testimony for Congress is a well-known way for allowing project managers to concentrate on fixing problems.
piratedan
@Gin & Tonic: it was going to happen regardless, after all, we’ve spent how many months on Benghazi? If this rollout had worked seamlessly, this was still coming, after all Faux has even been lying about the problems that don’t exist without much focus on the ones that do and you have asshats like Barton that don’t have the slightest clue about HIPAA sitting as board chairs.
Jockey Full of Malbec
@MattF:
Joel is well-respected in certain circles. I’ve read his site on and off for years.
Yet it’s also frankly amazing just how wrong he can be at times. Example:
Every well-run large scale program I’ve ever worked on explicitly called out a “Risk Reduction Demo”… aka “The First One We Know We Will Throw Away After We Learn From Its Mistakes.”
Sometimes, you really do have to re-break the bone and re-set it, if you want it to heal properly.
Richard Mayhew
@liberal: Billmon is referring to the 834 file — ideally, those things are cleaner than a baby right before his first set of pictures — usually they are clean as a baby after bathtime where there were bath crayons involved, and sometimes they look like a baby that just discovered his big sister’s box of poster paints and had his first exploration of the properties of various colors
JohnK
Date driven development sucks. In this case the October 1 date was fixed and the President and his crew probably demanded that date and apparently nobody dared tell them otherwise. Oregon on the other hand, delayed their web launch because they knew their site was not going to be ready. Now Oregon has their site up and apparently it works good. Oregon even made headlines for their implementation rate even without their web site running. People who live in a state without an exchange and who want to get signed up now, call the phone number or chill and wait a few weeks. How hard could it be? Life should not be totally dependent on a stupid web site.
Origuy
@Robert Sneddon: I’m repeating this part:
Because it seems to me that this is likely to be a root cause of the problem. Especially since some of those databases are in states which didn’t want the ACA to begin with. Sure the front end has problems, but the contractors may have had to divert people from it to deal with the back end interfaces.
They probably didn’t have enough people on QA, either.
Just Some Fuckhead, Thought Leader
Fuck the healthcare.gov website. I should be able to like the plan I want right from Facebook and get a monthly billing statement from Twitter.
RaflW
A chunk of this still goes back to the state Governors and Legislatures (hello, GOP asswipes!) who refused to set up state exchanges. They were basically guaranteeing that this whole jalope would be a helluva lot harder to kludge together and/or fix.
So thanks, guys (you’re mostly guys, of course). Way to keep people from accessing affordable health care! How swell of you.
catclub
@Trakker: “But I find that unlikely. Most likely the contractors have the expertise needed”
Why do I think they advertised for coders with three or more years experience on the ACA website.
And plenty of people had resumes with just that item listed.
Elie
@Kay:
Well said, Kay. The “complicated product” is a very important point and I think will need to be an ongoing focus for “navigators” and other groups/people/organizations that want to actually help. The training needed was probably way under estimated but we have no choice but to keep moving forward and doing what needs to be done.
catclub
@RaflW: This.
I want the press asking all these governors and legislatures pointed questions about why they did not do their own, so that THEIR citizens could get enrolled and “avoid the Federal Government control of healthcare” as they so quaintly put it.
ThatLeftTurnInABQ
The Tech Surge ought be modeled on the highly successful Iraq War Surge: unleash ruthless ethnic cleanings in all the databases, and give out monetary payoffs to the end users to say they are all having a wonderful experience regardless. Then we can declare victory and label anyone who doesn’t go along as a defeatist and traitor.
catclub
@ThatLeftTurnInABQ: I am surprised McCain has not already come up with this.
Matt McIrvin
@slim shady: Nooo!!!! Obama invented them because tyranny! He went back in his time machine to 1919 so he could plant the term during the Wilson administration.
Kay
@Elie:
I saw a piece on a navigator who had set up in a drugstore, so that’s a good spot, but he’s working within the bounds of the site! The poor man.
I also saw they went to paper in Texas, which was smart. It was never going to be a 15 minute process in low income neighborhoods in Texas anyway, so it’s not like PAPER! is this huge deal-breaker. By the second or third day Texas HHS people were saying they could just use paper applications.
mikefromArlington
If this CGI win was a least cost, technically acceptable proposal review format, HHS got what they paid for. The contract selection process is critically flawed. A company might have a top notch proposal writing team knowing full well they don’t actually have the staff on hand until contract reward. It’s a scam.
Kay
@Gin & Tonic:
It’s truly disturbing how many times that’s true, isn’t it? I think we have this idea that everyone everywhere knows what will happen and it’s simply a matter of “planning” but really, in you adult working life, has that been the case a lot?
In real life, I think a lot of us admit “well, it’s amazing THAT disaster worked out okay”, no matter what we do for a living.
Cris (without an H)
HA HA HA HA HA HA HA HA HA HA HA HA HA HA HA HA HA HA HA HA HA HA HA HA
schrodinger's cat
@Kay: Going to war under false pretenses, no problem but a buggy website, is the end of the world. I hate our media overlords.
schrodinger's cat
We are all IT experts now, bonus itteh bitteh kitteh committeh
schrodinger's cat
@Cris (without an H): I once inherited 20 pages of code for a simulation of magnetic materials from my adviser’s previous grad student in C, with zero comments! Good times!
Kay
@schrodinger’s cat:
But they’re always like this. They have a huge and (in my view) childish faith in the Power Of Planning.
I was somewhat interested in the response to the flu outbreak in 2009 (?) , it was one of Obama’s Katrinas.
I watched Campbell Brown harangue this CDC official because vaccine hadn’t arrived in Brown’s tony NYC neighborhood. The vaccine is distributed from federal point A to state health dept point B and goes from there to county point C and then Brown’s tony neighborhood. Federal, state, county. It’s like a 4th grade civics lesson. She was OUTRAGED over what I don’t know, and why yelling at the person who is at Point A when she should have been yelling at Point C Person helps AT ALL, I have no idea. Her whole world was crumbling. I don’t know what imaginary world she lives in, but there were bound to be fuck ups, and asking the wrong person wasn’t helping matters.
They do the same thing with terrorist attacks. “could this have been PREVENTED?”
I don’t know. There’s chaos loose in the world. Just roll with it :)
J R in WV
I completely agree that we should make sure that the current staff has what they need and stay out of the way. If people need to testify, that may keep the systems from improving until testimony is done, unless management can testify and leave the techs to work on the real problems back at the ranch.
I retired from a systems decelopment career after successfully implementing lots of new systems in state government. We dealt with government beauracrats who knew the requirements for tracking data in their program who absolutely did not want anything to do with computers.
They kept all their data in notebooks in their back pockets, and if the governor wanted to know how many new ones there were last month, it took two weeks to know the answer. You had to wait until everyone had been to their office, counted their own new ones, reported that numbert to their manager, and all the managers had reported those totals to HQ.
Imagine their amaze when a governor told them they had 24 hours to get data to him!
We were successful much of the time because we were employees without much technical management, the inmates were running the asylum, and we could be patient about getting answers. Also too, top management in general knew they needed help in the form of modernization.
The first major system we built once I was systems analyst capable had a tiny detail that few if anyone paid much attention to. Field staff were asked at the foot of a form how long the described task took, since budgetary folks were always asking how long something took and no one really knew.
The first time a guy turned in 30 hours of work a day for two weeks… well, his top level manager was a Sgt major in the Reserve, and mumbled about wearing out boots on his rear end. This meant he was happy with his new system, fo those without direct experience with sargents.
The “add more people” method really does fail. Sometimes adding some task knowledgable people to help with requirements really can help. In this case, I suspect since it is a really new thing, task experts are thin on the ground.
If it works even a slim most of the time, then we can relax and it will get better as long as people are allowed to work on it at their own pace.
I will say the one time we had federal contractors involved (at $138 an hour – I refused to accept help that billed at the first price of $175/hr) it really didn’t move us forward a bit faster than we would have gone with our current staff of employees and contractors that we procured ourselves at half that rate.
Other times when I helped manage (by committee!) a federal project of some complexity, it went very well, because one relatively small company with quite a bit of expertise in the tools and in the tasks was the company awarded the contract, which did include payments for work successfully accomplished.
I think this time, even with Republican sabotage, things are actually better than we should have expected.
pseudonymous in nc
Read The Mythical Man-Month about 15 years ago; my copy’s with a friend who definitely needed to read it for his work, and who benefitted from doing so.
@Trakker:
Honestly, it shouldn’t be. HC.gov isn’t going to be the last big federal IT project. Reed and Johnson’s NYT op-ed rightly pointed out that there are still federal web projects that are shitshows, whether they’re user-facing like USAJobs.gov or primarily internal.
You can say that this is going to fixed and still want reform in IT procurement to ensure that future projects don’t need this kind of post-launch treatment.
Ruckus
My job is like a code writer. Logically that is. I take a drawing and create products. But many times on that drawing there are bits missing or wrong or that don’t fit into the project or are just totally impossible to mfg. Changes have to be discussed and found and made. The more complex the project, the more places the part has to interface to an existing part(s) the more chances are it will not work as desired or designed or built. The process is ongoing requires knowledge and thought at every step and still fails on occasion. Lots of occasions. And then the process is repeated and the mistakes are fixed.
I wonder how many occupations are the same? Because the problem is humans, the result of their work, the result of their differences, the result of the tools that they can or decide to use. As it ever was and probably will be.
When did we decide that everything or even anything will/is possible to be perfect?
ETA And I didn’t say anything about my successes or failures to actually make the part correctly. Each person adds/subtracts to the mix at every step of the way.
LAC
@Another Botsplainer: I am actually praying for another baby Jesus snowden chapter so that mistermix can take a break from “Glitchgate” or “That’s Not It, Obama”
Ms. D. Ranged in AZ
@Kay:
I was on the site last week and it allowed me to get all the way through the application, review and sign it, and then nothing. So it looks like it’s not able to present the insurance options (nor send out an email notifying me that my options are available for viewing).
I still think it’s not a UI problem but a back end, integration issue. I’m trying to be patient and wait it out. Unfortunately a tech surge will make my wait longer….sigh.
RareSanity
I know I’m a little late, but whatever…
I don’t necessarily agree with this. I guess in general terms, it’s a good rule of thumb, but it shouldn’t be taken as gospel.
I’ve been developing software for 15 years, and managing it for the last 5, and there is some benefit to bringing people in, even in the late stages of a project. But there is a big “IF” associated with it.
If there is a particular technology, interface, or protocol that the team seems to be struggling to get right, bringing in someone that is experienced in developing for that particular area would be helpful.
Hopefully, if your system architecture was done competently, the very specific parts of code related to that area, are isolated from the more project specific stuff around it. So the “expert” would be able to fix that area, without necessarily needing to be brought up to speed on the entire project.
For example, if the current bottleneck on the Healthcare.gov website is authenticating user’s information against various governmental records, someone that has successfully implemented a system that finds and returns information from various governmental systems, would be a benefit. Doesn’t matter if the information that is being sent is an application for health insurance, and doesn’t matter if the information returned is from the IRS…it’s that data exchange that’s causing the issue, not the actual information being exchanged.
If I had to take a wild guess about what one of the issues may be, it’s that the load that the Healthcare.gov website placed on the servers at the other agencies from which it needs to exchange data, was not accounted for properly. The “remote” servers can’t handle the excessive load Healthcare.gov is generating, and the sporadic responses from those servers is causing Healthcare.gov to crash. While one user may have gotten to one point the registration/application process successfully, and since the request failures at the remote servers will be random, it does not guarantee that they will be successful at any subsequent steps in the process.
This would explain why there wasn’t a problem until there were literally millions of real (not simulated) parallel requests occurring simultaneously.
Redleg
I agree with your points, especially “What could help is hiring a few good, seasoned project managers to go to each team and ask them what they need, and give it to them.” Unless the tech folks are incompetent, what they might need most, without adding more tech folks, are managers who can help them get rid of obstacles and provide needed resources to them. If this had been done all along, these problems would probably be fairly minor and relatively quick to fix.
schrodinger's cat
May be the idea of contracting out everything is not such a good idea after all.
fuckwit
I love that.
In Ye Olden Days the less-PC version was occasionally thrown around in meetings “The boss thinks if you fuck 9 women you can have a baby in 1 month”.
I had no idea it came from Mythical Man-Month.
It’s true though: more people isn’t a solution.
Obama’s campaign did such an outstanding job on IT for their campaign, but they did it with young kids, coffee, tequila, and adrenaline. A small team of super-bright, super-aggressive Rails hackers working around the clock for a few months, is nothing like an army of government contractors working for years interfacing with IBM mainframes in multiple states.
Someguy
One has to wonder, given the technoglibertarian leanings of a lot of IT types, if this system hasn’t intentionally been sabotaged by some racist little Rand-ian cabal in the software company.
C.V. Danes
The problem here is almost certainly not the developers, but the management. Perhaps they should start by swapping out the managers.
WereBear
From a coding point of view, this cracked me up, because ol’ Bezos used to handle programmers somewhat the way Romans treated their rowers in Ben Hur.
cckids
@Kay:
This. Have none of the complainy people EVER bought health insurance? My spouse has been self-employed forever, so the times when we have had insurance, we’ve had to do the whole compare plans, deductables, etc, etc, until I thought I’d rather drink hemlock. The whole process takes a long time & is a PITA.
The website will get fixed. Its not like its been a year or something.
catclub
@Someguy: “given the technoglibertarian leanings of a lot of IT types, if this system hasn’t intentionally been sabotaged by some racist little Rand-ian cabal in the software company. ”
I suspect not. 1) plenty of similar projects with similar results. 2) Could such an asshole really keep his ( definitely his) mouth shut? I think not.
DDOS is more likely possibility than internal sabotage. And easier.
RSA
@low-tech cyclist:
Me, I’m irritated to hear people so often described as consumers, as if we’re nothing more than the front end of an alimentary canal.
pluege
@low-tech cyclist: always struck me that “resources” was an excellent way to refer to people contributing to an end product. What else would you call them? I think “resource” has a positive connotation albeit it impersonal and abstract.
Hob
@schrodinger’s cat: I work with a guy who is ideologically opposed to putting any comments in the code. When anyone brings up the subject, he’ll hold forth on how comments are bad because they might be inaccurate, or might become inaccurate in the future if someone edits the code and doesn’t update the comment to correctly describe what it’s doing now, so the only valid way for people to understand anything at all about the code is to read every line of it. Our boss seems to be cool with this, or maybe has just gotten tired of arguing with the guy. At least he doesn’t edit other people’s comments out.
BruceFromOhio
Its surprising how applicable this is to systems other than software, for example, environmental regulation or calculation of farm subsidies.
pakfur
@killer:
It is a common misconception that viewing the source code makes the system built on it “insecure”. That is not how it works. For example, Linux is open source and the code is available for downloading, inspection and modification, but Linux is not considered insecure.
What opening the software would do is increase transparency and offer a greater possibility of improving the code quality.Any security bugs would be visible to everyone (not just the developers) and would be easier to fix.
tom
Everything i’ve read on this points to not enough time to properly implement it. Only way to fix that is either with more time and or with more people. I work in web dev and we often bring more people on to assist, particularly coders, they don’t need to know the entire system, just the language to write in and the task so the comment about needing to get them up to speed on the entire system is bunk.
mapaghimagsik
@killer:
If you’re relying on secret source code for security, you’re screwed
schrodinger's cat
@Hob: Does he at least have variable names that mean something or does he use single letters?
schrodinger's cat
Has anybody checked out Sully? He is at his drama queen best on this issue. Since when did he become a tech expert.
brantl
Ifthere were a diagram of the chained steps in using the website, and a histogram of typical response times, for each step, you could easily tell where the bottlenecks are, and revamp them, leaving, largely the rest of the programming contructs alone. This would be easy, if the current contractors weren’t in cover-your-ass mode, which they undoubtedly are.
BrianM
@pluege: My understanding is that the great Peter F. Drucker coined the term “human resources” because management of that time viewed people in terms of cost rather than benefit. Taken as he meant it, it’s a positive term. But the term isn’t usually meant that way.
(It’s telling that the “human” is usually omitted.)
Ruckus
@schrodinger’s cat:
Experts, schmeberkperts. Don’t need him none of them.
pseudonymous in nc
@RareSanity:
That would be my guess: essentially, backend timeouts to legacy systems.
There’s a point at which end-user-facing services are hooking into systems that previously were originally built for internal use — and perhaps were even being accessed through terminal emulators. It’s the kind of transition that was needed to put web interfaces on GDSes for flight reservations.
MikeInSewickley
I brought this very thing up (including Brooks’ excellent book) in my web design class the past week. As a teacher, the whole web site fiasco has been a learning experience for my students and will be a case study guaranteed for a number of years to come.
This whole damn mess did not have to happen for Christ’s sake. We don’t have enough talent in this country that, not only want to do tech work, but see the ACA succeed? I am depressed…
And they didn’t attempt a cloud computing strategy to at least get past the initial ramp-up scaling? Of course, as mentioned by earlier comments, most of the continuing problems are back-end integration issues to numerous federal, state and private systems.
But AGAIN… why the hell didn’t someone start at least starting to project manage this think within weeks of it’s being passed as a law?
It will eventually settle down but the stigma will take a long time to disappear especially with all the jackasses using it as an ACA concept failure.
RobertB
@Hob – I heard “don’t use comments” in a C# class back in 2001, when MS had went from liking Hungarian Notation, to liking Camel Case. The C# instructor had the exact same rationale – the code is going to change, the comments might or might not, and the comments are going to get further and further from reality.
Personally, I think your hater of comments is mostly right; if for whatever reason I have to dig into some function/method/whatever, I’m not going to trust the comments in the function itself to be accurate. Comments on the outside of the function, like ones generated by Javadoc and used for documentation, _might_ be more accurate if used for real live documentation, and developers who don’t keep the comments in sync are shot. But if not stayed on top of, even those might not necessarily agree with code reality. Does that mean I never use comments, or name every variable x1? No, but any comments I make in the code itself are more as reminders for myself, than they are for descriptions useful to the future developers. The future developer is going to hate it anyway.
Thoughtcrime
@pluege:
Disagree. The term “Human Resources” started replacing “Personnel” at the same time corporate management decided employees were expenses to be minimized rather than assets to be developed.
smedley the uncertain
@OldDave: Right on. Just as relevant now. Brook did point some places where additional talaent could be useful but that wasn’t coding. “…while the earth was cooling….” Yeah, we had to fend off pterodactyls and other small brained critics too.
muricafukyea
roflmao….
You really need to stop this muckymux. You are just embarassing yourself talking about this stuff when you don’t even know how to admin a wordpress website.
Take a lesson from libertarian curious wr0ng on everything Cole and just talk about your pets, what you ate, the color of your bowel movements. Things your brain is capable of grasping the general concept of.
JoyfulA
@c u n d gulag: When I worked at an EAP (employee assistance program) group long ago, in the Reagan years, we got an RFP (which demands all staff CVs) requiring a sociologist on staff. Some people wanted to recruit a part-time on-call sociologist; I figured it meant the contract was already a done deal for the only EAP in the country with a staff sociologist, and the RFP was a formality. I was right.
Mnemosyne
@MikeInSewickley:
They probably did, but had to halt it until the ACA case ground its way up to the Supreme Court. Given that 99 percent of legal observers were shocked when the SC invalidated the requirement for state exchanges, I’m not sure how the project managers were supposed to know ahead of time that they were going to have to link directly to old systems in non-Exchange states.
You really should be thinking of it this way — they had less than 18 months to figure out a way to get access to the servers of 21 states that weren’t going to cooperate. I’m kind of surprised it’s working as well as it is, to be honest.
sm*t cl*de
“tech surge”, i.e., adding manpower
So a new term for what was traditionally known as a Mongolian Clusterfeck?
johnny aquitard
@RareSanity:
In other words, the Union army has to rely on Confederate agencies to supply them with the proper caliber musket balls in the necessary quantities for use in the Battle of Gettysburg.
You absolutely know there isn’t any urgency on the GOP states’ end to fix it.
Matt McIrvin
Hearing lots of “my brother’s premiums are doubling because of Obamacare”/”my sister has no insurance because of Obamacare” stories now. Tell people to look at the exchanges and they just get mad, or they say they did.
I’m thinking now that the myths are going to sink this all on their own. People are going to go uninsured and get hit with the penalty just because they refuse to use the system rationally for ideological reasons, and this will drive the narrative when their friends of friends spread it around. We were probably naive to think people would like it just because it would work.
Matt McIrvin
@RobertB: I think Bjarne Stroustrup is down on comments for that reason. The thing is, these people all say code should be “self-documenting,” but everyone thinks their own code is self-documenting no matter how unreadable it is. Most of the biggest comment-haters don’t use meaningful names or structure code to make the flow clear either.