I am not a programmer (I am at best a mediocre SAS and vaguely decent STATA programmer — let me use SQL please) and more importantly I am not an IT project manager. However I worked at UPMC Health Plan in the System Configuration department which maintained and updated the claims system that was mission critical to paying billions of dollars in claims per year. One of my first assignments for 2+ years was tracking system change requests. The big project was “EXPANSION” which meant adding either two and three additional characters to Plan ID, member ID, and a few other fields to a claims system that had been first deployed in 1983( +/- a year or two as my memory is faulty.) That project took over a year, and most of the company’s dedicated claims systems programmers. We had multiple promotions from DEV to TEST and back to DEV as weird things happened, and then back to TEST and finally to UAT for a month before a 96 hour mandatory overtime promotion to PROD. If EXPANSION failed in PROD, roll-back would have occurred to a stable saved version but it would have been painful.
So this is bad, (per Josh Marshall), right?
The DOGE team, which appears to be mainly or only Elez for the purposes of this project, has already made extensive changes to the code base for the payment system. They have not locked out the existing programmer/engineering staff but have rather leaned on them for assistance, which they appear to have painedly provided hoping to prevent as much damage as possible — “damage” in the sense not of preventing the intended changes but avoiding crashes or a system-wide breakdown caused by rapidly pushing new code into production with a limited knowledge of the system and its dependencies across the federal government.
Bring on the meteor
Mr. Bemused Senior
They will probably break something.
sentient ai from the future
this is not “bad”
this is “make sure you have two weeks of nonperishable food and water available”
matt
they’re gonna get rid of the old versions of the code in source control.
Keith P.
Pushing directly to PROD without approval is usually something that will get you fired (or at a minimum, severely reprimanded). There are various reasons – safety, legal, auditing – for this, and it is a huge no-no in enterprise environments. I’ve seen the smallest, most innocuous change somehow manage to cause an unexpected error.
WaitingForMountainLife
this is horrifying – I have 25 years experience in software and I’ve worked for several of the biggest and most well known of the software companies in the US. I wouldn’t dream of changing code in any of their large, essential systems on the basis of less than one week of access. Not to mention, my understanding is that these 6 guys are at the level of interns or new college grads. No software company trusts employees at this level with anything critical.
(I’m mostly a lurker, but this post drew me out)
WaitingForMountainLife
@Keith P.: Totally agree. I’ve seen a minor config change cause millions in losses – it was fixed in one minute.
sixthdoctor
@matt: Pray for tape backups in a vault or a patriot with a thumb drive…
Bring on the meteor
Going for false vacuum decay myself.
TheOtherHank
What sentient ai from the future said. Holy shit, it’s all going to go up in flames.
@mistermix.bsky.social
If the code is in version control, they can roll back the changes, after these fuckers are arrested.
That’s in a just world, don’t know what’s going to happen in this world.
CaseyL
I keep going back and forth between “fight everything, with everything you have” and “let them crash it all” – because it occurs to me that making the data unavailable at all, to anyone, might limit the amount of damage they can do.
It increasingly seems to me that Obama and Biden were mere interregnums in the accelerating process of the US becoming undone as a country.
Our differences are too vast, the systems too decrepit, and the underlying structure too corrupt. I last thought this way during W’s Administration, though really you can trace it back to the 12 uninterrupted years of GOP dominance from 1981 to 1993, not to mention that Clinton’s election was the signal for the GOP to declare that any Democratic Presidency was per se illegitimate, to be stymied and stifled and undone as soon as possible.
And now the US voters are evenly decided between wanting sane good government and lunatic, repellent sociopathic government. Doesn’t much matter how we got here; this is not a chasm that can be bridged.
I depend quite a bit on a functioning Federal government, but I have reached the end of my rope in tolerating what we have to endure to have one. Let Trump be Trump and bring on the Apocalypse.
MobiusKlein
It’s bad. I certainly complain about the extra red tape sometimes applied at (big fintech company you know), and there have been overreactions to incidents. But no way we let the guy with experienced in hours start pushing to prod directly.
At a company, the CEO is answerable to the shareholders. Here, fuck if I know who.
oklahomo
I’ve done COBOL and RPG on mainframe/midrange for most of my adult life. I don’t know what data base and high level languages are being used, but I’m trying to picture myself just sitting down and starting to CHANGE things that I have no idea where they are used, what programs need reviewed BEFORE the changes…it’s insane. There was a reason during the pandemic that it was hard to upgrade a lot of the old state unemployment systems — finding people with the skills to upgrade a complex system in ancient legacy code.
Tim C.
@CaseyL: Agree on everything except for one thing that might not be popular. A significant number (no idea as a percentage, but I’ll ball-park it at 25%) of GOP voters are not sociopaths. They are just totally inside an information bubble that’s going to pop hard. I absolutely feel the rage others do at all of them for what they have done or will do. But… we have to be ready for the chance to fix things, and that means working with repentent former adversaries when we can.
Jaybird
This is nothing short of horrifying. I am a retired IT program manager who started as a programmer. Interns working directly on production systems (especially ones this large), and I worked for a *large* financial institution for most of my professional life) is a) nuts by any reasonable standard b) instantly fireable c) Hopefully they haven’t destroyed the version control system.
Edmund dantes
@@mistermix.bsky.social: if Dems ever get back into power they need to bring the full force of the Feds down on these guys. Like should have been done every previous time. Maybe this time the Dems learn their lesson and remember to look back.
Urza
I’ve done text updates in a table that can cause problems in prod. I despise the modern way of not doing QA for code, giving it a quick once over in preprod and pushing it out. But at least that is a slow rollout with stages and stopping when theres a problem even if it might affect customers it never goes to the world all at once.
TBone
WTF, OVER! WTF, OVER!
(Buddy from ‘Nam would shout this repeatedly at inappropriate times here at home because it finally got him out of a foxhole when the helicopter couldn’t find him.)
Dougb
@Edmund dantes: We’re sadly talking about the Dems here – so no they won’t ever learn any lessons related to holding those responsible for past malfeasance accountable by the full force of the law. “Look forward rather than backwards” was always the most stupid shit imaginable as a justification for not enforcing the laws with respect to right wing and Republican malfeasance and norm and law breaking.
Mr. Bemused Senior
I can understand why Obama didn’t pursue wrongdoing by the W administration. If he had there would be no ACA. But I agree with you and Heather Cox Richardson: failing to punish criminals leads to more lawlessness.
Elizabelle
@CaseyL: Didn’t Fox News debut in 1993 or so? The original sanewashing. Of John Bircher ugliness and idiocy, and good old American racism and “exceptionalism.”
Race to the bottom ever since, with some pushback as our institutions were hollowed out too.
Rush Limbaugh, in Hell, also did this.
brendancalling
@Dougb: They won’t learn shit, as Chuck Schumer deems determined to prove this week.
As for the computer systems? Gonna be fun when the Social Security checks don’t reach those nice folks in the Villages or whatever that stupid condo complex in Florida is called.
oldgold
I do not understand why this activity cannot be enjoined.
What legal status do these Doge team members have?
I read somewhere yesterday, could Biden have invited Selena Gomez to join the executive branch as an advisor and then allowed her to shut down ICE?
sentient ai from the future
it’s even quite a bit worse than even the corporate IT folks here are proposing. those systems folks have worked on were, uh, not also incredibly obvious targets for a hostile state’s security services.
musk and his teenage goons are absolute pikers when it comes to security and tradecraft. musk himself talks to vlad and his actions are clearly informed by someone who’s executed an administrative coup.
so while your hair is turning white(er), lemme propose this scenario.
this dipshit incel plugs in a usb key to the treasury systems.
and then the treasury itself becomes the target of a ransomware attack.
Paul W.
After COVID I thought that Republicans were a Death Cult in the sense of that one disease only, but now it appears that Republicans at every level (from Senators, to Cabinet members who should know better, etc) are dead set on shooting themselves in the foot until they themselves and everyone around them is also dead.
I hope John Roberts is happy he played interference for these fucks, with whatever months we have left as what I would have once called a country.
Michael J Windbigler
This is bad. And in the billion dollars health provider i currently work for you would be walked out the door.
Elizabelle
@Dougb: auditioning for the pie filter ??
sentient ai from the future
@Dougb: there are some voices out there. wyden just today talking about elon’s “hatchet men” and how this is coup. it’s out there if you want to look. we as citizens need to induce other electeds to follow their lead.
Kelly
I’m a retired IT guy. Worked for 3 Fortune 500 corps. Mostly IBM mainframe Cobol and a few years of Assembler. I retired in from IT 2002 after a lot of Y2K work. Haven’t worked IT since so I’m not at all knowledgeable of current methods.
However…
Oh Hell No Newguy
CaseyL
@Tim C.:
I know the country has been here before, many times, and many even worse times. The damage I rage at now is nothing compared to what has happened in the past; it’s just that I’m alive now to experience and witness it personally, so it hurts more.
And also, I guess, because we’re supposed to know better now. We’re supposed to have learned from our mistakes, not go boldly forth to deliberately make them again.
Belafon
And what changes did they decide needed to be made?
Keith P.
@@mistermix.bsky.social: Unless their code corrupts the data, in which case they have to restore the data and then (meticulously) reapply any changes back to the data with corrected logic
@Kelly: I’ve been at it for 25+ years, mostly in the enterprise. The main change has been going from waterfall to Agile/Scum where releases are smaller and more frequent. But the thing is, even the new methodologies require adherence to procedure. It sounds like Musk’s team is doing “cowboy coding”, which can work on small projects and at small companies, but the US government is the most enterprise of all environments – lots of their requirements are *law
sentient ai from the future
@Kelly: *whispering* how long do you think it will take to propagate if we rename “sandbox” to “prod”?
Betty Cracker
@Edmund dantes: I’m not optimistic on that score, but the appropriate consequences would be dramatic, like nationalizing Musk’s companies, seizing his assets as payout to victims and deporting his ass to South Africa (in handcuffs and an orange jumpsuit via military transport) for lying about his immigration status. Hey, as long as we’re dreaming, dream BIG, amirite?
kindness
@Dougb: Can we please have a thread where Democrats aren’t the bad guy? Seriously.
sentient ai from the future
@Keith P.: if there’s anything i really do wholeheartedly trust the federal government IT force at, it’s making regular, thorough backups kept at multiple remote locations
Belafon
@Betty Cracker: I would also give him a laptop and a bag containing the materials for a parachute and tell him to figure out how to put it together on the way down. “You will have a Starlink connection.”
Tim C.
@CaseyL: RIGHT!??!?!
Like… so much of this is just…so completely stupid. It’s not hard. This is easy crap. But so many people are just broken inside or so blindly manipulated.
stinger
@WaitingForMountainLife: Is this how they handle coding at SpaceX? Send in a new kid, let them play in the sandbox for a day, then upload to PROD?
Josie
@sentient ai from the future: If a luddite like me does that with my writing product, I would hope that the federal IT people would have done that regularly.
Belafon
@Tim C.: I agree with you, but where I also am is that they’re going to have to go through confession now before we can move forward. They have to admit they messed up and why they did it.
Belafon
@stinger: Musk isn’t allowed anywhere near actual SpaceX rocket creation.
A Ghost to Most
“Tell em about the Twinkie.”
ArchTeryx
@TBone: If that situation didn’t sound so deadly serious it would be hilarious. “Whiskey Tango Foxtrot, you a**holes!” sounds about right.
ArchTeryx
@A Ghost to Most: “WHAT ABOUT THE TWINKIE?!”
Gin & Tonic
I am also retired from decades in enterprise-level computing, and agree with all of the other posters here. “Bad” does not begin to describe what is going on here.
What surprises me, at least a bit, is what college-age kid knows anything at all about COBOL?
Steve LaBonne
@stinger: That would explain all those unscheduled rapid disassemblies, right?
WereBear
@sixthdoctor: Already had the nightmare and working on the novella, myself.
scav
@Gin & Tonic: What makes you think a functioning system is the end-goal here? Actual knowledge and COBOL experience might only get in the way. Why learn how to land the plane?
Steve LaBonne
@Gin & Tonic: That was going to be my question too. I’m not sure the civil servants who are helping this kid in the hope of averting a meltdown are actually doing us a favor.
schrodingers_cat
@Gin & Tonic: I don’t need to know COBOL if I just want to destroy the program.
WereBear
And of long duration.
MobiusKlein
On the “What is the worst that could happen” front – (big company) had an acquisition that turned out to be p0wned by two or three nation state actors – and not just the big ones like USA/China, but Jordan?
They shut it down, and wrote off the entire purchase as not worth the $ to fix it.
Imagine the US Treasury being compromised like that. The worst that could happen is >>> reverting to prior version and restoring from backup.
cmorenc
@Paul W.:
What will John Roberts do when Trump & Elon thumb their noses at SCOTUS rulings and injunctions? Roberts’ “Federalist” vision for the federal government is essentially to entrench a business/commerce-friendly structure facilitated by pared-down regulatory power and secondarily, to facilitate return of more old-fashioned conservative social norms. But not really to be the enabler of authoritarian neo-nazi rule, despite the presidential immunity ruling. Roberts is like the many chamber-of-commerce / big financial / corporate GOP wing in that they didn’t really believe Trump would follow through on his campaign rhetoric, but would actually govern as a business Republican. They still labor under the delusion that they can control the Trumkenstein monster they have enabled.
ArchTeryx
@schrodingers_cat: But you do if you want to specifically halt payment streams. This lot of young putsch pushers probably looked at some YouTube tutorials and figured that made them experts in COBOL. THey could bring our entire Treasury to its knees accidentally, which is what makes this growing disaster so scary.
kindness
I learned Basic, Fortran and Cobol my first semester in 1982. The Cobol was actually done with a deck of cards fed into the computer to compile the code each card had. Yea, I’m old. I have a hard time believing that current data structures and financial tracking software the US uses are in any of those languages
@cmorenc: You are being too kind to Roberts. Roberts had to know Trump would go this way and Roberts gave him a king’s powers anyhow. We’re going to need new circles of Hell for these people.
schrodingers_cat
@ArchTeryx: This has been scary since he was reelected.
Steve LaBonne
@schrodingers_cat: Musk doesn’t want the massive payments to SpaceX to stop. They may break the system out of stupidity, but Musk wants control, not a complete shutdown.
Kelly
@Gin & Tonic: I’ve also been puzzling over what college age kid knows COBOL. My best guess is there’s a layer of modern code over the legacy code and he’s messing with that layer.
Lobo
@TBone: My new way to curse.
Gin & Tonic
@kindness:
If you have not spent a career in IT you have no idea how much COBOL code is actually running, today.
Steve LaBonne
@cmorenc: That is the eternal story of how fascist regimes come to power. They always need the initial help of conservatives who are soon discarded when no longer useful.
sentient ai from the future
@schrodingers_cat: given the reliance on “AI” (one of the henchteens was previously at musk’s “xAI”) it seems pretty likely that the security services vlad employs have made use of the KNOWN, PUBLISHED VULNERABILITIES OF THOSE PROCESSES that do things like allow a remote-code exploit to be embedded in the resulting models.
i’d bet a shiny new nickel that they’re using “AI” to help them understand the code. things are just that badstupid.
WereBear
@Gin & Tonic: Systems are also evolving organisms.
ArchTeryx
@schrodingers_cat: That goes without saying. Do I think this is a fun roller coaster ride? At this point, the completely vile fictional villain, Lord Suzocon the eight-tailed kitsune, from one of the few webcomics I follow? I’d rather be stuck with HIM than the current crew. He at least is bloody competent and doesn’t randomly break shit.
Yes, being a writer, I support other writers and artists. And sometimes, even I need a break from the ongoing catastrophe that our country is rapidly descending into.
Quinerly
The Independent has picked up the Wired reporting with some additions.
“According to Elez’s social media profiles – including his now-removed LinkedIn and a Twitter page reviewed by The Independent – he graduated from New Jersey’s Rutgers University in 2021 and subsequently worked at SpaceX as a software engineer. Musk’s tech prodigy then joined X where he worked on search AI.”
And he was a teen “soccer star.”
https://www.the-independent.com/news/world/americas/us-politics/elon-musk-marko-elez-treasury-doge-b2691932.html
TBone
@ArchTeryx: he wielded that with aplomb for the rest of his life! And it was, frequently, hilarious. Alternating with “strike massive fear in your heart immediately.” It could make people STFU instantly in either situation!
He had a way of emphasizing the “fuck” so it carried.
Quinerly
Truthout has picked it up in its latest story on the federal employee unions suing.
https://truthout.org/articles/unions-sue-to-halt-musk-teams-access-to-critical-treasury-payment-system/
Glory b
@kindness: I know right?
Folks here simultaneously want Dems to win but trash them at every turn?
Talking about messaging, has no one ever heard of message discipline?
snoey
@Kelly: Often more than 1 layer, and that bit of COBOL may be on top of some even more ancient assembler language
ChrisSherbak
Not sure I can add much to all this – but I have 30+ years in IT and it’s … butt clenchingly scary to think that this is going on. FWIW in much of the non-serious IT world, “break it and fix forward” might be ok but coming from 23 years of banking and another 30 in non-banking financial I was shocked at the level of … lack of care in change control for source code at the non-bank shop. It has gotten better over time, but I’m sure the amount of discipline with barely out of their teens IT people is miniscule. Years of sweat and tears makes one very very cautious with changes and all that “annoying” waterfall/change control “stuff” saves your bacon on more than one occasion. We dodged a bullet at Y2K, guess we’re going to have that crisis 25 years later.
cmorenc
@kindness:
I am willing to bet that much of this legacy code is not written or structured in a well-designed modular or intelligibly documented. Lots of it is likely the cobol or fortran version of a “goto” spaghetti maze.
Quinerly
This is an excellent site to follow in general on what’s happening at Treasury. No paywall.
https://www.crisesnotes.com/day-five-of-the-trump-musk-treasury-payments-crisis-of-2025-not-read-only-access-anymore/?ref=notes-on-the-crises-newsletter
Steve LaBonne
@Glory b: Republicans could end this shit today if they wanted to. But we no they won’t so we focus our rage on our own side. It’s understandable but dysfunctional.
TBone
Bucknell is now testing its emergency loudspeaker and siren system. Not a sound I needed to hear today but…
They’re on alert. On the job!
gene108
@Tim C.:
The information bubble means they will not know or accept who the real culprits are for causing the problems we have.
If things go to shit the people that run their information bubble will start targeting minority groups as the reason. They’ll buy in.
The white people who’ve been wanting a race war for the longest time will have a reason to let loose.
JoyceH
What I’m wondering – with the permissions they have in the system they’re on, would anything stop them from scraping out a billion or two or three or ten for themselves?
TBone
@gene108: so will I.
sentient ai from the future
@Quinerly: just called that firm, they were polite but eager to get me off the phone.
there have to be avenues to respond preemptively. if the data leaks, then it’s out there forever.
Kay
@Quinerly:
Interesting. Thank you. Retirees are suing. That’s smart.
Captain C
@Belafon: Maybe drop him over a wilderness refuge, so at least when he doesn’t figure out the parachute some beasties will have a lunch (though they may just decide he tastes too awful).
Captain C
@Belafon: And some atonement. Make real-world amends for what they did/voted for to the victims.
No One of Consequence
@sixthdoctor: Unlikely. Access to the data itself is bad enough, but the workers who facilitated the system would have enormous incentive and training to NEVER copy any data to anything like a thumbdrive, etc.
Worked around software development for a few decades, though I am not a developer. Since the Health Information act stuff, Personally Identifiable Information carries significant legal risk to have and access.
This assumes, of course, that our federal government has significant control mechanisms in place for errant personalities accessing things they should not, or taking data actions they should not.
We’ll be lucky to make it to the meteor…
Twice.
Somehow, my fellow citizens managed to elect this fuggin’ orange imbecile TWICE.
We’ve earned it, for sure, but damnit if it doesn’t grate. Stoopid fucking Americans. Good and hard, indeed.
-NOoC
Belafon
@Kelly: You can download and learn OpenCobol if you want.
Captain C
@cmorenc:
Cry, and pull a FTFNYT “Who could’ve known?” bullshit. Probably on the FTFNYT’s editorial page.
Quinerly
@sentient ai from the future:
I’ll keep poking around for you. Kay might have some thoughts too.
Hang in there.
sentient ai from the future
@Steve LaBonne: that. is. the. behavior. of. abusers. and. their. enablers.
*tears up picture of the pope
Old Man Shadow
Yes, pushing code to your live PROD environment without ample QA testing is insanity that only sounds good to someone with a narcissistic personality disorder who is convinced they can do no wrong.
Shit is going to explode. People are going to die.
Not to mention they are giving a private citizens complete control over the payment systems of our government. No oversight. No accountability. No checks and balances. Just one guy now who can be ordered to stop payment that Congress has authorized. One guy who can cut off doctors if they get too uppity. One guy who can cut off the political and business enemies of Trump or Musk without due process or an avenue for redress of grievances.
We have a king.
sentient ai from the future
@Quinerly: i dont actually think i am under fire here directly, i’m just trying to figure out of my situation, like that of my parent-to-a-trans-kid status, will get me standing on something that SOMEONE can do something with to delay or stop this bullshit.
Quinerly
@Kay: you’re welcome. I think I read there have been around 25 suits filed by various groups/people with standing since the first EOs on day one. That # seems low to me.
Kelly
@Belafon: Don’t want. Had all the 2 am the system down calls I’ll ever need
gene108
@sentient ai from the future:
It’s happened before. U.S. military personnel in either Iraq or Afghanistan bought USB drives from the local market, which were infected with malware. A foreign country flooded the local market with a bunch of infected USB drives, with the hope an American would buy one and plug it into a system connected to more important systems.
https://en.m.wikipedia.org/wiki/2008_malware_infection_of_the_United_States_Department_of_Defense
Spanky
@@mistermix.bsky.social:
You don’t think these fuckers are going to trash that too?
The purpose here is to irrevocably break shit.
oklahomo
@cmorenc: Or the dreaded, unholy ALTER.
comrade scotts agenda of rage
Somebody here said this last week:
The Trumpanzees are ripping out the wiring of the federal government.
sentient ai from the future
@gene108: the hothouse flowers of musk and thiel’s PayPal Putsch cannot possibly fathom the difference between what “security” means in their lives of comfort to this point, and what it means when you are messing with bedrock structures of a nation state.
so i assume it’s already happened, and now we’re just counting down to the blast so that we can then assess the damage. careful of double-taps.
Sister Golden Bear
@Gin & Tonic:
They probably don’t. But as others have said, it’s far less about knowing the programming language and far more about mucking around in a system they can’t possibly understand (as I’m sure you know). Not just because they’ve at best had a few days to analyze it, but because 1) these systems are inherently huge and complex, and 2) any system built up over decades accumulates huge amounts of business logic that has to figured out before you should even think about touching it.
I only dabble in programming, but for my jobby-job I work closely with programming teams. One of big career projects was modernizing an e-commerce site’s shopping cart system, built on 15-year-old start up code. There were parts of it we simply couldn’t change because no one understood how they worked. And the rollout, while fast compared to government/financial/healthcare institutions was definitely cautious. Because ya know, fucking up could cost the company literally millions of dollars per day.
Sister Golden Bear
@kindness: There was press release last year from one of the government agencies last year proudly announcing about how they’d finally moved a system off Assembly to one of the modern versions of COBOL. So yeah… there’s plenty of government systems still running it.
Baud
Never should have stopped using vacuum tubes.
Sister Golden Bear
@JoyceH:
Feature, not a bug as far Musk and his IT’ler team are concerned.
CaseyL
@JoyceH:
No. In fact, I expect that to happen. I have visions of Trump announcing to the world that he is now the world’s richest man, thanks to the US Treasury; and then Musk raiding it more so that he’s back on top; and so on.
Musk said he was targeting $2 trillion – that’s trillion with a T – so I figure that’s what they’re gonna take.
I also anticipate a lot of GOP politicians getting the odd $100m or so apiece to keep them happy and in line.
Kelly
Some of the things to understand a simple. Back in the day I’d explain several times a year to different accountants that had run some DB queries that the reason Payroll, AP, etc didn’t match the GL right now because updates were still running.
Another Scott
I’m not an IT person, but I know enough to know how much I don’t know.
I’m reminded of a guy who had an early on-line software shop and was the owner of one of the very early on-line OS/2 2.0 software stores. He decided to go all-in on upgrading his stuff to IBM and Lotus on the promise that it would be faster and more efficient and better service for his customers. (You know what’s coming…)
It was a disaster for him. The few times I tried using his website afterwards it was ungodly slow, timed out, etc. His business didn’t survive.
And that’s with “professionals” doing the work.
Big changes to software are always, always, extremely risky. Untested changes to software are always, always, extremely risky. Letting outsiders jump in and quickly change things in working systems is always, always, extremely risky.
If things are happening as reported, it’s obviously going to be a disaster. Of course they’ll spin it as some great success, it’s what they do, but …
Grr…
Best wishes,
Scott.
H-Bob
@JoyceH: “a billion or two or three or ten” — You’re thinking waaaay to small!
Jaybird
@Gin & Tonic: You’d be amazed by the amount of COBOL still underlying things. I retired 16 years ago, but I understand from family still in the field that that’s changed very little. One of the initial “goals” of the Y2K team I led was to “eliminate” all the COBOL in the large financial institution’s systems. After about 1 week of analysis we had to report back that that was completely unrealistic. We did at least *identify* most of it. Some of it is no doubt still there 25 years later, chugging along.
Professor Bigfoot
@Dougb: No, we’re talking about white people.
White people elected him, white people support him.
”There is no horseshoe. There is only white people who are at best uncomfortable with any power being held in Black hands. Those white people are at all points of the ‘left-right’ spectrum.”
tobie
I’m going to go out on a limb here and say that Musk and his pubescent henchmen didn’t rewrite the code and hit PROD. They’re talking big so they can say, “See, 6 smart guys were able to change the entire US Treasury’s payment system and it was so easy that there was no disruption of service.” Then they’ll call to privatize everything. They’re salespeople. They’re good at self-hyping and that’s about it.
RSA
A few computer science and information science departments specialize in legacy systems and languages. They know that there’s continuing demand, because these systems are still running and need to be maintained and sometimes expanded, even as the generation that put them together is well into retirement. (It’s also a nice reminder that a solid undergrad education is possible, if maybe a bit specialized, even based on obsolete technology.)
LeftCoastYankee
This is bad, at a minimum completely amateur and arrogant, and depending on the architecture of the systems in question it could be catastrophic.
Theres likely a great deal of audit trail capture if they’re using modern development, data and hosting tools. The catch is more than any other sector government agencies have proprietary functional needs and therefore frequently have proprietary (ie designed specifically for them) systems.
On the plus side federal systems are more likely to be properly funded and managed than say state or local government.
TLDR: degree of badness depends on details we don’t have yet. But insanely unprofessional.
comrade scotts agenda of rage
@LeftCoastYankee:
Not necessarily. I worked for 27+ years with state DOT IT departments as part of my job within DOT. On balance, I’d say they were far better managed than anything we were doing within DOT. My experience with every state DOT IT people I worked with was always good.
I can’t speak to funding levels other than to say that ours were always looking to be cut, all the while trying to outsource everything…which increased costs significantly.
Larch
I’m a retired business systems analyst who worked ~25 years in IT at a Fortune 500 financial services company. It’s not just that they don’t know COBOL — they don’t have the experience or are they taking the time to understand how kludgy legacy mainframe systems are. I started out working on one of those – it had been built onto, tweaked, enhanced, etc, etc, etc, until it was a spaghetti-like mass of inter-twined and inter-dependent libraries, parameters, processes and programs. And that was _just_ for a 401k record keeping system. I can’t even fathom the mess that the entire Treasury must be.
I also worked on the project to move our system from a mainframe to a distributed system. That took years and included one massive systems crash and multiple near-misses before we got the thing stabilized. If I had a nickel for every bug I found along the way, I’d be insanely wealthy right now.
What these idiots are doing WILL crash systems and introduce major and minor bugs that will take years, if not decades to root out.
Dorothy A. Winsor
@Professor Bigfoot: Good reminder.
zhena gogolia
@Glory b: Flashing back to June 2024.
Another Scott
@kindness:
One thing to remember about the US Government is that there’s a gigantic infrastructure behind everything, and comparatively little has been spent over time on modernizing it. And when a decision is made to modernize it, it costs a mountain of money and takes years or decades.
IRS.gov (150 page .pdf from FY23):
That’s just one agency. It’s a 5 year plan that was funded at $4.8B via Biden’s Inflation Reduction Act.
And of course, Donnie and his GQP minions want to destroy the IRA and the IRS so don’t count on things going as outlined in that plan…
Grr…
HTH!
Best wishes,
Scott.
Lynn Dee
It is unheard of — particularly in government, but not just — to put even minor code changes into production without non-live testing. Just gob-smacking.
LeftCoastYankee
@LeftCoastYankee: Updated after reading the linked newsletter
ignore my prior post. This is a clusterfuck.
polyorchnid octopunch
It’s been a long time since I’ve been here. I am an IT person; my most prominent gig has been the dozen years I spent running a major Canadian telco’s mail/web to text gateway, which basically meant I was running a system that mediated email or webchat to SMS text messaging for app. 10 million people.
We considered ourselves cowboys wrt our willingness to do shit live, esp. wrt things like the underlying storage architecture and so on. There is absolutely NO WAY IN HELL we would have ever done anything close to that kind of shit.
Given the incredible importance of that system, these people have clearly never understood anything below the cloud layer and they all think that all computers function like cloud systems do, and when they fuck up and say “well roll that back” and then discover they can’t it’s going to break serious shit.
Puts the lie to the “tech” part of the “tech bro” moniker; they clearly have no idea of what they’re fucking with. They’re going to make “only a minor change” and then all of a sudden something will break and prevent any kind of mortgage being able to be completed, or perhaps not being able to pay anyone that works in a civilian capacity for the USN, or something like that.
Where I’ve been working lately (dev shop/distributed compute service https://distributive.network) we expect devs to need at minimum three months before they’re going to be allowed to submit merge requests unsupervised because it just takes time and a fair chunk of reading to get up to speed on a code base; you can’t just parachute in and start doing live changes and not expect to fuck things up
I really can’t emphasize enough how completely insane this is. These people should end up facing the full force of the state via the law for doing this, even if they manage to finesse their way through without sledging the fine china.
Glory b
@Quinerly: Some are likely filed as class actions, maybe hundreds of plaintiffs.
Dougb
@kindness: The problem is if they’re never acting, when they do have power, to slam Republicans with the full force of the law then they’re aiding and abetting them in their future law breaking. The lack of action by Merrick Garland was absolutely unforgiveable and he was Biden’s appointee. Similarly in 2009, when Obama assumed office and could have acted against both the lawlessness of the W administration officials they did nothing. And that’s not even counting his lack of action against the titans of the financial industry who just about destroyed the economy.
If they don’t take action against these things when they have the power, and continue to falter in acting in ways that could at least slow things when they don’t have power, then why shouldn’t the Democratic Party be criticized. If they act in ways that show that they aren’t fit for purpose the only conclusion then is that they aren’t fit for purpose. And that argues for either a replacement of the party or a tea party like campaign to force them to act in appropriate ways or replace them with members that will.
polyorchnid octopunch
Wait, Treasury? Gonna be great when all of a sudden they completely croak the ability for the United States Government to honour T-bills.
Jesus Christ.
This could crash the world economy super super hard. These people are fucked in the head.
jefft452
Is it bad?
Depends on how good a job you think the kids running the coalition provisional authority did
Sister Golden Bear
@tobie: Per Talking Points Memo changes have already been deployed on Prod, although the fed staff did persuade the Elon goon to do so limitedly (for now).
Josie
@Professor Bigfoot:
This is something that needs to be pointed out every time someone brings up the horseshoe theory. It is glaringly obvious, even though some super lefties don’t want to admit to it.
A Ghost to Most
This is “I’m glad I’m not in charge of government databases anymore” bad.
Rudi666
@stinger: Thats what Elmo did with Twitter. He didn’t write any code, but forced understaffed coders to make changes. How much did Twitter suck after his interference after Elmo bought Twitter.
LeftCoastYankee
@comrade scotts agenda of rage:
I agree with the IT department people being good and understanding the problems. I contracted for a DOT department for a few years, but a while back.
The “Management” deficiencies I referred to were around tackling business ownership (and funding), particularly in addressing older and outdated systems. There was a number of key systems anchored on outdated technology which expanded the overall costs and work for IT, while pretending IT was overhead that could be cut rather than part of operations.
Yikes I just had a teeth grinding flashback to those days!
polyorchnid octopunch
@@mistermix.bsky.social: The problem here is that being able to roll back the code changes won’t do any bit of good if their code change trashed the data sources that the code manipulates and uses to do its job.
Steve LaBonne
@Professor Bigfoot: 🎯
WaitingForMountainLife
@stinger: No clue – I never worked for a Musk property thank god. I certainly hope not.
Professor Bigfoot
@Keith P.: Former “real-time” developer here; and the most incompetent computer engineering outfit I ever worked for was exactly that— a bunch of “code cowboys” whose FIRST move was always to dive into the code.
There was no design. There was no documentation.
They were “programmers,” not software engineers. They had absolutely NO discipline; no processes, no solid testing… when a major automotive company with your equipment in all their assembly plants take some of that equipment and give it to one of your competitors to be able to actually upgrade them…
So if that’s what the dipshit’s minions are, well…
Philbert
@Kelly: retired after 35 years COBOL Aessembler here. Yes, most big systems have a COBOL core, middleware, and a Web front end. Knowing the language is barely the start. There are hundreds of systems and databases and database types interacting, most of which have been patched with varying skill for 30 years. ‘A little knowledge’.
IThe remaining dedicated experts I’m sure are furiously preparing reload-and-recover. Bless those backups!
NO meteor needed, a modest Carrington Event would do …
Marc
Well, if I were a super-villain and was given the keys to the Treasury systems, I’d convince (at threat of firing, if not prosecution) some of the key employees to identify the critical systems through which all payments pass. Next, ask the employees to install hooks in those systems. Then I would have my AI folks bring in their own servers, run the data streams through an unsupervised learning model, and and categorize all of the payment streams while matching them against contracts and grants picked up from other departments (using similar hooks and systems), then simply cut off (using the existing systems) payments to departments, corporations, organizations, and people I don’t like. That money that is not being paid out builds a surplus in the Treasury, some of which will be used as an excuse (“Efficiency!”) for the government to cut taxes on myself and my super-villain buddies, then divert the rest into cryptocurrency “investments”.
Citizen Alan
@gene108: As a practical matter, what is the difference between someone who wants destroy America out of spite and cruelty and someone who supports policies that will destroy America because the information bubble that they actively choose to cultivate and maintain reassure them that those policies will lead to sunshine and roses. As I have been saying for years, Donald Trump is not a master hypnotist and Fox News is not beaming mind control waves into people’s heads. Both of them have traction because of the large number of people who actively seek out liars who will tell them lies that comport with their own bigotries and anxieties.
Off topic: I have been considering moving in with a friend to cut my monthly rent in half. It’s a very nice four BR house with a pool that’s ten minutes closer to work. But he has just (last night) informed me that he will likely be renting a room to a gay couple who are also friends and who are in our mutual gaming group. And I am suddenly very ambivalent because one of the guys (who I like) is, I suspect, a closet Republican, no pun intended. He’s gay but he’s also from a somewhat rich family that works in the financial services area, he’s a gun-humper, and he’s expressed admiration for Elon Musk. And I just don’t know if I can live with that. We all agree not to discuss politics at all on D&D night and most of the group is pretty leftwing. But I don’t know if I have it in me to keep my mouth shut about the state of American politics 24/7, not even for a $1200 reduction in rent.
NeenerNeener
I was in IT for 30+ years and spent the last 10 of them doing system testing. There’s no way in hell we would be allowed to push code straight to Prod from DEV, even doing so-called “Agile” coding. If that’s what they do when they work for Emperor Elon no wonder his cars and rockets explode.
The good news:
I’m reasonably sure that there are multiple off-site backups of code and data stored at Iron Mountain and possibly other places.
The bad news:
A lot of those stored tapes are trash because of errors during creation, but you won’t find that out until you mount the tape and try to read it in an emergency.
Alce _e_ardillo
@WaitingForMountainLife: What do they care? It’s not their money,right?😱
kindness
@Dougb: With all due respect there are Democrats out there right now doing great work. I take issue with what you said because you made it sound as if Democrats are doing nothing, and that just isn’t true. Circular firing squads kinda suck is my feeling.
Kelly
From what I’ve read the young Muskrat is installing a backdoor to the Treasury’s systems. I’m sure no malicious actors will be able to exploit code some twerp banged out over weekend.
TBone
@Lobo: GOOD
TBone
@NeenerNeener: greaaaaaat
It just keeps getting better, innit?
bbleh
@Gin & Tonic: @scav: @ArchTeryx: @Kelly: yabbut said 25-y/o is a Certified Genius. Also Move Fast And Break Things, something something agile something disruption something.
And definitely concur re the risk of spyware, and not just from, say, China but from Musk’s own organizations and very much on purpose.
Even in the (to my mind) unlikely event something doesn’t go Very Badly Wrong, undoing what they’ve done will be a Herculean job, not least because I can’t imagine them following any kind of a plan or documenting what they do, and given how they’ve behaved at OPM, I gotta wonder whether the legacy folks will be able to prepare properly for disaster recovery.
And meanwhile, the Senile Orange Guy will intone something about “cutting out gummint waste,” and scores of millions of ‘Murkins will nod their heads…
Ben Cisco
Sysadmin here.
This is catastrophic.
Not a matter of IF, but rather when.
TONYG
I’m an Old Retired Bastard now, but I had a “career” in I.T. for about 40 years, include 26 years providing mainframe operating system support (hands-on and as a supervisor) at a New York City cancer hospital. Yes; this clusterfuck could not have happened without the regular computer system staff helping Elon Musk’s Teenage Keyboard Kommandos. (You can’t just “break in” to a computer system the way you can break into a house of liquor store.). My question is why these thugs were helped by anyone. Was it cowardice? Complicity? Maybe we’ll learn someday or maybe we won’t. I don’t buy the argument that “these bad people might crash the system”. I worked for a cancer hospital (Sloan-Kettering). When a system crashes the clinicians can do manual workarounds. But a CORRUPTED SYSTEM is much more dangerous, because the data might not be obviously corrupt. Elon and his posse of entitled assholes should have all been in jail (with no bail) since Friday night. And anyone who assisted these criminals should have already been detained for questioning to see whether they have a reasonable alibi for having done so.
Steve LaBonne
@TONYG: Detained by whom? Trump is the boss of everyone who could do any detaining. They’re actually talking about prosecuting anyone who tries to get in the way of the Muskrats. We’re all the way into crazytown.
artem1s
Social security payments/EFT’s occur on Wednesdays starting on the second week of the month. If they broke it, that’s when we’ll find out.
Steve LaBonne
@artem1s: If they fuck up interest payments on Treasuries we will find out sooner than that and very much the hard way.
TONYG
@TONYG: On a human level, though, I can understand why the regular I.T. people might have helped the Elon Elite commit their crimes. I.T. people, with all due respect, are the ultimate nerds. Their bosses tell them to do XYZ and, by gum, they do XYZ. Like assembly line workers, they do detail work that their bosses tell them to do, under the assumption that their bosses are competent and well-intentioned. When Elon’s A-Team blundered into their offices on Friday, I’m sure that they announced that they were the new bosses that were now in charge — so, of course, the I.T. people “did their jobs” and helped those new bosses. They weren’t being ordered to “steal money from widows and orphans”. They were just told to modify a few system access parameters. Nazi Germany knew how compartmentalization aded genocide. Elon and his crew knows that too.
TONYG
@Steve LaBonne: Yes, I know. I was being a little bit sarcastic there. They SHOULD HAVE BEEN and WOULD HAVE BEEN arrested in the before-times when we had a functioning government. Now we’re ruled by fascist criminals.
TONYG
@TONYG: Of course, once Hitler was in power he wiped out the undisciplined SA in favor of the “more disciplined” SS. This time around it might be the forces of the Richest Man in the World (Elon) wiping out Trump and his impolite MAGA hordes. Interesting times, but I probably won’t survive them.
patrick II
Does anyone know the goal(s) of these changes? What new functionality is being added? This seems like more than patching errors. That would take more time to understand the system. What is their goal? What new functionality is being added?
Aziz, light!
@Kelly: This makes sense. They can’t be stupid enough to interdict the next round of SS, Medicare, and other payments (although they might break the system inadvertently). Most likely Musk is setting up channels for future monkey business in ways that won’t immediately trigger a response.
The sound of crickets from the House is worrisome.
tobie
@Sister Golden Bear: Thanks for clarifying. I was wrong. It’s so hard to tell what the techbros do given their self-regard and swagger. They love to shoot from the hip. They also love to praise themselves to the sky. I try not to feed their narcissism.
bbleh
@patrick II: indeed, and in whose interest is functionality being added (or data scraped)?
Personally, I suspect anything that’s being done is AT LEAST as much in the interest of Elon, Inc., as in anyone or anything else, the Senile Orange Guy very much included, and in the general welfare of the American people pretty much not at all.
Professor Bigfoot
@TONYG: Every damn day there’s another brick in that concentration camp wall, isn’t it?
Folks simply do not want to accept that they are Nazis (not NAZIs, as they’re not members of the NSDAP but they’re Nazis nonetheless) and it’s been obvious since they waved their “MASS DEPORTATION NOW” signs at their convention.
They’re Nazis, and they are driving us directly down that same road.
Kayla Rudbek
@gene108: there was an air gapped computer at the JAG in South Korea that got infected with the Melissa virus when it was going around
Kayla Rudbek
@polyorchnid octopunch: yeah, I’m considering going out and buying gold instead of yarn
Bill Arnold
Frankly,
We have no evidence that he/they have not backdoored these systems, with theft and/or other malice intended. They have admin access(es).
We have no evidence that he/they are not compromised by foreign intelligence agencies.
We have no evidence that he/they have taken the oath of office. (To defend the Constitution etc.) Or had their background(s) checked.
There should be 10 page dossiers on each and every one of them, including browser history, porn habits, any criminal records, movements for the last 5 years, all online social media activity, email records, phone records, bank records, investigation of families. If e.g. any of them have Russian or Chinese nationals among their close relations, actions must be taken. IMO.
Note: much of this can be done with open source methods.
Kelly
@Kayla Rudbek: Gold is hard to sell and easy to steal. Stockpile mundane, tradeable survival supplies. Canning jars and lids, water purification filters, hand tools maybe camping stuff ;-)
karensky
@Quinerly: just read the first 6 paragraphs of the text. Thank you for posting this link. I will keep reading this site.
Bill Arnold
@sentient ai from the future:
That’s stunt attacking. Unlikely ATM IMO, i.e. this month.
I’d worry more that the DOGE people involved are deliberately installing more traditional backdoors, and hidden vulns.
Also, suspect that their main goal is effectively an API that Musk can use via a console that they make for him, that allows fine grained control over government payments to his enemies, and uhm, payments and “payments” to his friends and himself.
azlib
I was an IT system and network professional for 49 years. I retired in 2021. This is catastrophic. There is no way anybody with a week of exposure to a complex and probably fragile system like this with a lot of legacy code can properly understand the system much less modify it basically on the fly. At least when I do not get my SS check, I will know why it happened.
neldob
I don’t think anyone should “help” them. Let them take the heat when it fails to deliver.
Kayla Rudbek
@Kelly: we have camping supplies already, iodine tablets might be a good idea as well as the canning supplies and vegetable seeds
MomSense
@Quinerly:
I can’t wrap my head around this. Did they use force? Why the hell would treasury give them access? It’s completely illegal.
Emily B.
@TONYG: You know the joke about the engineer and the guillotine?
bluefoot
@Professor Bigfoot: Thank you for continuing to highlight this.
TBone
@Kelly: and seeds!
ETA Kayla you beat me to it
Ebony
Does anyone remember last year’s Crowdstrike outage? This make the Crowdstrike outage seem like small potatoes.
TBone
@Ebony: ahhh yes…though vaguely because information overload causes me to offload some stuff so new can fit.
TBone
@Emily B.: snort! New to me, thank you.
Hunter Gathers
Former Systems Admin
These fucks have not had enough time to read and understand the documentation, let alone the coding itself.
They can’t roll this shit back and I doubt the Ketamine Addled Afrikaner understands that.
A Ghost to Most
As a DBA, I would refuse to work with a tainted system. As a programmer, I would be poring over the changes made.
TONYG
@A Ghost to Most: Yes; that’s what a good I.T. person would do. Not everyone is that good though.
TONYG
@A Ghost to Most: Most likely “someone” just gave these thugs signons with absolutely authority, and the Elon-teens then did the rest themselves, cutting the real I.T. staff out of the loop. Change-control procedures only work if the people making the changes don’t have absolute authority.
TONYG
@Ebony: Yes, that fuckup. My boss (a woman about ten years older than me) would have literally crucified me if I had ever done something like that.
Quinerly
Leaving this here. Entire piece is quite the read. My new go to site.
Musk and his cronies are clearly aiming to redesign the payments system to serve their agenda. The most chilling sentence is this one from the Wall Street Journal:
TONYG
@Emily B.: I haven’t heard that one. Good joke. If you saw “Schlndler’s List” years ago you might remember a not-funny version of that story. A young Jewish woman prisoner, who has been trained as an engineer, tells the commandant that the building foundations are incorrectly designed. The commandant agrees, and issues an order to the slave-laborers accordingly. Then he orders the woman to be shot for insubordination.
Quinerly
@MomSense:
Lots of info here.
https://www.crisesnotes.com/elon-musk-wants-to-get-operational-control-of-the-treasurys-payment-system-this-could-not-possibly-be-more-dangerous/
TONYG
@TONYG: Of course, in the Elon/Trump version of that story the woman would have been shot without the foundations being fixed.
TONYG
@MomSense: An inside job. That’s why in the before-times, when we had rule of law, there would have been arrests and an investigation by Friday evening.
TONYG
@azlib: Literally kids “led” by a drug-addles psychopath. I doubt that even their theft-software will work.
TONYG
@Bill Arnold: Yes; I’m sure that that’s Elon’s intent. But, like Trump, Elon has a decades-long history as a fuckup. This whole thing might turn out like one of Wile E. Coyote’s schemes.
TONYG
@patrick II: Every fucking place where I worked in forty years has had strict change-control procedures Not a single byte of code or a single parameter is changed on a production system unless “the appropriate people” have signed off on the change in the test system. That is standard practice everywhere, most likely since long before my first job in the Disco Era of 1977. This is sabotage and absolute insanity.
Quinerly
@TONYG:
Musk now has all the personal and financial information of all his rivals.
And Bezos and Zuckerberg’s
AND TRUMP’S INFORMATION
TONYG
@Hunter Gathers: Reminds me of a “contracting assignment” that I had at a manufacturing company a few years ago. I knew nothing about the company or its workflows. I asked the manager for a copy of the documentation. “Sorry; there is no documentation. Feel free to look at the code (hundreds of programs) though.”. I asked whether I could talk to any experienced programmers to clarify things. “Sorry; all of the experienced programmers were laid off last month.”. Well, OK then.
TONYG
@TONYG: I had several “contracting assignments” because I.T. managers had been too dumb to realize that firing experienced staff had a downside.
BellyCat
But… but, ho will work with these three people?
Ryan
Remember those stories about the Russians holding the Ukrainian workers at the Zaporizhzhia nuclear power plant so they could keep everything going? This… kind of sounds like that.
Gretchen
@TONYG: That’s what I wondered. Who gave the Muskrats the passwords to get into the computer systems in the first place? Even if they couldn’t bar the doors, they didn’t have to hand over passwords to someone without security clearances. In fact doing so is a prosecutable offense itself unless they were, say, threatened at gunpoint.
Ruckus
@brendancalling:
I am well into my senior years and my living needs are met by SS.
If they fuck up Social Security, a lot of citizens, and I’m one of them, are going to be just a tad upset. In 2023 just under 60 million people received SS benefits. It might be over
nowor soon. Now I’d bet that some of these people supported shitforbrains. I wonder if they will see the concept of their action having a rather strong effect upon their very lives.Ruckus
@Paul W.:
It’s not their feet they are aiming at. It’s either crotch or brain that they are aiming at and they seem stupid enough to aim for the brain. (Neither are good areas to hit but hitting the brain does more damage – possibly a hell of a lot more. OK in their case it might not cause any damage at all. Because they really aren’t using said brains for any rational thought whatsoever. IOW it might be helpful. For everyone else)
And anyone that says this is bad is likely not actually saying enough. It is far worse than bad. This is pure shit walking.
JaneE
Please dear God let them have a good version management system that can back out bad updates quickly. And I pray that someone had the sense to take an extra source backup before the boy genius got his mitts on it.
We learned those lessons more than half a century ago, but I don’t have any confidence that Elon et.al. will follow best practices. It takes extra time you see.
SH
@TONYG:
That is so true and sadly funny
‘Uh What documentation?’
SH
Ex SAS contractor