In a completely predictable series of events, immediately after rattling the cup to defray server costs last night, the website immediately began shitting the bed and we had probably a dozen short outages over the last 20 hours. Rest assured, we are working on a solution.
It’s not the cloudflare configuration or not enough space or anything like this, and nothing that throwing money at can fix. The problem is basically (and will always be) this- WordPress was not designed to have an archive of eleventy billion comments, so some of the coding messes with this website but leaves virtually every other WP website untouched. You’ve heard, no doubt, of the different meanings of the phrase “the exception that proves the rule,” well, we are ALWAYS going to be the exception.
In this case, there is a query that involves going through every single comment (again, for the god damned recent comments thing that you all love so much and I have NEVER used), and it just blows up the site. Here’s a picture:
Basically, that query needs to be rewritten, but it can not be rewritten solely for this website, it is going to need to involve the WP developers, and it has to be done in a way that won’t jack up the millions of other websites that use WP but DON’T have our unique situation. So we are working on it. Basically, the recent comments may disappear temporarily so we can confirm that that is in fact the case.
To continue the grotesque metaphor, we have gone through five sets of sheets and are now putting a plastic cover on the mattress and handing out adult diapers until we can get some immodium or get to a doctor.
The End.
zhena gogolia
FWIW, I’ve never used “recent comments.” (I know I’m only one person!)
chopper
@zhena gogolia:
neither have i, but GODDAMNIT WE NEEEEEEED IT
PeakVT
I don’t know software involved, but I see no obvious requirement to keep comments more than 2 days old “live”.
Sure, it’s fun to sometimes go back and find a choice comment proving that a certain Juicer is an fucking idiot, but that doesn’t strike me as mission critical.
Can’t the comments table be segmented or the old comments archived somehow?
WaterGirl
@PeakVT: Hahaha that seems like a glaringly obvious solution, but you don’t know how WordPress saves everything.
Major Major Major Major
This is just not true. I’ve monkey-patched WP before.
cleek
Man, just drop that list. We can take it.
A Ghost To Most
If it is the query that WaterGirl and I have been discussing, it was written ass backwards, from the view of this veteran SQL wrangler.
WaterGirl
@Major Major Major Major: Our developers do not want to support a monkey-patched version of WordPress, and I think that is very smart.
WaterGirl
There are solutions, but there are not Christmas-week solutions.
PeakVT
@WaterGirl: No, I don’t, which I stated.
Anyway, functioning of the live site should take precedence over stale comments, even this quality (but much-ignored) one. So take them offline.
Major Major Major Major
@WaterGirl: I don’t, but obviously your devs’ mileage varies. This isn’t the sort of thing that’s going to change ever, their database structure and core loop has been the same for ages.
ETA agree that you should just ditch it. Although that graph doesn’t really seem to indicate a problem with a query that runs every time the page is loaded.
schrodingers_cat
Just get rid of the recent comments feature. It’s nice to have but not have to have. Just my 2 cents
chris
Recent comments list? What is that?
zhena gogolia
@schrodingers_cat:
Me too. It seems to have caused trouble since time immemorial.
I just go to the bottom of the thread and scroll up to see if there’s anybody I like.
CaseyL
As much as I love the fact that we can search the entire archive of comments (like the recent memorial to the commentors who’ve died), I’m pretty sure keeping the live site functional is a higher priority.
@WaterGirl: Does that mean the archives can not be exported to a separate library database, with a widget installed to open/access/search it? Because that was my first thought!
PS Let’s see if this comment posts; if so, it’ll be the first time today!
ETA: Victory!
Ohio Mom
I hadn’t ever noticed the Recent Comments. I just found them, they are on the way, way bottom of my phone screen.
I’m another vote for tossing them.
If someone wants to keep up with recent comments, they can do it the old fashioned what, keeping many tabs open and scrolling away to their heart’s content.
schrodingers_cat
One good news out of India in the last two days, BJP lost power in another state, Jharkhand. It was their lynching lab.
A Ghost To Most
Strategic application of query focused indexes would be simple, safe, and likely help
mvr
I’m not going to tell you how to fix it (as if I knew). But I do want to thank everyone putting in work on this and everything else on the site.
And Merry Christmas!
ET
Aren’t we special……..
?BillinGlendaleCA
I found out this thread had been posted by seeing it in Recent Comments.
piratedan
possible option…. multi-part solution…
a) take comments offline post a certain time factor expiration date
b) add query function to run off line, set it to admin privs
c) gawd fucking forbid you need to look up something/someone, submit a request
d) front pager/admin submits it when they get around to it to run in the wee hours
e) you get snail mailed the results
working under the assumption that each time someone submits a comment that there are at least two keys, the posting handle and a time stamp to index but then again, this is FYWP, so who the fuck knows.
JaySinWA
I for one like and use recent comments. OTOH I have lived without them before.
It does seem that there have been a couple of design proposals from members that should have minimum impact on the blog performance, and while I know nothing of WP internals, the post code seems to be very changeable.
I thought the previous recent comments issues were understood and to be fixed with this revision.
HarryBee
Personally, I don’t give a flying eff about “recent comments.” But there is no doubt that this place is exceptional.
Rob
Did this go through?
eta: This is the first time one of my comments went through since yesterday afternoon or evening.
eata: I had been trying to post on the donate thread yesterday, to say that I had done so. i had been getting Error 502 messages every time I tried to comment (about 10 times all in all).
Ok, back to lurking.
MazeDancer
This is offered with zero knowledge, which will be obvious from its absurdity,, but is it possible that, say, the first 10 years of comments could be offloaded to another site. BalloonJuiceArchive.com? Or more or less of their weighty number sent elsewhere.
Then, those that want to search them, can go do that.
Also, recent comments are not worth all the extra time and money.
Mo MacArbie
Seems like everyone was frozen out for a half hour there. I guess when you turn off recent comments, you turn off recent comments.
wvng
I would just like to add that anything that can inspire the blog father’s unique ability to string words together in previously unimagined ways is not an entirely bad thing.
zhena gogolia
Is that a young John Wood in the clip in the OP? Wow, it is. How time flies.
J R in WV
@Major Major Major Major:
The worst possible problem — one with its roots buried in the original and fundamental design flaws executed in the very beginning of the system development cycle, and never addressed for repair and correction of the initial errors. No one ever dared to tell the original designers that they had made a serious and fundamental error in design and execution of the original concept, so the error was promulgated00 to every bit of the system over the years FYWP has been in use. Hip, Hip, Hurray!
TS (the original)
Somewhat off topic It’s Christmas morning in my part of the world, & Mr TS ended up in hospital early morning yesterday, like 3am (at least a good time for driving). The good news is, the visit won’t send us bankrupt with the hospital lady telling me “You have no excess to pay” & the government picking up most of the non-hospital payments. Otherwise I had to make my own morning tea (which I have done rarely for the past 45 years) and it will be a quiet day with the grandbabe lamenting no grandpa here.
Another positive – nothing major & he should be home tomorrow, so Happy Christmas BJ people & with or without the recent comments, it’s a good place to be most of the time & especially when home alone.
Baud
While I am a recent comments proponent, I concur about removing it if it’s causing that much of a headache.
Or delete my comments, which should free up about a 1000 terabytes of space.
CarolDuhart2
I’m with everybody who wants to archive. Cole, your site is a rarity: it’s almost old enough to go to college. I would archive everything from 2012 back with some exceptions for special occasions like Election Days and such. If anyone wants to know what you thought in 2008, they can go to the archive site, like HistoricalBalloonJuice.com and do a search there.
That rarity is why WordPress has a hard time here. It’s not just an old site, it’s also a very active old site, which means that a lot of WordPress isn’t capable of handling it.
Gvg
I like recent comments, but not enough to want the site not to work. Since so much work was involved trying to have them, I thought it worth saying I appreciated them even if it’s time to say bye bye.
i have had a few issues with access to this site, but not as much as other people evidently.
zhena gogolia
@TS (the original):
Oh, that’s too bad! I hope you can have a decent time once he gets back.
zhena gogolia
I love John.
Last night we tried to watch Sherlock on BBC America. But an ASPCA commercial about dogs left out in the cold sent us screaming for the remote.
?BillinGlendaleCA
@Baud: You comments take up no space, they’re almost a short as mine.
Patricia Kayden
JaySinWA
@Gvg: True. Better than no blog at all.
And a happy Hollandaise to you all.
JaySinWA
@Patricia Kayden: They rounded all the liberals up on a bus? Sounds sinister to me. ;-)
joel hanes
@?BillinGlendaleCA:
Guessing:
It’s not the space needed by the text that’s the problem; it’s the length of the rows returned by the queries.
phdesmond
@schrodingers_cat: i am glad that you keep us posted on the news from india, and i’ve recommended your tweets to several people i know.
phdesmond
@Baud: won’t that severely hamper your presidential campaign??
dr. bloor
Some folks just can’t adapt to prosperity.
?BillinGlendaleCA
@joel hanes: Please have your snark detector checked.
NotMax
Recent Comments was removed for a while only a few days or a week ago, and the hosting/server errors were happening just as frequently without it.
/giving side-eye to the hosting company for trying to shift blame
P.S.: There’s already an instruction to stop allowing comments after X amount of time (a month?), so there already exist two distinct classes of comment data. No glaring reason such an instruction couldn’t be extended to also include stopping a Recent Comments query for class 2 (i.e., closed) comment pages.
Ripley
I’ve worked on more than a few sites, and always aimed for site function over shiny bits. Many of those sites were WP with “Recent Comments” and – you guessed it – it’s been a thorn for a long time.
It’s sometimes a matter of traffic at a given moment.
It’s 5 comments – cut it. Then cross your fingers and back away.
cleek
@J R in WV:
AKA: The middle-age of every successful software package.
Baud
@?BillinGlendaleCA: Well, I’m no Tony Jay.
@phdesmond:
No, my campaign would immensely benefit from the destruction of incriminating evidence.
Odie Hugh Manatee
I’m another person who never uses the recent comments…
NotMax
@Baud
Jobs program! Bigly!
:)
phdesmond
@Baud: hee!
Mike J
you can’t really dust for vomit.
?BillinGlendaleCA
@Baud:
Give Rudy a call.
NotMax
Shall mention too that reducing where Recent Comments appears to ONLY the front page would do for me, as that is where I check it to see where the action is. Not having RC appear on any comment page itself would be A-OK.
J R in WV
@cleek:
So true !! And not-so-successful packages also too.
FlyingToaster
@Ripley: That was my experience a couple of eons back, when I was helping someone with a WP blog that was going nuts and their regular staff was out-of-country for a week. That convinced me that I had no effing business working in middleware.
I second the effort to either get rid of recent comments, or restrict them to the front page.
satby
Yep, seeing who was the last comment on which oldish thread just ain’t worth it. I never used them either, but even if I did, I’d want to ditch them.
OzarkHillbilly
Speaking only for myself, fuck the recent comments, I never used them either.
Seriously tho, even if I had, I’d say fuck them. The past is dead. Let it rest in peace.
I know iknow… all you fans of recent comments are going to slag me… Have at it. I really don’t care.
Yarrow
Comments close on posts older than X days and people are no longer able to comment on those posts. I’m not sure what X is, but it’s maybe a week or two. Seems like there’s no reason for the recent comments query to be looking for recent comments in threads where it’s impossible for there to be recent comments.
FlyingToaster
@OzarkHillbilly: I’m pretty sure the people who care about Recent Comments are studiously avoiding this thread.
In other news, I’m at ~halfway on the mixing of the baked goods. When the current batch goes into the bowl to rise, I can break out the refrigerated gingerbread dough and we’ll start cutting cookies.
Mary G
I don’t ever read the recent comments either, so add my vote to the dump ’em and forget we ever had them pile.
Since I am in a bit of an emotional seesaw pattern lately, I was irrationally cheered up by seeing a John Cole post about asphyxiating vomit and shitting the bed. Possibly because misery loves company.
I also need some more birthday wishes for me and Zinsky, please. Like McCartney, 64 is freaking me out a bit.
satby
@TS (the original): Happy Christmas to you and yours TS. Glad the mister will be home soon and it wasn’t serious.
satby
@Mary G: Happy Birthday to you and Zinsky too!
64 is a breeze!
?BillinGlendaleCA
@Mary G: Yeah, but Sir Paul was loosing his hair…happy b-day.
joel hanes
@?BillinGlendaleCA:
I retired from Apple five months ago, and since then, my ability to detect technical snark has atrophied.
I choose to regard that as a form of healing.
NotMax
@Mary G
Have a great one.
As I remember it, 64 is the new 63. ;)
Yarrow
@Mary G: Happy birthday, Mary G, you bright ray of sunshine! And happy birthday to Zinsky too.
???
Baud
@Mary G:
???????
Anne Laurie
@Mary G: Happy birthday, and many happy returns!
As we used to say in my family, think about all the people you’ve amazed
& disappointedby making it this far…HinTN
Nominating as a rolling tag.
From all of us to all of you,
Merry Christmas, Cole.
HinTN
@Yarrow: X is generally a variable
ThresherK
@Mary G: Happy “only one gift birthdayXmas”, MaryG.
chris
@Mary G: Happy birthday! You’re not getting older, you’re getting better!
Mary G
TomatoQueen
@Mary G: Oooo a holiday/birthday person turning 64 a week ahead of me. (I’m the 31st and fuck off in advance to all those who insist on saying they don’t care about NY Eve). May you have ample, thoughtful, and separate gifts from Vera, Chuck, and Dave or other fine people.
Amir Khalid
The server issue has eated yet another of my comments. Sigh.
Dorothy A. Winsor
@Mary G:
Happy birthday to both you and Zinsky!
Oh to be 64 again.
frosty
Do I dare say I like Recent Comments and was glad to see it back? I know nothing about software, having narrowly passed freshman FORTRAN and never going back. The ideas about not querying closed post makes sense to this non-software person. But I expect WaterGirl and the developers already went down that path.
Dorothy A. Winsor
@frosty:
Fortran. Wow. I think Noah learned Fortran
middlelee
I’m happy to admit I’ve never figured out what purpose is served by recent comments. I barely have time to read any comments, no matter just posted, recently posted, or historically posted. I’m lucky to get through the headlines on Front Page articles.
Baud
@Dorothy A. Winsor: Damn.
NotMax
@Dorothy A. Winsor
Worked with Fortran IV, COBOL and SNOBOL. Even dabbled for a short stint with Assembler.
CaseyL
@frosty:
@Dorothy A. Winsor:
I took FORTRAN about, oh, 30 years ago, during a brief and inglorious attempt at majoring in Computer Science.
I remember liking it quite a bit, esp. compared to COBOL, which I hated hated hated. IIRC, FORTRAN was a more intuitively-structured language.
zhena gogolia
@Mary G:
Okay, trying an hour later to say Happy Birthday and hope it isn’t another 502!
Kattails
Just got in, not even looking at the other comments, just do WTF works, I’m always just happy that we’re here. I don’t need bells and whistles. I was OK with roll-down windows on cars for ages. I live in NH and can work with un-heated car seats and having to start the car by actually getting in it and turning a key. Recent comments is just extra swirls on a fully iced cake.
Merry Christmas Cole hugs to all of youse & scritches to pets.
Baud
My favorite part about FORTRAN is when they all came together to form a big robot
Amir Khalid
Help! My new laptop is for some reason draining its battery, instead of running off mains electricity and maintaining a full charge. It’s Christmas. What should I do?
Aleta
@Mary G:
Happy Birthday ?????? to you and to Zinsky!!! ?❤️???
Nicole Pesce, piano: https://www.youtube.com/watch?v=S75gYhODS0M&t=0m9s
bubbles in slow motion: https://vimeo.com/379731088
w/ love ? ? ? ? ? ? ? ? ? ?? ? ? ? ? ? ? ? ?
A Ghost To Most
Ah, so the hoggish query dredges up Recent Comments. That explains a lot.
It is written very poorly for that purpose. I include it here.
SELECT wp_comments.*
FROM wp_comments
JOIN wp_posts ON ( wp_comments.comment_post_ID = wp_posts.ID )
WHERE ( post_status = ‘publish’
OR ( post_status = ‘inherit’
AND post_type = ‘attachment’ ) )
AND comment_approved = ‘1’
ORDER BY comment_date_gmt
DESC LIMIT 10
Another Scott
Do whatever needs to be done, but be aware that these problems will only get worse as B-J graduates from an almost top 10,000 blog to the next level. Should we be contributing for B-J NT?
I’m no dB Wrangler, but it’s baffling to me that the Recent Comments seemingly can’t be run differently. It should have at most, what, 2 days x 12 threads x 300 comments = 7200 possible comments to be listed the last 5 at a time. It shouldn’t be that big a job. WP really does seem to be broken in that respect.
Please do what’s necessary to get it over the hump, and let us know what we can do for the future.
Thanks muchly.
Cheers,
Scott.
zhena gogolia
@Kattails:
Great comment. I really miss roll-down windows.
Baud
Maybe we can get someone to handle Recent Comments manually.
Another Scott
@Amir Khalid: Most likely the charger has an issue. Are the sockets on the cord ok? Are they fully plugged in?
Good luck!
Cheers,
Scott.
Another Scott
@Amir Khalid: Most likely the charger has an issue. Are the sockets on the cord ok? Are they fully plugged in?
Good luck!
Cheers,
Scott.
HalfAssedHomesteader
Hope I’m not the billionth one to point this out but “Balloon Juice” seems to have disappeared from the top banner.
Miss Bianca
I like “Recent Comments”, she said plaintively, but if they bork the site I can live without them. Sic transit gloria mundi, and all that.
Off to watch the animated “Christmas Carol”, and then “The Lady Vanishes.” Good night and happy holidays to all!
Martin
So, here’s my suggestion:
SELECT comment.comment_id, comment.comment_author, comment.comment_date, post.post_title
FROM comment
INNER JOIN post ON post.post_id = comment.post_id
WHERE comment.comment_date >= DATE_ADD(CURDATE(), INTERVAL -1 DAY) AND comment.comment_approved = 'Y'
ORDER BY comment.comment_date DESC
LIMIT 5;
Make sure comment_date, comment_approved are indexed. Put this query in a block and give it a 1 minute cache so it doesn’t run any more often than that.
But given that it’s WordPress it probably doesn’t let you do any of that.
Another Scott
@Amir Khalid: Trying for a 3rd time…
I would supect the charger. Make sure the pins and sockets on the cord are ok and that they are fully inserted. If that doesn’t get it working, then you may need to try a different charger and/or take it somewhere.
Good luck!
Cheers,
Scott.
Another Scott
@Another Scott: So now it works… :-\
Cheers,
Scott.
Yarrow
@Amir Khalid: Obvious stuff to check–the charger works okay, the wall socket is functional, etc. Does the laptop run if you remove the battery but it’s plugged in? If that’s the case there may be something you need to reset to tell it to charge.
Yarrow
@Baud: For awhile I would get the 502 error if I tried to comment. After half an hour or so the comment would show up. Now comments seem to be posting.
Yarrow
For the record, I also like recent comments. It lets you know where the commenting activity is. Sometimes the most active thread is a few back from the top one. I’d rather have a functioning site than recent comments, though.
?BillinGlendaleCA
@Amir Khalid: First see if the charger is charging the battery without a load, note the battery percentage, shut down, charge for about half a hour, start it back up and see if the battery percentage has increased.
PJ
apologies to everyone else who has already made this joke.
The Golux
@CaseyL:
I took FORTRAN 45 years ago, and found it a drag. On the first day of my first job out of college, I was handed an APL textbook, and have been programming in it ever since. It eliminates everything I disliked about FORTRAN. APL adapts to your brain, instead of forcing your brain to adapt.
AnotherBruce
@Amir Khalid: Hire a red nosed Reindeer.
SWMBO
@Mary G: We will still need you. We will still feed you. When you’re 64.
Happy birthday to you and Zinsky!
NotMax
@Yarrow
Yeah, half a dozen things it could be. As it is a used laptop, initial suspicion is the battery is in need of replacement or got jostled enough to impede its connection. A battery on its last legs or without a firm connection can play hob with the power settings enough to keep the laptop from recognizing the AC charger.
As said, first thing is to check the charger. Unplug it from the wall socket, then poke around the female plug with a toothpick to remove any grit or dust which might have accumulated there. Then plug it back in and make sure it is seated firmly in the laptop’s plug socket. If you have access to another compatible charger, check if that one works, of course.
Second thing, if there is a removable cover to get to the battery (rather than having to open up the laptop), is to shut down and then open up access to the battery to make sure it is seated firmly, then start up and check if that fixed things.
Third thing would be (again if you don’t need to open up the laptop for access) to shut down, unplug from AC, remove the battery completely then see if it starts up and runs on AC without the battery.
phdesmond
@SWMBO: good one!
Martin
Around 40 years ago, taught myself BASIC at the local library – the only place I knew with a computer I could access. My dad bought me my own computer later that year – his ‘hey bud, I gotta move across the country for a job’ present. Taught myself assembler. Then took a couple of FORTRAN classes at the local community college. Then a couple of Pascal classes. Parallel with that, I taught myself C because enough people convinced me that it was the better path for general programming, and I couldn’t find any classes that taught it. Somewhere in there I dabbled with Forth for some driver development. In college C branched into C++, and I picked up shell scripting, mostly perl, but python was new on the scene and perl never really felt comfortable to me, so I mostly switched into that. Lots of scientific computing in college meant that FORTRAN came back into use, mostly on the colleges old mainframe. I taught the faculty that they could write their computational libraries in C and execute them from Python and do it on cheap PCs which were MUCH faster than the mainframe. So, they hired me to do that for a while including porting a shit-ton of LISP which was admittedly pretty dope for doing mathematical work, but was also slow as shit and there were only like 4 of us that knew the language. Also returned to writing drivers to interface with equipment in the labs.
After college I got a job in a language research project doing a range of programming work. The most fun was keeping a 20+ year old HP 1000 running, but I was also helping write 68000 assembly for a bespoke computer system developed as part of the project. The project had just shifted to distribution of content on CD-ROM (first application of them) and the only system that could read CD-ROMs were Macs and Hypercard had the first CD-ROM driver, so I did Hypercard programming to reduce the cost of access to the system from bespoke hardware to a Hypercard app that could be run on a generic Mac with no special hardware. That turned into a project to find a way do distribute the content online. The web hadn’t been invented yet, so I was developing a server that could deliver the information using the just being developed Unicode standard, marked up in SGML, and accessed via Gopher. The system had potential, but the Unicode guys weren’t ready to dive down our rabbit hole that early on, SGML+Gopher was barely up to the task (HTML was a year off yet) and our ideas for having users pay for the system were downright primitive relative to what would eventually show up on the Web.
Anyway, I wrote up those ideas as we were way too early, and moved onto a job as a developer on a small team writing an object database engine in C++. Did that for a while (it sucked, the work was fine, but the workplace was garbage) and then took a job developing in-house databases. That evolved into a bunch of different directions including online educational software and data science and junk like that. Got into policy work, with a very technical skillset. I still write code almost every day.
I’ll probably start writing indy games when I retire. I’ve been writing code since I was 10. It’s kind of what I do.
Amir Khalid
@AnotherBruce:
On Christmas Day in the tropics, any reindeer is hard to find. The red-nosed ones are really hard to find, and they refuse to work at Christmas for less than crazy money.
Amir Khalid
@NotMax:
The new laptop was running off mains power just fine all day Monday and Tuesday. It only started acting up this morning, running down the battery and not charging. I’m using the old laptop right now, plugged into the same socket.