A couple OF things (HAPPY EEMOM?):
First, server performance? How bad was it? I know I will get a couple of passive aggressive emails from Mistermix and DougJ about the inability of the server to handle the load during the debate. And I don’t mean that dismissively- when it comes to server performance, no one is looking out for you guys more than DougJ, who gets totally pissed off when the server fails. But we look at things from a different perspective, and let me explain how. He’s an advocate for you guys, and thinks any time the server is down for ten minutes or if the site loads slow, or if there are FYWP issues, they are things that need to be dealt with immediately and with the full force of Allah himself. Mistermix is an equal advocate for server performance. But the long story is we are just looking at different things and judging performance
This is not to say that I do not agree on the importance of server issues, but that I have a more holistic look. I look at tonight’s numbers, and as much as it pisses me off that I could not access the site for about 90 minutes, I’m still ok with it. On average, during a normal day, we get 3-4 k hits per hour. When a debate happens every couple of years, or an election night, or some other seminal event, traffic explodes. During that two hour period of the debate tonight, we had 8-9k hits per hour, and that only counts the ones who who got through. I bet if the server had unlimited bandwidth, we would be pushing 14-15k or more during that two hour period.
Here is where the conflict arises. As much as it pains me to admit this to you all, I just do not see the value in having a server that can handle 30k hits a day all day every day. The site works well with normal traffic, and if once or twice a year, it gets bogged down, well, that just happens. The same thing happens with cabs in towns on Homecoming- the normal 100 cabs is no longer enough, and while I fully appreciate the people being pissed off waiting for a cab that one week-end a year, it surely does not justify having 200 cabs year round, when 99% of the time 150 of the 200 cabs will be un-utilized.
So that is where we are. At this point, it makes no sense to expand bandwidth to google type numbers for just a few days a year. It sucks we can not have the site function the way we want on the few nights a year when we have this excessive traffic, but from a managerial standpoint it would be malpractice to make changes. Right now we are handling 75k hits a day. If I notice a decline in performance during regular days, I will without question make changes, but if the site goes down while 100k drunks are hitting reload on one of three debates every four years, I’m just going to pour another drink and hit reload myself. None of you, when buying a car you intend to drive between 60-80 miles an hour are going to purchase a Ferrari for the one occasion a year you need to go 180 miles an hour, and neither am I.
Second, This is probably the last you will see of me for four days. It is a 25 year college reunion for a bunch of my friends this weekend, and I have 14 people staying at the house or with friends. I have 60 people coming for a bbq on Saturday, and I have a pork shoulder to cook, 10 lbs of ground beef, 36 chicken drumsticks, and enough hot dogs to service Coney Island for a weekend. Not to mention all the cole slaw, potato salad, and macaroni and cheese I need to make- we’re talking shitloads.
So behave yourselves, and I will chat with you on Sunday.
Another Halocene Human
Ha ha Cole, I just laughed and moved on.
I think it’s reassuring that after checking hourly you were finally back for me at 1am. That’s pretty good.
Our Worthly Wonkets apparently screeched to a halt during the over 1500 comment debate thread of doom, so don’t feel bad.
LGF was up like a pro at 10pm, tho. So do feel bad?
MikeJ
Bandwidth isn’t your problem. WordPress is a pig and the database routines suck.
On heavy traffic nights, pull every plugin. No recent comments, no rotating tagline, no recent posts. Blog posts and comments and nothing else (except ads of course). It might buy you some breathing space.
Just Some Fuckhead
I just figured the slowdown was caused by 43 frontpagers simultaneously throwing up 43 Debate threads.
jwb
Server totally sucked throughout the debate, but it has been acting funny for the past few days, especially after mistermix did something to the database on Tuesday.
ETA: It was also nice to have this explanation of your thinking.
WaterGirl
Cole, I understand what you’re saying, but I think it’s a mistake.
Balloon Juice was worthless for the entire hour and a half of the debate. I tried to post a comment twice, but it puked. It was obvious that a lot of others were having the same problem.
Who wants to be online with a bunch of strangers on election night? There must be a way that you can get more bandwidth for that.
What MikeJ said is right about reducing your plugins for the big nights. It would at least help some.
And I don’t want to be mean, but I think you should hire a WP guru. They wouldn’t have to change the design, but I’ll bet there can be a lot done behind the scenes that would make the site behave much better.
Yutsano
@WaterGirl: I thought we had the blogmistress for that. And from what I understand she has more clients besides us crabby folk to deal with.
As far as I’m concerned you don’t owe us any explanation JC. We can suck it up or go somewhere else. And maybe it was just me, but I didn’t think it was all that terrible.
Donut
What with all the vast sums of money I put out, as a paying customer of Balloon Juice, I deserve to get much more than I got tonight, when I couldn’t get a connection for about, like, 30 minutes.
Oh, wait…
khead
Don’t sweat it.
I bitched because that’s what I do.
The site came back roughly 10 minutes after I griped.
? Martin
I can appreciate that perspective. In that event, my suggestion would be to have FPers liveblog and shut down comments for 2 hours, let the sever cache everything and then open things up after. Works for a lot of other sites that get traffic spikes like this.
Election night is a different kind of problem, though. For that, maybe get a stripped down design for the day. No recent comments, etc. Just the minimum.
freelancer
So it’s like Target with 45 lanes and one cashier open. Except on Black Friday. In which case the cashier quits.
Steve T.
That many people, and that much food? Seriously, watch out for the dogs. They will beg and beg and get fed and fed, lots of illicit food, and human food ain’t necessarily good for them. A hotdog is bad enough but god forbid a drumstick! Jeez, warn your guests, John. We don’t want anything to happen to Lily.
Pet her for me, cause I miss my dog.
WaterGirl
@Yutsano: I think we have a blog mistress for that, but I don’t get the impression that she’s a guru with WP.
She seems very nice and all, but a blog like this should have top level people doing the programming. Maybe my 25 year career in IT makes me too critical, but that’s what I think.
Cole raised the subject, so I felt like I should be honest. I have thought that for a long time and have never said it. Maybe my frustration with the debate is impacting my judgment.
Just Some Fuckhead
Gotta say, I didn’t really have any problems commenting. Must be the preferential treatment I get around here. I did get an error once, something like Whatever you are searching for cannot be found but I considered that more of a philosophical problem than an issue with Balloon Juice.
Higgs Boson's Mate
To pay continuously for the server capability to handle a load that only happens once in a while is a waste of money. This is a blog, not a business. If we had to pay per comment then those who want more capable servers, or a paid webmaster, would have an argument. You want more robust servers or Johnny-on-the-spot site administration? Then send Cole money, lots of money.
MikeJ
Some more ideas:
Fire up chrome, hit menu, tools, developer tools, audit, run.
According to that you don’t even have gzip turned on. That’s web malpractice. You could be saving 2MB per page hit for flipping one switch on the server. Which doesn’t help back end performance, but it gets requests off the queue faster.
Combine your css files. Combine your javascript files. You don’t save much on bytes xfered, but you save building and tearing down a new connection for each file you send. And by the way, 6000 of your css rules aren’t used on the front page. That’s half of them. Rethink your css.
You have 103 things on the page marked as non-cacheable by the browser. Think about those and see which ones can afford to be a little bit stale.
Serve static content form a cookieless domain. You’re resending cookies on static content. Bad move.
burnspbesq
During the two-hour period immediately before, during, and after the Supreme Court handed down the decision in the healthcare cases, SCOTUS Blog had something like 8x it’s normal load, and it performed just about flawlessly. I assume they went to Amazon and bought enough capacity to scale up on a very short-term basis. I’m sure Tom Goldstein will tell you how they did it if you ask him.
eemom
[obligatory swoon]
Rathskeller
@MikeJ: Please listen to both of Mike’s sets of suggestions.
You might also consider replacing the Balloon Juice logo and any other images during anticipatable periods of high load with a skeletal version. The logo is 43K.
I agree you should not spend money on servers to cope with temporary loads. I use USAA for my insurance, as you probably do to. Years ago they endeared me greatly by sending out a letter that said basically, “We can see from our stats that wait times on Mondays are simply unacceptable. Well, we’re not going to staff up for one heavy day a week just so you can get faster service. Call us on other days of the week, or stop complaining.” It was translated into corporatese, but that’s the essence.
MikeJ
@MikeJ:
To reply to myself: this doesn’t require a different machine, just some dns tricks, reconfigure apache (or better yet, something a lot more lightweight, nginx maybe, for the static stuff).
jwb
And fix the fucking spam filter while you’re at it. It’s been eating more and more comments without a trace for using mysterious verboten words.
max
First, server performance? How bad was it?
Well, I just figured your pants were trapped in the BJ gears. Either that or the hamsters were on strike. (Possibly the hamsters choked to death trying to eat the pants?)
and if once or twice a year, it gets bogged down, well, that just happens.
It’s fine, John, really. Every site I was hitting was slowed a bit, so I figured the BJ machine was getting hammered and didn’t worry about it too much.
That said, as Mike says, tweaking the layout and your server switches might not be a bad idea. In fact, I would vote in favor of that instead of purchasing bandwidth because even if you up the bandwidth, if the site architecture has issues, the site will tend to implode at high bandwidth usage anyways. So then the machine wouldn’t choke because of lack of bandwidth, it would choke because of code issues.
max
[‘Additional hardware pays off for a little while, optimization pays off forever, more or less.’]
Redshift
Dunno if it’s possible with the platform you run on, but there are deployment platforms that allow replication and burst bandwidth, so strictly speaking, it doesn’t have to be a choice between paying for maximum bandwidth all the time and dying during heavy-traffic events. But obviously I don’t know if that’s possible without a major overhaul, and though I’m a techie, I don’t really know anything about it beyond having read some of the Amazon Web Services docs. But since that kind of traffic pattern is going to be a permanent feature of any political blog (and lots of others), it seems like something to at least include in the longer-term planning.
Redshift
In answer to your actual question, I got home late in the debate, so I wasn’t posting comments, but the site was pretty useless for me until the debate was over. A lot of database errors, which suggests to me an architectural problem, not just a bandwidth problem. (But I could be wrong.)
James E. Powell
I ended up following the live blog at GOS. My mistake was reading the comments. Fortunately, I had GoJo to clean up with. I read the posts there regularly, but the comments not so much. Those guys are on our side?
Anyway, this is Over-reaction Day. Let’s reconvene after the next round of swing state polls come out and in the meantime, continue that early voting.
The prophet Nostradumbass
Why are you giving that twit the attention?
Odie Hugh Manatee
So hamster power it is, forget about upgrading to chinchilla power. ;)
I hear that Redstate runs on rat power and it’s not the Chevy kind of rat.
eemom
@The prophet Nostradumbass:
oooh. Bitter aren’t we, that no one cares we exist.
Take heart, however — I only have three personal trolls, and you are totally in the top two.
Arclite
@John,
Thanks for being clear on the numbers. Out of pure curiosity, what are you paying, and what would it cost for a server that handles the traffic? I work in tech, so I knew what the issues were, and thus wasn’t frustrated by it, and I know you run this site out of the goodness of your heart, but I’m curious.
Arclite
Yeah, like that’s gonna happen. I’m sure that we’ll see pics of all those folks on the site at some point over the weekend.
SiubhanDuinne
@eemom:
(Psssst, eemom. Not so fast with the swooning. Did you notice there was no comma after “happy”?)
rageahol
you keep talking about “the servers” as though they were a set of physical machines in a rack somewhere.
if that’s the case, then you’re ~5 years behind the times.
you can expand your bandwidth dynamically with most of the brand-name services, e.g. amazon. if you’re still stuck with a pipe that has hard limits, then you’re paying for a tier of service instead of for traffic, and this is a big enough website that that’s kind of dumb.
unless you’re doing it to provide employment to yokels in your neck of the woods, in which case, play that shit up.
rageahol
@Higgs Boson’s Mate: this is exactly the misperception that needs to be corrected.
virtualization. it’s a thing.
Maude
I was on Twitter.
I did tweet to Sooner and got an answer within minutes.
I think I should get some kind of prize for not getting into comments and telling all of the thousands to email John.
Things are okay the way they are.
We can all come back later and gripe about it. What more could you ask for?
BruceFromOhio
For a free site that runs pretty well the other 99.973% of the time, there’s little value in re-engineering for load spikes. You either have to pump up everything, or do the suggested sleight-of-hand tricks to make what’s in place work more efficiently. Unless there is some serious skin (read: $) to be had for it, leave it go.
@Another Halocene Human:
Its easy to run a site that no one visits. ;)
pseudonymous in nc
@rageahol:
There’s a Donate button at the top of the page; your contribution will be welcomed.
As somebody who deals with this quandary fairly regularly: yeah, it doesn’t make sense to pay for hardware (physical or virtual) to cope with the spikiest load when day-to-day load is orders of magnitude less. You could go with something like Amazon RDS — because yesterday felt a lot like a database bottleneck — or with HyperDB for replication and sharding, but that requires a bit of expertise to deploy more hamsters when needed, and that usually doesn’t come for free.
In the meantime, a barebones FYWP config for busy times may make a difference.
Lee
As a web programmer, listen to MikeJ.
The only other thing I might recommend is to see if your service provider will rent you some extra processing power for the next month.
My guess is now until Nov 6th you are going to see a steady increase in traffic…and then a sudden drop off.
Put a Donate link up for the extra cost of need be.
Soonergrunt
@SiubhanDuinne: [shakes head] That reminds me of something my daughter used to do.
My son, who has Asperger’s Syndrome, also has its evil twin, OCD. He’s not as bad as he used to be, but when he was about 12 and she was 7, he had this habit of arranging his video game controllers just so. It was very important to him to have everything in place, and the cords properly routed and so on.
Whenever she was mad at him for something, she’d go in his room when he wasn’t around, and move one of the controllers just ever so slightly. We caught her doing it once. He’d come back to his room from wherever, notice the change, and then we’d be off to the races. For HOURS.
slag
@MikeJ: Exactly what I was thinking. These suggestions seem like a gimme. And no extra expense required. Just barebones it when things get busy.
blingee
The loser of last nights debate was definitely Cole’s server. Epic fail sauce.
BGinCHI
Shit, I bought my car in case I have to go 180. Now that I think about it, with the car seat in back and all, it was probably an overreach.
Phoebe
Thank you EEMOM and thank you John Cole for catering to her. Maybe this will spread to the general population, maybe it won’t, but that one instance has made me all happy. Sigh!
Pseudonym
Did Jim Lehrer take over your server last night or something? I’m no FYWP expert but I do have a background in building scalable web serving systems, and I’d suggest hiring/bribing someone like MikeJ to optimize your site. It definitely seemed like the database was the bottleneck rather than connection bandwidth or web server resources. Are you running the database on a separate machine from the web server? That’s one possible step you can take to increase performance. Serve static content from a CDN on a separate domain if possible and make sure you have all the resources set to cacheable. You have a bunch of images in the “Meta” section that link to incorrect URLs and result in loading a useless 86K 404 page; fix those links and use an HTML redirect for the 404 page or something instead. Keep Tunch away from the hamster.
CommishTheFirst
My only qualification to discuss is this is that I’ve hung around here for several years and posted a few dozen comments in that time, but… it seems to me that when people are constantly refreshing a blog comment thread and looking to it for real time snark-streaming, they are trying to use a blog comment thread as something it was not designed to be. It sounds like what people want is a chat room. Is there a chat service out there where the community could gather for big events like debates and election night, maybe run it in a window alongside the main blog page?
Or, Twitter? I just created a Twitter account yesterday so I could follow jgcole, tbogg and a few others during the debate. Despite the debate being, allegedly, the most twitted event ever, it seemed to work fine. Maybe there’s a way to move this community over to Twitter for big events without hurting B-J.com?
Pseudonym
Also, too, it’s all about graceful degradation. When the server’s under a lot of load, turn off noncritical plug-ins and features like the Recent Posts/Categories/Authors/Archives box and the Recent Comments box. But using a separate service for live streaming also makes sense and stuff.
Terry Chay
For the hundredth time, Cole, call WordPress VIP program. For your blog metrics and style (single blog, high traffic) that program will save you tons of money and support time. Kthugs blog is on it and it doesn’t go down and unlike WordPress.com they will let you run all your crummy plugins (including ads… They power Icanhazcheezburger too) and audit them for you so they don’t do thinks like destroy block quotes and make it possible to refer to a political movement without hitting the span filter or say republican with turning into a registered trademark.
Note, I no longer work there (automattic) I now
Work at wikipedia (wikimedia), but as someone who has followed you for 8 years (back when you were a righty) this has been the most painful fuckup to watch.
Send a goddamn email to automattic’s VIP. Show them that Maddow, kthug, and others read your blog and save tons on hosting and maintenance.
BattleCat
There’s gotta be at least a dozen people who read BJ daily that would volunteer time for this.
That or WordPress VIP.
I fixed the reply button gratis back when; in the same vein, I’m sure you can put the call out and get some other guys to fix a few issues each and be done with the whole thing.