Programmers are Expensive, But Operations are More Expensive · 22 December, 09:49 AM
I really like reading Coding Horror, even though I disagree with Jeff Atwood more than half the time. I have several unfinished posts rebutting articles of his, but I don’t really want to become the Coding Horror Rebuttal site, and I do have other things to do anyway (and I couldn’t come up with graphics half as spiffy as Jeff’s anyway). Recently, he wrote Hardware is Cheap, Programmers are Expensive, which is a well-known aphorism; certainly it’s been the rule of thumb for all of this century. Another way to state this is “Just throw hardware at the problem”.
Jeff’s overall point is that you should “spend your way out of a performance problem first by throwing faster hardware at it”. The problem I have with this idea is that it’s from the perspective of a small programming team operating their own server. While it’s true that optimizations are often a waste of a programmer’s expensive time, it neglects the operational cost of additional hardware. It’s pretty simple to manage a single server, but each additional server starts to increase operational complexity. Before long, you’re hiring people to manage the additional servers, and these people have a different skill set. They’re not programmers, they’re experts at running networks. A good engineer is a godsend to a programming team, relieving programmers of all kinds of tasks that shouldn’t be distracting them, and that programmers probably aren’t that good at anyway. But operations experts aren’t cheap either, maybe not the price of a programmer, but not cheap by any stretch. Furthermore, horizontal scale introduces new problems: power consumption, heat dissipation, etc. If you think it’s hard to get enough power into your personal workspace to run all your gadgets, try doing it for an entire datacenter. That’s why Google has taken to building datacenters next to hydroelectric dams – certainly they scale massively, at a cost.
The other problem is that additional boxes create additional points of failure and latency. It’s true that services like Amazon and Google are examples of massive horizontal scale, but these systems didn’t happen by accident. They were the result of careful planning and good architectural principles. The danger in the “throw hardware at the problem” aphorism is that it’s really easy to ignore true architectural problems in the process. As always, there’s a balance between thinking an acting that’s at play. Like all silver bullets, Moore’s Law comes with caveats.
— Gordon Weakliem
Comment
Commenting is closed for this article.
Reminds me a bit of this:
http://groups.google.com/group/comp.lang.lisp/msg/b3b24fb7512f220f
“There are no mom-and-pop oil rigs in Norway.”
— Zach Beane · Dec 24, 08:21 AM · #
<i>It’s pretty simple to manage a single server, but each additional server starts to increase operational complexity</i>
Not really. If you’re scaling horizontally these systems are identical and thus even the most primative build and configuration management system will handle it.
Complexity in system administration does not correlate well to the number of servers but the difference between those servers.
— Serge · Dec 28, 05:25 AM · #
Not to mention budgets, deadlines and all those other “distractions” related to operational costs. If the project in question is a focused marketing sideshow, for instance, such as a support site for a TV commercial, there simply may not be time nor money for any improvements at all, once the site is live. Having considered operational issues up front could easily become the do-or-die factor. Having an experienced engineer in on the project from the start, if at all possible, is always the better option.
— Vingborg · Dec 28, 06:24 AM · #
@Serge, actually in my experience scaling horizontally can be a hard move. Additional servers require possible rewriting of code to split tasks amongst separate servers, additional skills to manage load balancers, more knowledge in how to setup servers to operate in tandem, etc. Even with servers that are exactly the same, it still requires a lot of work and specialization to get everything working the way you want them to.
I think it depends at one point you have to optimize the code. Google is the perfect example, they throw money at both ends of the problem, lots of hardware but lots of customization of both database and code to optimize solutions.
— Justin Yost · Dec 28, 09:01 AM · #
Initially plausible, but falls apart on further consideration.
Suppose you are starting a small software company that requires production servers. You will need to have someone in charge of this network – you’re mad if you want one of your programmers to do that “in their spare time”.
Now, in order to save programmer’s time you double(!) your number of servers. Does this double the amount of time your sysadmin must spend administering them? Assuming he’s reasonably savvy, the answer is, “Absolutely not.” If you have more than one server machine already, your competent syadmin has already set up scripts to manage it. The time costs for him are setting up boxes with a brand-new purpose, not cloning boxes that already exist.
Certainly balancing and the like takes some time – but this is a task that should have been done already – and you want to know how it’s done now, not later when you are finally popular but your servers start dying left and right because you don’t know how to scale.
Google takes this to a fine art where you can double the number of machines by simply changing a few numbers in a text document.
We don’t all have that capability. And of course there are sometimes when “throw more hardware at it” a bad idea. But generally, if you can help your programmers by buying more hardware, it’s a no-brainer – the amount of extra sysadmin time you’ll require is marginal and you would have had to expend that time if you became successful anyway.
— Tom Human · Dec 28, 09:18 AM · #
I am one of those “Opperations Experts” that you mention, although I have recently gotten more into the managment side. So I have a bit of a different perspective. Generally speaking the phrase “Hardware is cheap, programmers are expensive” is true. But really it should be modified to “Hardware is cheap, PERSONAL are expensive.” It really doesn’t matter what kind of personal, when you include salary and employee overhead it quickly can become a cost negitive proposal.
When a request comes in for new funtionality to be added to the network I have to make a number of decisions related to the needs of the project. Is this critical? How quickly does it need to be done? What skill sets do I have in my enterprise to support this feature? What is the expected lifetime of the feature/application? Is it a modification of existing features, or a completly new feature and is there a applicance like product that will do the same job? Lastly I have to consider funding. Do I have a lump sum to play with? Or do I have to encorporate this into an existing budget?
Now I don’t belive in reinventing the wheel, and a programmer employed to write funtionality that already exists in some form is simply a waste of money unless you have a under utilized dev team that is required to support some pre-existing applicaiton. In that case they had better be looking for new jobs as it would be generally more cost effective to depreciate the applicaiton and replace it with something that requires lower HR overhead. As I said, personal are very, VERY expensive.
The situations where it would be better to hire a dev on contract are those where either there is no pre-existing appliance or application that fits the needs of the enterprise, or where modifying an existing applicaiton or creating a new one costs less over the lifetime of the project than the licencing and upfront costs of the applicance/application and I have a large lump sum project budget to fund the program. If I have to encorporate the costs into an existing budget, the increased lifetime costs of the applicance/application may be acceptable in order to get the job done.
Lastly there is the support aspect of the application. Who is going to support it? How much training will they need? As any in-house developed solution will be a custom one, I know I can’t just hire the skill sets I need and instead will have to absorb the full costs of training. Generally speaking custom applications aren’t cost effective. From a opperations perspective, these custom applications can take more support time than any other issues combined and from my experience simply aren’t worth it for most organizations (I speak of small to medium/large enterprises.. very large enterprises are a different story as they have mission critical apps that can’t be easily removed and require devs because of a large investment made many many moons ago). In this case it makes sense to first look at a custom application as you have the HR overhead anyhow, depending on how heavily tasked your devs are.
As you can see, the formula is far more complex than that just of compairing HR costs to hardware costs. Even though generally it isn’t cost effective to develop a custom application for a task, there are times when the equation comes down the other way (Web site design and maintance for instance). However, I would say that generally the need for devs in the enterprise is far smaller than is assumed by most enterprises today. Quite simply there are more programmers than we need for opperational stablity, and we pay them far too much for the value that they provide to the organization in a traditional setting. Generally their jobs could be better funded through having contracted a third party specialist rather than an on staff full-time dedicated developer.
— M. Edlund · Dec 28, 03:26 PM · #
Throwing hardware at a problem is becoming less and less viable, as machines become more parallel instead of faster.
Many times I’ve heard people buying machines with quad quad-core processors to try and speed up programs that just aren’t written to be that parallel. The end result being that lots of money was spent on hardware with next to no benefit.
Unless your software is written to be more parallel, more hardware isn’t going to save you.
— Jesse McNelis · Dec 28, 07:49 PM · #
Lots of good points, but they all start with the assumption that you’ve hired an operations staff and have money to give the operations staff good tools. That wasn’t my starting assumption. I’m starting with the straw man that you’re going to start with the assumption that you’re going to buy bigger hardware as a first step out of performance problems. Not think about the problem first, spend money first. Naturally, nobody reading this would be that dumb, I understand that much…
Spending your way out of production issues isn’t a silver bullet. Here’s a straw man: you have a site that’s serving static images without cache-control or an ETag. You could go out and spend a few on servers to serve images, but that’s money thrown away compared to applying cache-control and Conditional GET. Spending your way out of that kind of performance issue is, as my mom likes to say, “having more money than sense”.
So if you have some limited number of dollars to spend, do you buy monitoring software, or spend that on a beefier server?
As for the silver bullets “use identical configurations”: If you have an existing set of servers, do you add them to your cluster, or replace existing servers to maintain identical configuration? Better yet, “use AWS”: now you have even less control over your harware. Use AWS is explicitly not buying the best hardware money can buy, you’re buying a virtualized slice, for crying out loud!
— Gordon Weakliem · Dec 29, 03:07 PM · #
You don’t spend your way out of O(N^2) or worse problems.
Fuck bloggers.
— Stop Blogging · Jan 1, 10:47 AM · #