Scalability Notes

Jon Udell has an article on the Scalability Myth. Scalability is like optimization, premature scalability is the root of all evil. As Jon mentions, there are a lot of axes to scale along. Scalability usually brings up discussions of Amazon or Yahoo!, big websites with high loads. But I've seen discussions of scalability completely kill a project that never will see that kind of volume. Yahoo! and Amazon weren't always the size they are now, they grew into what they are. But people still go around wasting time on ill-advised caching schemes and what have you, when I think it's more important to get something working, get some good measurements, and let them guide you.

There are a few principles that generally hold true, however. When you're talking about scalability in the commonly understood sense, statelessness is definitely an important feature of these architectures. My last project was a set of web services running moderately high message rates (on the order of 7 digits / day), and the (mostly) stateless nature of the system was a definite asset, both in performance and management, but the killer was the message existence time on the server. It's one thing to scale up with messages that get processed in a second or two, it's something else with messages that can take 20 - 30 seconds (that's not a misconfigured DB or anything, we deal with a lot of external data sources that can be slow). Caching slow data sources helped out, but what we really discovered is that the problem doesn't really go away until you have a really decoupled architecture. You can't just block on a socket for 30 seconds and hope your thread pool doesn't fill up. The thing is, we did extensive performance testing, but never anticipated that we'd have 20-30 second waits for messages.

— Gordon Weakliem at permanent link

One Word Makes the Difference

I was just trying to research a (unconfirmed) problem with Visual Studio.NET importing a WSDL that uses <xs:element ref="..."/> style element declarations in the <types> section. What's interesting is that Google returns two fairly disjoint sets of results for the query Visual Studio wsdl ref, and VS.NET wsdl ref. The former returns sources approaching the issue from the perspective of a Java developer trying to figure out why a WSDL they generated won't import, and the latter comes from the perspective of a .NET developer trying to do the import. Actually, the latter also has a lot more articles on the mechanics of using .disco files, which is pretty much a .NET specific thing. Offhand, I'd guess that the Java developers are referring the development environment formally, holding it at arm's length, while the .NET developers use the shorthand.

— Gordon Weakliem at permanent link