Thinking about optimizations
It appears that performance is getting pretty bad for the
software that runs this weblog. There are a couple components to
this: first of all, I knew going in that my method for selecting
the entries that needed to be rendered was not at all efficient.
The current method simply builds an entryquery
structure from the CGI REQUEST_PATH variable and then walks through
all the posts in my post store, checking each to see if it matches
the query. This entails opening and parsing each file to have a
look at the posted date, category, and post ID; obviously,
selection gets progressively slower as I create more posts. There
are a couple of possible solutions for this problem:
- Create a separate index file for metadata used in the query. This seems like the most straightforward method; if I maintained a list of (id, post date, mod date, category, filename) and used that list to determine what files I needed to actually parse, that should speed up querying quite a bit. There is a risk of the index falling out of sync using this method, but I think it's manageable.
- Use the filesystem metadata as an aid. For example, I can use the create date on each post file to determine whether a file could possibly fall into the query. This would help the primary use case, which is "render the 5 most recent posts", and could be used to help the use case of rendering a specific post, which is also pretty common. Unfortunately, for things like the calendar display, it wouldn't work very well, since an action like migrating my post store from Atom 0.3 format to Atom v.next could wreck all the file create times. Also, it doesn't help for queries by category. This is why I don't like this solution, the limitations of filesystem metadata were a primary reason for moving away from pyBlosxom.
- Ditch the filesystem altogether and move to a relational store. In my case, my hosting provider lets me have a mySQL database. This is my avenue of last resort, however, I don't like the complexity that adding a relational DB would bring.
I've also been reconsidering my decision to go with totally dynamic rendering. I think that I can still get decent performance from a fully dynamic site, but on the other hand, Apache is really, really good at serving static files, and implementing things like etags and Last-Modified would be trivial with static files. This is a pretty big change from the design I started with, however, and I'd really need to think through the implications of a change to a post. One other problem is that I carried over pyBlosxom's idea of "flavors"; for example, my atom feed is at http://www.eighty-twenty.net/blog?flav=atom, and this differs from the url for the HTML version and the RSS version only by the query string. If I stored my feed in a static file, I'm not sure how I could get around handling the query string. I could possibly use mod_rewrite, or my CGI itself to issue redirects to the static file, but I'd have to research that. This does bring up the point that when architecting a web application, the design of an URL really does matter. The URL your application's interface to the rest of the world, and if you change the interface, you need to be prepared to deal with the breakage. Personally, I hate when a feed that I'm subscribed to asks me to manually resubscribe at a different location, so I really don't want to be forced into that change.
Finally, one last point is that the source code for this site has roughly doubled in the 2 months since I went live. Since Scheme is interpreted, and I'm running as a pure CGI module, parse time should factor into the overall time, and this time is incurred on every request. mzScheme has some sort of compilation facilities that might be able to reduce this time, I'll have to look into this as well.