Detail-oriented Programming

June 7, 2007 | 12 Comments

What is a full-stack web framework, and what is it good for?

Back in the day (which day being 1995 or thereabouts) there was essentially one web framework. It was a Perl module, called CGI.pm, which decoded the CGI request and provided functions to construct HTML and a CGI response. (CGI.pm is still around, but the past tense adds to the mood of misty reminiscence.) It didn’t have a template system (but Perl “here documents” let you embed HTML directly in your script), or session management (you rolled your own—it’s not hard), or a persistence layer (you used the filesystem, or DBI.pm to talk to SQL databases), but it was simple, easy to use, and easy to understand.

Now, after more than a decade of progress, we have this gigantic Java stack trace of a system running various frameworks and libraries. Somehow I don’t think this monster of delegation is the “full stack” that people have in mind. This setup does not seem easy to use and understand. How did we get here? As usual, with the best of intentions. We’re programmers, so we invented abstractions.

A lot of frameworks seem to be motivated by a handful of interrelated fallacies:

  • Abstraction is always a good thing.
  • All applications have common functions that can be usefully abstracted out.
  • There are details you don’t have to think about.

It is natural and valuable to seek abstractions when programming. A well-chosen abstraction can eliminate pages of similar code, and make the program easier to understand. But abstraction has a cost: it is a layer between you and what is actually going on, so it can also impede understanding. Carving up a program into useful abstractions is an art.

Can we abstract out the common functions of applications? Sure. From 30,000 feet you can’t tell the difference between a Porsche and a Pinto, so go ahead and abstract out a Car. Is this a useful abstraction? A Porsche goes a lot faster than a Pinto. A Pinto blows up when you crash it. Whether Car is a useful abstraction depends on whether these qualities matter to your application. In the same way, whether the abstractions the framework provides are useful depends on your application. But you’ll pay the cost either way.

The worst possible use of abstraction is as a fig leaf. These frameworks, with their menagerie of strategies, managers, and delegators, seem to ascribe no cost to complexity. “You just write your application,” they seem to say, “and we’ll take care of the details”. You can hide arbitrary amounts of complexity, confusion, and bad design behind a nice-looking interface. The problem is that it won’t stay there. When your application isn’t fast enough, or you have a bug that doesn’t seem to be in the code you wrote, either you play Black Box, or you dive into the framework code. At that point, the layers of abstraction start to really hurt. Ultimately, there are no details in a program that you don’t have to think about—all programming is detail-oriented programming.

At Skydeck we are in recovery from frameworks. In the past we’ve used Java, Tomcat, JSP, and a home-grown object-relational mapping tool on top of JDBC and MySQL. We never went all the way down the path of MVC, IoC, JDO, JTA, EJB, and all the other framework-related acronyms, but we had our share of deep stack traces.

This time we’re trying to keep it simple. For web programming we’re using the excellent Ocamlnet. Ocamlnet does what CGI.pm did, and does it simply and well. It also does a ton of other useful things (HTTP and SMTP clients, MIME message parsing and construction, character set translations), but those features are structured as a set of libraries that you can call when you need them rather than as a framework that you have to take all or nothing. When we need to dive into the source, it’s written in a reasonable language.

For storing data persistently, we have been using MySQL and OCaml-MySQL, with no vendor abstraction layer or object-relational mapping. Lately we’ve been feeling that this isn’t the right default, but we’ll save that story for a later post.

I’m not arguing that you should reinvent the wheel, or that real programmers should code on bare metal, but rather that every abstraction has a cost; that web frameworks are often collections of expensive abstractions; and that you should make sure you’re getting good value for your expense.

Author’s Photo Posted by: Jake
Categories: Technology, Programming

Comments

12 Responses to “Detail-oriented Programming”

  • Peter Thomas on June 8th, 2007 3:32 am

    *groan* no, not another blog referring to that stack trace as a reference for “Java Frameworks Suck”. Sometimes I regret having posted that.

    I completely agree with your final point, every abstraction does have a cost - and you need to be sure that you are getting the value you are comfortable with.

    Also I have to say, that post is a little old and I got rid of three of the biggest offenders - so I use Jetty instead of Tomcat/JBoss and Acegi + Spring WebFlow are out. Also note that there is a Ruby on Rails stack trace linked from that blog post, and that makes for interesting reading. The point I try to make is that if you start using the size of a stack trace to compare frameworks - I mean, who knows, once you flesh out your OCaml framework and decide on some ORM or equivalent, security, page-state and other stuff - then run Dtrace or something and look at the results - you may be surprised.

    You can find out more about the actual (open source) application “responsible” for that stack trace here: http://jtrac.info

  • jaked on June 8th, 2007 9:09 am

    You’re right, the big stack trace is kind of a strawman. I’m not arguing specifically against Java frameworks, but rather against the thinking I characterize (caricature?) above.

    As you say, the size of the stack trace isn’t a perfect metric, but once we launch I’ll post the deepest stack trace I can find.

  • andrey kartashov on June 14th, 2007 8:04 pm

    Hey, Jake!
    As always, you bring up some interesting points, but I daresay that there *are* some details I don’t have to think about. More than a few actually. Think about it, the stacktrace you quoted is just the tip of the iceberg, there is whole OS, many layers deep behind it. There is also hardware, a complicated beast all in itself. The simple reality is that today nobody really understands ‘the whole thing’, there are many layers of abstraction involved.
    So how do we deal with them? - We mostly don’t;) We choose the set of tools/frameworks that solves the problem we are trying to solve. Nobody insists on using as many as possible, there is no ’standard’ software stack either. It’s like a toolbox, you pick and choose what you need. You try it, you measure performance, you identify the bottlenecks, only at that point do you start paying attention to details. But not to all details, just the ones that matter. You keep what works well and you replace or fix what doesn’t. And, yes, you pretty much completely ignore the rest….
    So, as you’ve pointed out, there is a cost, but do we avoid frameworks? - No. We simply choose them based on whether or not they solve our problem. Or to put it another way, if you can’t clearly explain why you are using something, you probably shouldn’t be using it.

  • Jake on June 14th, 2007 10:19 pm

    Hi Andrey. I think we are in basic agreement: complexity has a cost which must be justified. You put it very well in your last sentence.

    My argument is that too often we take on complexity without justification. Extremely abstract, extremely flexible frameworks are extremely complex, and yet hardly any application needs the flexibility they provide.

    You make a good point that the stack includes everything from the hardware on up. Still, when something goes wrong (be it bug, security hole, or performance problem), it can go wrong anywhere in that stack. This means we have to keep the stack as shallow and as simple as possible in order to get our jobs done.

    (A data point: check the Facebook job listings, where they have open positions for PHP internals and performance experts–this part of the stack, at least, is not a black box for them.)

    Unnecessary complexity is a drain on resources, and a startup can’t afford that. While we’d love to hire some OCaml internals and performance experts because they’re probably great hackers, we hope that by keeping things simple we can spend most of our time at the top of the stack, writing the applications that are our business.

  • Michael Lee on June 18th, 2007 9:06 pm

    Hey Jake

    This is a rather odd set of posts. And not to belabor the point but Isn’t what you’re saying somewhat obvious? Who would disagree with the following statements: abstractions have costs directly and indirectly associated with them; introducing complexity into a computer system requires justification.

    Your statement, “… we take on complexity without justification.,” may be true but usually it is justified to some degree. It may be long or short sighted or just plain wrong–lack of imagination, experience, etc… Regardless, there’s usually a justification.

    Anyways, good luck with your startup.

  • Jake on June 19th, 2007 7:25 am

    Hi Michael. What I’m saying may be obvious to some, but it certainly bears repeating. As you point out, sometimes the “justification” for overly complex systems is mistaken.

  • Mike "Szii" on June 26th, 2007 1:40 am

    While it looks horrendous, the reusability is there. Rolling your own is great, if you have the QA staff and time (and time-to-market!) to back it up.

    When you need to execute fast, prepackaged code is a great way to do it. While “fast and tight” is an awesome goal, sometimes you just have to say “you know, I just need it to WORK. I can optimize later if it’s a huge problem.” You can’t be blind about it, but if you try to optimize everything, everywhere you’ll optimize stuff that doesn’t really need to be optimized.

    If the run-time doesn’t suffer, doesn’t choke the hardware when you scale it…who cares how big the stack is? After all, you just scan right over it with the scroll bar. /shrug

    In perspective, something like Gentoo is arguably the “fastest” linux distribution due to the compilation tweaking you can do. But, really, how many do it? Especially compared to the number of RH/Debian boxen that run quite well? And how many Gentoo “oopses” did you go through (rolling your own) to get some semblance of speed over a RH/D distro in which you didn’t REALLY need that speed anyways?

    IMHO, chase run-time and not stack-length. You can overthing/overengineer/overtweak anything. The skill/art/knowledge is not always what to tweak but what NOT to tweak. (Not that I know it myself! :)

    Found this through a Craigslist posting. Haven’t been in the web world in a number of years (re, CGI and PHP 1.0), so going to go check out this OCaml stuff. Good luck! Glad to see people are still doing startups here in the Valley! :)

  • Dave on July 6th, 2007 1:37 pm

    I could not agree with you more. I have never been a fan of frameworks. I feel that in most cases they add unneeded complexity and abstraction for systems that have no need for it in the first place.

    –Dave

  • Marc Conley on August 1st, 2007 12:53 pm

    I am a senior J2EE developer (25 years experience in the field of software development…lots of paradigms/languages under my belt). I agree with your assessment of Java/J2EE community over-complicating things in an attempt to follow prevailing design/architectural wisdom. I personally feel that a good developer doesn’t need near that much imposed structure to do the right thing. And someone is paying for all those extra layers and design work — the customer. I personally prefer a language called newLisp over ocaml, although ocaml looks pretty cool as well (especially compared to Java). I am located in GA and can’t move or really switch from my full-time gig currently — it’s too good an opp. But I am interested in startups and equity and those kinds of things. Been there, done that…wouldn’t mind doing it again. :-) It can be a very exciting and creative and powerful environment where one can show their stuff and be rewarded for it accordingly.

  • Victor on August 6th, 2007 12:43 am

    I totally agree that the Java for Web is somewhat overcomplicated.

    Once, I have written a small web-app in Python+Clearsilver+(Sqlite and PostgreSQL afterwards), as a CGI app, and then as a mod_python on Apache-HTTPD-2 one. This web-app processed user input, queried the DB, constructed the HTML on the fly (out of more than 300 rows of data fetched from the DB) faster than the Tomcat loads its default greetings page.

    Next time I would like to try the OCamlnet, though, from my point of view, it lacks some features.

  • Paul on November 17th, 2007 1:45 am

    I’m curious, what do you use for generating html?

  • Raoul Duke on February 14th, 2008 4:35 pm

    I wouldn’t mind a framework if it were really honestly a good pay-as-you-go model. On the other hand, p-a-y-g taken too far means you are working in C++ with no garbage collection, yucky-poo.