I’m not a big fan of the Zope 3 Component Architecture.  A couple packages came out recently that made it clearer to me why.

The packages in question are repoze.bfg.httprequest  and r­epoze.bfg.restrequest­.  restrequest is easier to understand because it is more concrete: basically it adds marker interfaces to your request object for the request method, and you can add request_type=IGetRequest to your view declarations, hence declaring that a view is only applicable to that request method.  You can do this with a decorator like:

@bfg_view(request_type=IGETRequest, for_=models.IMyModel)
def my_view_get(context, request):
    return Response("GETn")

Right now repoze.bfg (a web framework) uses interfaces to do it’s dispatch to views, so this adds the ability to dispatch on request method.  The existence of these packages bothers me a great deal, and it took a while to figure out why.  These are entirely packages, with docs, tests, etc., that express something trivial.  If repoze.bfg supported conditionals on views — functions that would indicate if the view was applicable to a request — then you could express this like:

@bfg_view(for_requests=lambda req: req.method=='GET', for_=models.IMyModel)
def my_view_get(context, request):
    return Response("GETn")­

This is basically what Routes has done, though it additionally added request-method-dispatch as a primitive (just a keyword like method=’GET’).  I think this is a much nicer construct, and no extra packages are required.  More importantly, it doesn’t just read well (arguably IGETRequest reads a bit better), but it is easy to understand the mechanism and to extend or adjust that mechanism.  If you want to dispatch based on the Accept header, you just use for_requests=lambda req: ‘application/xml’ in req.accept.  If you don’t want to write lambdas, you can just abstract this in the obvious way in a separate function or a class with a __call__ attribute.

OK, so that’s simple enough: repoze.bfg should support this construct, and it’s just an unfortunately and hopefully temporary awkwardness that caused these libraries to be written.  In theory anyone could, right now, write an alternative implementation of @bfg_view that accepts this.  Of course, anyone could spend an hour or two and write a new framework from scratch, which makes this a less compelling argument: if someone decides to use repoze.bfg instead of writing their own framework or using a different framework, it’s not because they want to write or maintain their own infrastructure, or be in the situation where they’d ask “can you help me, but oh by the way I’m not actually using your stuff”?

The other reason for this, I think, is ZCML.  Here’s another way of expressing method-based dispatch:

<bfg:view for=".models.IMyModel"
      request_type="repoze.bfg.restrequest.interfaces.IGETRequest"
      view=".views.my_view_get"
      />

There’s no place to put functions into ZCML, but there is room for interfaces.  And more generally, interfaces are the tool that comes to mind most commonly when using the Component Architecture.  I think it leads to this kind of anti-pattern all too often, the use of marker interfaces and the accompanying indirection when a simple function would do.  There’s a claim that interfaces allow for more flexibility, but this is simply false: functions are completely general and ameniable to any kind of abstraction you want.  Interfaces add indirection, and sometimes that’s useful, but in many cases like this the indirection is added before there’s any need for it, and when there might never be a need for it.

It can be hard for me sometimes to articulate what bothers me about Zope 3-style architectures, because it’s hard to point to what exactly the alternative would be.  Usually the alternative is: a few lines of code.  Code is a powerful and extremely general abstraction, and especially in a dynamic language like Python the amount of abstraction you can add is substantial.  And you don’t need to add the abstraction until you need it.  So usually the alternative I see to the Component Architecture is simply a few lines of code, written for the specific case that is at hand.

Filed October 20th, 2008 under programming
  1. the smell is real here, but I think this is more of an bad engineering choice than a real indictment of the zope component architecture. For someone who has never understood any use for the zope component architecture, I can understand how it might be hard to distinguish good use from poor or for the temptation to use poor use as an argument against all use.

    in this case, it seems poor encapsulation… iow, not handling all the sensible cases inside the decorator but rather pushing the abstraction out to the level of demanding interfaces. By not supporting the simple case first, a developer is forced to deal with unnecessary abstraction.

    (Also functions and arguments could be added to the zcml by extending the directive, but there seems little reason to do that as the function would be more direct and easily readable. )

    otoh, the basic structure these libraries are trying to extend is pretty sound (the bfg view). zca is a sensible way to bind views to a variety of model objects, as the rendering of a view is highly dependent on the specific provisions of the context object for which it is rendered and using zca eliminates lots of arbitrary conditional code or adhoc registery for extension. And this sort of dispatch works well for a certain set of usecases (if not all the cases supported by say routes).

    -w

    Comment by whit on October 21, 2008 at 10:50 am

  2. Maybe a more specific thing I dislike is marker interfaces. I don’t think they are interfaces at all, but are just a way of using the interface mechanisms for purposes of indirection. I don’t think I’ve seen an example of marker interfaces that didn’t exude some kind of code smell. Of course sometimes development has constraints which make a bit of code smell the best response, but you have to recognize the tension to make the correct choice in each instance.

    I also think marker interfaces are generally less necessary when you delegate fully to code. That is, you can use something more akin to a “hook” (or event, or something else), where some code is entered, then exits, and the normal machinery continues on its way. If that hook can’t do quite what it wants at that time, it can set a marker and some other code later on can pick it up. I think wrappers work better than hooks because they don’t lead to this situation — when you call something, you should expect but not require that it will continue going through the normal mechanisms. This gives the code being called more power over how things will transpire, including most importantly branching. If your code is entered and exits, then it can’t do two completely different things… in the end all it can do is exit.

    Generally I think wrappers require slightly more code (since there’s no default of continuing the normal chain of calls), and potentially they can lead to a separate class of errors (if the wrapper doesn’t fulfill its now-larger contract), but it’s worth it because of the considerably larger amount of power they give the programmer, and the much more direct, localized, and code-based logic involved.

    Comment by ianb on October 21, 2008 at 2:22 pm

  3. I agree with you that marker interfaces are often abused to provide instance “identity” or differentiation where a simple text string would do (or in the case of the request method, a text string is *readily available*). The mechanism of dispatching on a string is not a common paradigm for some zope programmers, so frequently it is overlooked as an option.

    Part of the historical context for repoze.bfg.restrequest probably came from the fact marking the request in relation to a pattern in the path to denote some sort of change in display is a common pattern in zope3. This is actually a fairly powerful pattern as it allows wholesale UI change w/ selective inheritance if desired. The utility of the marker interface in this case is fairly elegant, considering the consistency that it allows in general for managing the problem of ui. Doing this via hooks and wrappers would probably lead to convergent evolution fairly quickly. Similarily, in zca based frameworks, it is a ridiculously easy way to add radically different behavior to a single context without having to anticipate ever needing such a “hook”. Granted, this utility is in the context of a greater framework, not the building of a framework or an application.

    I think the reason people who have no love lost for the complexity of the zopes (like mcdonc) continue to use pieces of the component architecture is that it does provide a well tested, sensible way to do deal with component registration by configuration for wrappers and hooks (because that’s all the CA is) that beats reinventing the wheel each time. and alot of the time, beats entry points in setuptools for it’s ability to bind specific behavior to specific classes. It’s not to say it doesn’t get used like a hammer alot in the zope/plone world (I’ve been guilty of that as much as anyone), just that it does have a place and use that not much else fills very well.

    -w

    Comment by whit on October 23, 2008 at 11:54 am

  4. For the simple cases, I think the component architecture is a lot of overhead for not much gain. Like you state above, it is more indirect, harder to understand, and doesn’t buy you much for easy cases. There have been may times when I’ve encountered utilities, events, adapters, annotations, whatever, and thought to myself: “Really? Why can’t I just use a function here?” Instead, I’ve had to “trick” some other code into doing what I wanted. And plenty of time it’s my own code that I’ve had to trick.

    My argument for the zca would have to be that it provides a consistent registry for programmers. For example, any time you have several strategies, you’re going to want to configure those somehow. Instead of creating ad-hoc dictionary-like structures that may balloon to something more complex down the line, the zca provides a consistent mechanism to solve this. Programmers familiar with the zca can understand and follow how and where behavior is getting chosen, without having to learn custom registration systems.

    In the request type example above, we can extend that same pattern for some powerful effects. Say I was creating a third party package that was designed to theme some common content types. I could register all my views under the IAwesomeTheme request type. Then I’ll create all my views or snippets or whatever under the interfaces for the model. Consumers of my package can now selectively turn these views on or off whenever they’d like by marking the request appropriately.

    That being said, I think the biggest problem is what Whit stated above: it’s a hammer that get used far too often. The zca has some nice cases, but once we use a little of it, we get entangled in it and try to fit everything into that model. I think zope3 itself is the biggest example of this. There are probably many things that could be expressed much more simply without using the zca, but that’s just the m.o.

    To address your concern about marker interfaces, I think that they do happen to solve a particular set of problems nicely. The problem with using a conditional function is that it ties the condition with the “handler”, since they are specified together. The marker interface is more indirect, but this indirection allows you to vary the two separately, giving you more flexibility.

    But when do you really need this flexibility? You’d need it when you varying the two separately is a big deal, which probably means it’s a big deal to change. Perhaps it’s most useful when you need to integrate with some third party code that you don’t maintain. The zca would allow you to override the condition if there was one already there, or confine some code to a more restrict condition.

    I suppose that if you had control over the handler and still wanted this flexibility without the zca, you could modify it slightly to be more functional. The handler itself could take an optional function as the condition. However, if you want to vary these predicate functions dynamically, you’ll have to build your own custom registry for that, and now I’m back to saying that the zca offers you a consistent registration mechanism.

    I also want to offer an alternate perspective on the component architecture that I’ve been thinking about recently. It’s essentially a pattern matching engine, not too far conceptually from that of haskell or erlang’s functional pattern matching. You have an action, foo, and some arguments, a, b, and c. When you invoke foo, it dynamically dispatches to an implementation based on the types of a, b, and c. In the functional languages with pattern matching, you call it like: foo(a, b, c). The separate definitions of foo is what allows the language to pick the right one. With zope, it’s a bit more verbose: getMultiAdapter((a, b, c), IFoo)(). Here the different registrations are what control the matching. It’s not as powerful, because the pattern matchers are able to take advantage of more of the language during the matching process without having to mark things with interfaces, but I think it’s the same basic idea.

    Anyway, I strongly believe that most of the time we go too far with these patterns. Using a more functional programming paradigm should get us most of the way there. In fact, it probably gets us all the way there for most cases. But in the complex cases, we need to build our bridges in a consistent way for larger systems to scale. I think the zope component architecture is just an attempt at standardizing those bridges. There are other ways too, and I’m thinking of middleware as an example, but I think these two tackle it from different angles.

    I think the point that I’m trying to make is that each one of these approaches is good for a particular type of problem, and none of them are good for all problems. I’ve talked about this here: http://robmarianski.com/best-tools-for-the-job/ . I also recently thought about abstractions in general here: http://robmarianski.com/the-cost-of-abstraction/

    Comment by Robert Marianski on October 28, 2008 at 3:44 am

  5. Ian, the examples you gave make me very sad. I think this an example of abstraction abuse - a disease some developers get infected with (”let’s use decorators and interfaces for everything!”) What makes me sadder, though, is that I actually like the Zope 3 Component Architecture - but not when it invades every aspect of a codebase - i.e., your typical Zope-centric package … Or when it’s simply overkill to use - as with the IGetRequest, IPostRequest nonsense.

    Let me offer an example where the Zope 3’s CA or any CA, for that matter, can be very useful - and not leave code smells lying around. Plugins! Having an interface for a component, and then being able to easily adapt one object to a provider of the interface is very powerful. Authentication comes to mind:

    def myview(request):
    user = IAuthenticatedUser(request)

    Using adapters setup in a plain old python module specified in your setup (.ini file or whatever - anything but friggin’ ZCML) you can customize what authentication methods to use (username/password, kerberos, etc.) and the application code (myview) doesn’t need to know about this through module imports, etc. I find that to pretty nice, and it makes writing custom components less of guessing-game - just implement the interface.

    Comment by Drew Smathers on October 30, 2008 at 9:29 am

  6. I’m one of the perpetrators behind the “repoze.bfg.httprequest“ package and while I agree with Ian that this particular piece of software might work well as a showcase for how not to use the ZCA, I think some of the commentators misunderstand the objective.

    In any web framework, there’s a request coming in. Some where down the code chain, decision are made based on what this request is all about, perhaps foremost based on its path and query string.

    However, it may also depend on such things as language, method and the type of data the request is looking for in a response. We’re talking about intrinsic qualities of the request, not details like which theme to apply, or other application-specific qualities.

    Now, in Zope 3, besides writing a long list of if-elif-else statements to multiplex on these, for any given object and request, to find a function that will actually render the response, there is the possibility of using marker interfaces and use the ZCA to multiplex.

    This may seem like needless complexity for some, but developers who routinely use the ZCA will find it easy to understand and further, they’ll have a complete registry for these components, which may be queried in any way they like. It’s fast, too, being mostly written in C. That’s not good for the GAE, but that’s another matter.

    I agree that marker-interfaces are often an anti-pattern and their binary quality is too simple to handle any but the most rudimentary use-cases. But consider this example (in the context of a “repoze.bfg“ app):

    A request comes in; it’s an ajax request, which may readily be seen from the value of the HTTP header “http_x_requested_with”, which is “xmlhttprequest”. To register a view “show” for such requests, one would register a generic view function for this view and multiplex to two other views, “show_full” and “show_ajax”.

    Alternatively, we just register our ajax-view for an interface which matches the corresponding HTTP header. If we use “repoze.bfg.httprequest“ to create this interface, it’s automatically applied to the request when it comes in. No setup needed. It’s even quite readable, as long as the interface as properly named, e.g. IAjaxRequest.

    In conclusion, programming with Python is about having fun and not working too much; this means writing good code and being able to reuse that code often. The test of time will prove whether this package was a fluke or not. I’m still using it :-)

    Comment by Malthe Borch on November 7, 2008 at 5:44 am

  7. I think you are missing something important in your example: that the lambda functions you are suggesting are terribly expensive, because the framework cannot “get inside of them” to see what they say. They are black boxes; if a request comes in from the Web, then *any* of the views in the *entire system* could be ones whose lambdas will evaluate “yes” and want to handle the request. Therefore, the cost of selecting a view is O(n) over the number of views registered. If you have loaded libraries that, together with your application, register a few hundred views for various content types, then you’ve essentially got a runtime-constructed if-else cascade of 120 decisions that tries to find the view that should handle the request. That’s expensive.

    The actual mechanism that repoze is using, by contrast lets the framework “see” the decision you are trying to make — that it has to be a certain request type combined with a certain class of content, say. The framework can take all hundred registrations it might be offered and compose them into a decision tree that descends right down in roughly O(log n) time (or even less, if schemes based on Python dictionaries are in play) to the view that needs to be selected.

    So, efficiency is problem #1; but there’s an even deeper problem #2 in which your scheme is simply an *impossible* solution: where the developer wants to write a “general view” that handles all Documents, but then wants to override that with a *special view* that specifically handles PDFDocuments differently since, say, he does not render those to HTML. With your scheme, the distinction is simply impossible to make! When a PDFDocument gets a GET, then *both* lambdas will fire, and since all the framework will get are two True values, there will be NO WAY that the framework can even POSSIBLY determine that the PDFDocument match “binds more tightly” than the Document match and should override it.

    By contrast, the ZCA is built specifically to take registrations on classes and return the “most specific” one. It might be more complicated than it needs to be, I’m not sure; and I certainly have not learned all of the rules about what is a “more tight” binding when doing things like multi-adaptation. But it at least is making things possible which your supposedly “completely general” solution completely disallows! Your solution, in other words, is not general at all, because — in its current implementation — Python does not allow a framework to “look inside” boolean lambdas to compare matching booleans to see which is more specific.

    Comment by Brandon Craig Rhodes on November 7, 2008 at 8:42 am

  8. Malthe: In complicated circumstances I’m personally apt to simply write my own dispatcher. In an ideal world this is both pretty easy, and can be done imperatively (imperative code is almost always easier to think about than declarative, in my experience — and I say that as someone who is constantly fighting a natural inclination to write declarative code). Also dispatch can happen at multiple levels. A typical way to handle Ajax vs. non-Ajax calls would be through a custom decorator or class (in Pylons you’d put it in your Controller subclass, in TurboGears probably in some decorator). This essentially does a second dispatch — but it also has the advantage that you can easily put special hacks in that dispatch. For instance, maybe your dispatch only relates to HTML, so you can ignore it for non-HTML responses. Maybe you’ll look up a special method or add an argument to the signature of the function, but you’ll want a fallback for views that really just don’t care about the Ajaxness of the request (e.g., something that returns JSON regardless of who asks for it). Maybe in summary, I’ll respond to your claim:

    “In any web framework, there’s a request coming in. Some where down the code chain, decision are made based on what this request is all about, perhaps foremost based on its path and query string.

    “However, it may also depend on such things as language, method and the type of data the request is looking for in a response. We’re talking about intrinsic qualities of the request, not details like which theme to apply, or other application-specific qualities.”

    I don’t think the framework has to handle all these cases. Dispatch should not be at the sole discretion of the framework — it should always be possible to add dispatch to your application. Abstraction can happen at any layer. If you have these needs, writing a dozen lines of code to handle that is the simplest and easiest to maintain method (and really none of these use cases call for more than 10-20 lines of code).

    Brandon: I suspect for most cases (maybe all non-diabolical cases) that the constant factor in the O(n) vs. O(log n) cases overwhelms n, and that the simpler solution will be faster. In a dispatching system like Routes the n will almost always be small, because most potential routes will be eliminated due to path matching (which is quite fast), and the functions are only run on a handful of routes. If you have a system resistant to path matching then it’s often quite reasonable to do some other kind of dispatching in code, e.g., route different requests through a single “view” that then does some dictionary lookups and function calls to dispatch to a real view. Anyone can make a dictionary of functions, you don’t need to be a framework wizard to do so.

    As to generality: In Routes the matchers are ordered, so if you have a more specific match then you must order that before the less specific match. If you have the code in hand (which in Pylons you do have) then this is trivial (though admittedly it’s easy to get the ordering wrong, it’s something I do fairly often). Otherwise you don’t have that code then you do have to figure out some way of indicating priority. ZCA provides one way of doing that through the specificity implicit in inheritance — but that doesn’t make it *easy*. Does IPDFDocument subclass IGETRequest? Probably not, and then you are in just as annoying a place. Again, when things get complicated, writing your own dispatch using dictionaries and functions is easy: do more of that!

    Comment by ianb on November 7, 2008 at 5:08 pm

Leave a comment