Topp Engineering

We have expertise in a number of different frameworks, but one that hasn’t caught on much historically at topp has been django. I can think of a variety of reasons for this, but despite these, I think taking a fresh look at django is worthwhile for topp.

Django comes with a lot of batteries included. A lot of core decisions have been made about how things interact and connect. What you gain for buying into those decisions is a whole lot of functionality. These core decisions allow many higher level pieces to be written, which adds up to a whole lot of functionality quickly.

On top of this, pinax has been gaining more momentum as well. It adds another layer on top of the already rich django core, which provides many of the features that typical web sites demand. There’s a good introductory talk to pinax here: http://blip.tv/file/1952623

I gave pinax an honest shot, and there were a couple of things that impressed me about it. First of all, the build worked consistently, and so did all of the project templates that I’ve tried, probably because they’re based on pip and virtualenv Secondly, any kinds of trivial modifications that I attempted, were in fact trivial to do. It almost felt like a cms to me, but if I wanted to change some text, style, functionality, you name it, it was pretty easy to do so without having to understand any complex systems.

In the talk linked to above, it’s mentioned that one of the reasons for pinax’s good integration story is that most cms’s typically have an underlying system for everything, and then you write your own hooks to plug into that system. Pinax inherits the core underlying system from django, but instead of building up its own system, it flips the perspective around and tries to provide all the pieces that are needed, and controlling how those pieces connect is application specific. In the end, most of the common changes are intuitive.

Specifically to topp, I think django/pinax is important for a number of reasons:

1. It’s popular, and it looks like that popularity is growing. If we want to build tools the masses will use, we should work with the masses, instead of trying to convince them otherwise.

2. It has a strong geo component. While it’s certainly possible with other frameworks and back-ends, django has very good integration with its orm geo-wise.

3. Not only are the batteries included, it comes with several flash lights to choose from.

It looks like we’re moving in the direction of creating more prototype style web apps. With many of these apps, we try to focus on what makes it unique, and don’t really care much about the other mundane details. Interestingly enough, this is in fact pinax’s tagline.

To sum up, topp is still imho a light, flexible organization capable of changing directions quickly. While we figure out the answers to some of our tougher internal questions, we may very well need to move quickly, in a hurry. Taking advantage of django/pinax can be one way for us to do so.

I attended a great repoze.bfg sprint last weekend. We discussed creating a foundation on top of repoze.bfg that would allow for a complex content management system to be built. Tres wrote down the requirements of an example system here

To summarize, in a typical large content management system, there are multiple sources of content. Some can come from rss feeds, others are authored, and still others are media on a filesystem. Frequently, the various types of content come from different departments, and they each have their own applications/workflows to work with their content.



We divided up the entire application into several sections: the content authoring environments, the content management system, and the retail, or client application.  There can be several content authoring environments, which represent the mini applications that are geared towards a particular type of content. The content repositories would expose a thin interface that would interact with content through content_ids unique across the system. The content management system manages how the content appears on the retail site. This involves url structure, layout of the content, and who gets to see what. And the retail application itself is the end user front of the entire application.

The code that was produced can be currently found here: http://svn.repoze.org/plite/trunk/

The name is not set in stone, so if it changes, then that link may break.

Creating a generic interface for a complex system is challenging. Going through interfaces that are too thin make client code more difficult, while making the interface too specific can preclude certain content repository implementations. We began creating the various pieces of the application in hopes that they would help drive certain interface decisions.  When it was time for me to go, a basic content repository was created that would store files/pages. The content management/retail aspects were yet to be fleshed out, although they’re probably further along now.

All in all, I found that I got a great deal out of the sprint. I personally haven’t had to work with sites of this size/complexity, so it was very useful for me to hear the different opinions on design tradeoffs.

The repoze.bfg framework itself had gotten better in a number of ways since I last looked at it. If you haven’t already, it’s definitely worth checking out. The code itself is of extremely high quality, being very clean and organized. Additionally, it’s very well tested.

I look forward to seeing everyone at the next sprint!

As part of the GeoTrac project ( http://projects.openplans.org/GeoTrac/ ), I set off to be able to query on tickets via shapefiles ( http://projects.openplans.org/GeoTrac/ticket/78 ). I didn’t know anything about shapefiles before beginning this, and I still don’t know much, except that I’m not a big fan. So I consulted the font of knowledge: http://en.wikipedia.org/wiki/Shapefile

Shapefiles are several files that have different extensions but the same “first name”. Its the kind of convention someone would come up with if they were being “clever” (see http://www.statusq.org/archives/2006/11/03/1193/ ). Three files are mandatory:

* .shp — shape format; the feature geometry itself

* .shx — shape index format; a positional index of the feature geometry to allow seeking forwards and backwards quickly

* .dbf — attribute format; columnar attributes for each shape, in dBase III format (from http://en.wikipedia.org/wiki/Shapefile )

Unfortunately, these three files contain no projection information and so do not actually give you enough information to tell where the geometry is on a map. There is an optional .prj file * .prj — projection format; the coordinate system and projection information, a plain text file describing the projection using well-known text format (from http://en.wikipedia.org/wiki/Shapefile ). So while we didn’t have a .prj file to start off with or know the projection that we needed to figure out where the geometry lived on the map, this seemed the right approach to specifying geometry.

We wrote a Trac (see http://trac.edgewall.org ) component for the GeoTicket plugin ( http://trac-hacks.org/wiki/GeoTicketPlugin ) that will support querying tickets based on region: http://trac-hacks.org/svn/geoticketplugin/0.11/geoticket/regions.py

The import of the shapefile is done with shp2pgsql ( http://postgis.refractions.net/docs/ch04.html ). The resulting SQL is then read and used to create a table in the PostgreSQL database. The administrator of the GeoTrac site is then prompted which piece of metadata attached to the polygons should be used for query. The import was fairly easy to do. Next came the query itself. A SQL query was used to select the tickets within the region:

SELECT ticket FROM ticket_location WHERE ST_CONTAINS((SELECT the_geom FROM georegions WHERE commdist=212),
st_pointfromtext('POINT(' || longitude || ' ' || latitude || ')')); 

However, without specifying the SRIDs, this gives an error:

ERROR: Operation on two geometries with different SRIDs CONTEXT: SQL function "st_contains" statement 1 

This may be corrected by providing the SRID for the ticket location point:

SELECT ticket FROM ticket_location WHERE ST_CONTAINS((SELECT the_geom FROM georegions WHERE commdist=212),
st_pointfromtext('POINT(' || longitude || ' ' || latitude || ')', 4326)); 

However, by default, the SRID of the_geom is -1. So this gives the same error, and the previous case silently “succeeds”, though of course the projection is wrong you won’t get any data back. You can see the SRID from the geometry_columns table:

trac_fleem=# select * from geometry_columns;
f_table_catalog | f_table_schema | f_table_name | f_geometry_column | coord_dimension | srid | type
----------------+----------------+--------------+-------------------+-----------------+------+-------------- |
                  public         | georegions   | the_geom          | 2               | -1   | MULTIPOLYGON
(1 row) 

As a makeshift solution, I’ve included a text input to specify the SRID:

Import Shapefiles Upload Shapefiles Shapefile shape format (.shp) ____________________ Shapefile shape index format (.shx) ____________________ Shapefile attribute format (.dbf) ____________________ SRID ____________________ Submit

So, given the SRID, you *should* be able to transform the_geom to the SRID you care about, using `shp2pgsql -s`. Maybe. I haven’t finished the query yet [TODO!].

So the problem remains of how to get the SRID from the .prj file. After asking, we were given the .prj file:

PROJCS["NAD_1983_UTM_Zone_18N",GEOGCS["GCS_North_American_1983",DATUM["D_North_American_1983",SPHEROID["GRS_1980",6378137,298.257222101]],PRIMEM["Greenwich",0],UNIT["Degree",0.017453292519943295]],PROJECTION["Transverse_Mercator"],PARAMETER["False_Easting",500000],PARAMETER["False_Northing",0],PARAMETER["Central_Meridian",-75],PARAMETER["Scale_Factor",0.9996],PARAMETER["Latitude_Of_Origin",0],UNIT["Meter",1]] 

Ideally, in addition to allowing the GeoTrac administrator to specify the SRID (especially for the case where there is no .prj file), it would seem plausible to obtain the SRID from the .prj information.

So I looked around and tried a bunch of things and was refered to a project called GDAL: http://gdal.org/ . Specifically, I was told that ogr2ogr would be useful for shapefile vector data ( see http://gdal.org/ogr2ogr.html ). GDAL has python bindings ( http://trac.osgeo.org/gdal/wiki/GdalOgrInPython ), but unfortunately, it is not easy_install-able:

(Trac-2.4)> easy_install GDAL
Searching for GDAL
Reading http://pypi.python.org/simple/GDAL/
Reading http://www.gdal.org
Best match: GDAL 1.6.1
Downloading http://pypi.python.org/packages/source/G/GDAL/GDAL-1.6.1.tar.gz#md5=e8671d4a77041cf0f7a027f3f3e8280c
Processing GDAL-1.6.1.tar.gz
Running GDAL-1.6.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-jFd9bm/GDAL-1.6.1/egg-dist-tmp-MJllDq numpy include /home/jhammel/Trac-2.4/lib/python2.4/site-packages/numpy-1.3.0-py2.4-linux-i686.egg/numpy/core/include
Could not run gdal-config!!!!
gcc: error trying to exec 'cc1plus': execvp: No such file or directory
error: Setup script exited with error:
command 'gcc' failed with exit status 1 

I haven’t been able to compile GDAL by hand, though it is available on modern package systems. So at best, requiring GDAL makes a component optional (see http://peak.telecommunity.com/DevCenter/setuptools#declaring-extras-optional-features-with-their-own-dependencies and the terribly apt http://www.reddit.com/r/Python/comments/91bnb/the_terrible_magic_of_setuptools/c0b3lhe ).

So how do you use ogr? An example is given here: http://www.gis.usu.edu/~chrisg/python/2009/lectures/ospy_demo1.py

So we try this on our own shapefile:

>>> import ogr
>>> driver = ogr.GetDriverByName('ESRI Shapefile')
>>> ds = driver.Open("sen02.shp") 

Docstrings are not available for much of the python bindings:

class DataSource |
Methods defined here: | |
CopyLayer(self, src_layer, new_name, options=[]) | |
CreateLayer(self, name, srs=None, geom_type=0, options=[]) | |
DeleteLayer(self, iLayer) | |
Dereference(self) | |
 Destroy(self) | |
ExecuteSQL(self, statement, region='NULL', dialect='') | |
GetDriver(self) | Returns the driver of the datasource | |
GetLayer(self, iLayer=0) | Return the layer given an index or a name | |
GetLayerByName(self, name) | |
GetLayerCount(self) | Returns the number of layers on the datasource | |
GetName(self) | Returns the name of the datasource | |
GetRefCount(self) | |
GetSummaryRefCount(self) | |
Reference(self) | |
Release(self) | |
ReleaseResultSet(self, layer) | |
TestCapability(self, cap) |
Test the capabilities of the DataSource. |
See the constants at the top of ogr.py | |
__getitem__(self, value) | Support dictionary, list, and slice -like access to the datasource. |
ds[0] would return the first layer on the datasource. | ds['aname'] would return the layer named "aname".
| ds[0:4] would return a list of the first four layers. | |
__init__(self, obj=None) | | __len__(self) | Returns the number of layers on the datasource 

So it is hard to tell how to get the SRID.

A manual solution is available at: http://help.nceas.ucsb.edu/PostGIS#Import_a_shapefile_into_a_PostGIS_database

The key is the first string, NAD_1983_UTM_Zone_18N. If I do a query like the one in http://help.nceas.ucsb.edu/PostGIS#Import_a_shapefile_into_a_PostGIS_database I get a row:

trac_fleem=# select * from spatial_ref_sys where srtext like '%NAD83 / UTM zone 18N%';
srid   | auth_name | auth_srid | srtext                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | proj4text
-------+-----------+-----------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------
26918  | EPSG      | 26918     | PROJCS["NAD83 / UTM zone 18N",GEOGCS["NAD83",DATUM["North_American_Datum_1983",SPHEROID["GRS 1980",6378137,298.257222101,AUTHORITY["EPSG","7019"]],AUTHORITY["EPSG","6269"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.01745329251994328,AUTHORITY["EPSG","9122"]],AUTHORITY["EPSG","4269"]],PROJECTION["Transverse_Mercator"],PARAMETER["latitude_of_origin",0],PARAMETER["central_meridian",-75],PARAMETER["scale_factor",0.9996],PARAMETER["false_easting",500000],PARAMETER["false_northing",0],UNIT["metre",1,AUTHORITY["EPSG","9001"]],AUTHORITY["EPSG","26918"]] | +proj=utm +zone=18 +ellps=GRS80 +datum=NAD83 +units=m +no_defs
(1 row) 

In other words, I can tell, as a human being looking at the file, that the `WHERE srtext LIKE` clause should be transformed from the string “NAD_1983_UTM_Zone_18N” to the string %NAD83 / UTM zone 18N%’. But while you could tell a computer how to do this….it would either be hard, or I don’t know how to do it (and couldn’t find any good documentation).

One of the takeaways from this is that this is hard and it shouldn’t be. I don’t know enough about the Geo world to say exactly why this is. Maybe the .prj file should be able to support SRIDs. Maybe `shp2pgsql` should figure out the SRID from the .prj file. Maybe the string should be canonical so that select from spatial_ref_sys query could be more robust, assuming I figured out how to parse the .prj file.

The problem

Javascript usually applies behavior to elements on page load. Usually, there’s a list or some sort of registry of behaviors, and it’s iterated through on document ready time, and applied then. For elements that get created by javascript though, behaviors don’t get automatically applied to them. The application of those behaviors happens only at document ready time, and new elements don’t get processed magically.

Opencore’s solution

We’ve handled this in the past by explicitly calling a generic function that would apply the behaviors to elements. So when a new element was added, we would follow up the addition with an explicit “breatheLife” call. This worked, but we had to remember to call the breatheLife function every time we added elements that we wanted particular behaviors attached to.

Jquery solution

Jquery has a particular solution for this problem for a particular subset of behaviors: live events. The solution however, is a little bit different from ours. Instead of applying behaviors to new elements as they are attached to the dom, jquery gets the event as it bubbles up, and then figures out if its coming from an element that has special behavior. This technique is called event delegation. Currently, not all event types are supported, but a common set are.

Other applications

Using event delegation is also useful for situations when there are a lot of elements that need behavior applied to them. For example, if we had a huge table of links, or a large number of list items, we could add a handler to the table or unordered list, and figure out the target of the event from there. Instead of adding an event handler for each element, we can get away with only adding one that can handle all of the elements.

­Almanac ui

We’re nearing the end of another community almanac development cycle, where we changed the ui quite a bit. Here are some of my thoughts on it.

The operation that we wanted to optimize for was contributing a page to an almanac. We used to call them “stories” but we tried to stress the book/page metaphor a little bit more now. We also tried to make sure that adding a simple page was very easy, while also providing flexibility for the user.

Previously, we had taken a wizard-like approach for this case. A user was presented with a series of questions, taking them through the process of adding their story. This forced the user to go through the process of adding a story in a very strict way, and presented more questions to the user than was usually necessary, at least for the simple cases.

A different approach

This go around, we took an approach that feels a bit like basecamp. Instead of providing a wizard, or series of questions that the user should enter, the user is presented with a set of tools that control the type of content that the user is contributing. This allows the user to add the type of data that they are interested in, without forcing the concept of “metadata” to the user.

An advantage with this model is that it offers more flexibility for creating pages. Users can add the types of data that they want, in a way that makes the page flow more naturally. They can now have several locations, descriptions, or none at all. It also feels a lot more like what the site should be all about: adding pages to a community’s almanac.

Under the hood

Implementation wise, this means that we have to be able to generate lots of forms on demand through javascript. There are a couple of ways to manage this, but we decided to have the server generate these forms, and the client side would issue ajax requests to send/receive all data. This keeps the javascript simple, with the server handling most of the logic.

This is similar to how we handled opencore’s javascript. The client sends a request to the server (usually getting the url from an anchor or the form that the object is in), and knows how to handle a couple of different responses in a generic fashion. Usually the variations are simply to take some html returned from the server and put it somewhere, either by replacing an element, or adding it somewhere else on the page.

There was an additional complication on the almanac side, which was that we needed to apply some side effects for the content to be displayed properly. This is sort of like the “breathe life” problem, except we needed it to simply display the content properly, as opposed to interacting with additional events. For example, when a map is returned, some javascript needs to run to display the map correctly, with the right features displayed on it. For audio playback, some flowplayer behavior needs to get applied to it before it can play in the browser through flash. We did also have to tackle adding behavior as well (so the forms that got returned where submitted through ajax), but we were able to use jquery’s live events for a lot of it.

Always something

Since we’re loading a lot of javascript on page loads as well, we ran into an issue where you could click on a link before it got its document.ready behavior applied to it. The user gets presented with a download dialog box, asking to download the json return value. We worked around this by having empty onclick handlers in the markup itself, which prevented the dialog boxes. Ideally we would have non-javascript fallbacks everywhere, but we didn’t focus on dealing with that issue because a large portion of the site deals with map interactions, which isn’t really feasible without javascript. You can say that you can have page loads for each click, like panning or zooming, but drawing features on a map without javascript is tough.

Thoughts for the future

The similarities of the ajax approaches of both almanac and opencore got me thinking if these are both more specific instances of a general pattern. Kss as a solution comes to mind. What worries me about kss however is that it seems focused on completely removing all javascript from the application. But on the other hand, it seems like the concept of applying js behavior to elements in the same kind of way as style is applied to elements is a reasonable approach. And kss is a way of standardizing the behaviors, in the same sort of way that styles standardize on the display. In fact, most of the javascript that I write involves selecting some elements, and applying a simple behavior to them. Behavior will probably vary a bit more than style, but I definitely prefer having a solution for most cases than no solution at all. In any case though, I definitely think kss is a cool concept.

­

At this year’s pycon, I participated in the wsgi sprint. We shared the room with some of the zope guys, and it was good to see some familiar faces. There were a lot of projects going on, and I participated in just a couple of them:

mongodb

There was a buzz in the air about mongo. I’ve heard good things about it from a number of people. I paired with Reed on getting a proof of concept working with repoze.bfg. We attempted to simulate the zodb’s transparent nested dictionary structure with it, but I mostly used it as an excuse to play around with mongo I’m still a little bit skeptical as to whether simulating the zodb api can work without having the abstraction leak, but I’m willing to be convinced. Anyway, there are nice python bindings to mongo that make it pretty easy to work with. The mongodb site has a list of use cases where mongodb is well suited.

frameworks

Amongst the python frameworks, there was talk about sharing a common configuration system (minus django of course). I’m all for this if it happens, because it may lead to having the frameworks share even more code underneath in the future.

There was some general application framework exploration as well. It was mostly trying to see if we could come up with good answers to common integration questions. What would ideal client code look like? How would the configuration take place? Can we mix and match behaviors easily? Can we set up a transaction system in a general way that makes it easy to integrate other components with different data stores? What types of services does it make sense for the application framework to provide?

karl

The repoze guys have created an application that is very ploney. It’s still in the works, but there’s a lot of functionality already there. In terms of it offers, it’s very similar to the current incarnation of the openplans suite. I spent some time looking through the code, and was really impressed with what I saw. The code is very clean, well factored, and easy to understand. Naturally, it’s based on repoze.bfg and a number of repoze packages. I think that it’s a great source for repoze best practices. If you want to check it out, the buildout is here (it works last time I ran it!) If people are interested, I can put it up on a demo server as well.

­That pretty much sums up the different projects that I checked out. For some reason ponies, and the django pony specifically, were a central theme. Ask me offline for some more info if you’re interested

I’ll end by saying that it’s always a pleasure just to be around so many smart people at once. There was no shortage of that this year at the pycon sprints!

The question has arisen on two different projects (one hypothetical, one already live) at TOPP in the past week:

Should we (continue to) use ZODB as the primary data store, or should we use … something else?

Well. I’ve been using ZODB in my daily work since at least 2001, so I’m pretty well familiar with it by now. By contrast, I’ve done a little bit of work with MySQL, a tiny bit with other relational systems (SQLite, SQL Server, and Oracle), none with PostgreSQL, and almost no work with any of the current crop of object-relational mappers … nor with any of the hip new kids on the block like CouchDB.

With that background, you’d expect me to be a fairly ignorant guy who always prefers ZODB just because he doesn’t know any better, right?

But there’s also the truism that familiarity breeds contempt. Consider yourself warned about the rest of this post

That said, I’ll start by listing some of the things I really do like about ZODB.

What’s Good about ZODB

• Persistence is very transparent (to your application) … Getting started with ZODB is really, really easy. If your app is not performance-sensitive, you can get away with very little attention to storing your data, it mostly Just Works. And you hardly have to do a damn thing. Neat. (When performance is a problem, you have to start paying a little more attention - or maybe a lot.)
• Schema, shmema. You just throw Python objects in there and they just stick. Neat.
• Undo. When you can use it, this is fantastic to have built in.
• Transactions that mostly just do what you want. Okay, this is really about the default transaction policy in Zope, not ZODB per se. But this is something that I really think Zope got right. (And thanks to repoze.tm, it’s something that any WSGI app can now do with any data store that supports transactions.)
• Text Indexing … there are some nice full-text indexes available for ZODB. On the other hand, these work at the application layer - i.e. you have to do the work of updating the index yourself, it doesn’t happen magically when you save data. And they don’t play nice with Undo. I gather there are now some that you can use without the whole Zope 2 ball of wax, which is great.
• Container hierarchies of arbitrary depth are trivial. This is really nice and something that’s easy for a ZODB user to take for granted. Doing the equivalent in an RDBMS is typically not so fun, I gather. (Haven’t had to do it yet myself, but some quick browsing suggests that it’s really pretty icky. I suspect though that traversing a ZODB object graph will have the same performance characteristics as the fairly simple “adjacency model” described in that article.)
• Scaling is pretty transparent. For a while at least. ZEO is trivial to set up, and to the application, it looks no different than running against a local storage. Multiple mount points (analogous to mounting different physical storage at different directories on a filesystem) also help you scale transparently.

What’s Bad

• Persistence is very opaque (to anything other than your application).

Let’s unpack that a little:

• No ad-hoc queries.

  This is more and more often the showstopper for me.

  With an RDBMS, or CouchDB for that matter, as long as the database server is running, you can poke around and see what’s there. With even a small bit of application knowledge, this can be enormously useful for troubleshooting, quickly repairing data problems, and some simple migrations, to say nothing of actual feature development. With ZODB, you have to know a lot more about the app just to look around and guess what you’re looking at.
  

• You can’t even load the data without exactly the right application software installed.

  This is closely related to the previous point. It doesn’t bite you as often, but when it does, it is NO FUN.

  It’s not just that non-Python applications can’t talk to the database at all, ever. It’s that your database depends on your code too much.

  If you botch an install, or are trying to resurrect a really old one for some reason, sometimes you’ll make a mistake like having a slightly incorrect version of some dependency, such that some container class can’t be loaded. Since the ZODB is strictly a tree structure, there is no way to access any of the children of a broken container instance. Which could, if you are unlucky, translate to all your data.

  Think about that for a second: if you ever lose the ability to perfectly reconstruct your code stack, you also lose your data. Well, you could maybe try to parse something out of raw pickles, but that sure doesn’t sound fun!

  Of course, normally everything’s fine because you have the right software installed. But what if you’re doing forensic work and you don’t have enough information to know what that is? Or what if the build scripts that used to work perfectly no longer work just because some third-party upstream release is no longer compatible?

  Here’s a little story. I once did a quick job for a non-technical nonprofit that was in a bind. Their initial email went something like: “Hi, we hired a contractor to build our Plone site on a shoestring, and now he’s gone, and our production server crashed, and all we have is a .zexp export of the site, and zipfile of the code but we’re not sure if it’s the same as the production version. Can you help us get Plone started or at least get the documents and images out so we can throw up some kind of temporary static site?”

  I gave it a go for a couple days, but I was thoroughly defeated. I felt so bad I only charged them for a couple hours and felt guilty for even doing that. I never want to put an employer or client in that position again, ever.

• No non-container relations. Expressing something like a many-to-many relation in ZODB means writing the code yourself, or installing something like http://pypi.python.org/pypi/zc.relation … which presumably works fine, but I really can’t get excited about it: I’d rather spend my time learning relational technology that might actually be portable to other systems.
• The ZEO server is still a single point of failure and potential bottleneck; No live replication. There’s no free workaround. There is an expensive solution from Zope Corp. RelStorage could theoretically solve this problem for free by deferring to the underlying RDBMS replication, but it’s apparently not been tested.

  It’s important to note that I have never actually run a site where the ZEO server was the bottleneck, but the sites I’ve worked on have relatively small user bases, and the largest-scale ZEO cluster I’ve heard of was a news site: very read-heavy with a relatively small user base doing relatively few writes. I’ve never heard of anybody doing a large site with lots of writes from lots of users. If your goal is to build the next Facebook or Wikipedia, I don’t think there are any relevantly large real-world ZODB case studies you can emulate.

  Most of us just make do without replication or failover of any kind, and hope we will never really need it.
  

• No fine-grained control over which (and how much) data you retrieve.
  
  In any SQL database, you can trivially do “select foo from bar” and get only the values in the foo column, regardless of what other gunk is in each row. In ZODB, you get a whole object - think of this as the equivalent of every query starting with “select *”, so you always get the entire row(s). Results tend to be fat and you have no ad-hoc control over that, short of reorganizing the database. Which leads me to…
  
• Migrations are inconvenient and expensive. Migrations with ZODB typically take the form of a script containing two functions: one which updates a single instance of a particular persistent class, and another function which finds all the instances to upgrade. The latter is the non-trivial part, because there’s actually no way to find all objects of a given type short of walking the entire object tree. If you’re using Zope 2, you may have a ZCatalog handy that you can use, if it knows about all the objects you want to upgrade; or you can use the old ZopeFind API which is just a convenient (and no less expensive) way to walk the entire tree.

  And you can’t really do a migration atomically on a live site, because you’re sure to get ConflictErrors if you try to do it in a single transaction. You can solve this by taking the site down for the duration of the migration. If that’s not an option, you have to try committing and starting a new transaction after every N objects touched, which practically speaking means you’re not going to want to undo your migration. AFAIK there is no existing infrastructure for the latter approach, which means you have to rewrite it in every migration script you ever write.

• Often, you can’t actually use undo. If a transaction touches a frequently-updated object (like oh, say, the catalog indexes), you probably won’t be able to undo that transaction for very long, because other transactions will have since touched the same object, so undoing it would cause a conflict. A transaction is not a database-wide savepoint, like a revision in Subversion; rather, a transaction only knows about the objects that were changed during its lifetime. There’s no way to revert to an arbitrary point in the past.
• Indexing is not transparent. I very often see code in Zope applications to ensure that some index is properly updated after some value changes. It gets tiresome. By contrast, indexes in an RDB typically require no attention from the developer… but they don’t serve the same purpose.
  

What’s Debatable

This section could grow endlessly, but I’ll just list a couple items off the top of my head:

Speed

For years, the accepted wisdom was that ZODB was pretty fast for reads, and slow for writes. Some people claim that it’s actually fast for writes too. I don’t care much about raw benchmarks except insofar as they translate to real applications. The ZODB application I actually get to use the most - Opencore, built on Plone 3 - feels quite slow at the storage layer (some of this is catalog stress, some of it is due to storing binary files in CMFEditions, which we now know was a terrible mistake.)

I have no hard numbers to offer, hence putting this in the “Debatable” category.

Partitioning

Object traversal in Zope encourages you to map your ZODB tree directly to your URL space. I actually quite like this as it’s really easy to understand. But it makes it harder to reorganize your data for scalability reasons (eg. horizontal partitioning aka sharding) without also reorganizing the URL space and breaking links.

This is another one of those problems that’s mostly theoretical to me so far - I haven’t actually needed to do sharding on a ZODB app yet. And most people never will, but if you’re building something very ambitious, it’s something to be aware of.

As noted above, the ZODB can do one kind of partitioning by “mounting” databases in the object graph, like filesystems mounted in one Unix file tree; this is great and easy, but it’s only transparent if the mount point can replace an existing folder; it doesn’t help you with flat but dense data. Also, mounting multiple storages can be problematic when objects under one mount point refer to objects outside that mount point; see eg. the notes at the bottom of http://apidoc.zope.org/++apidoc++/Book/zodb/crossref/show.html

This is another case where I wonder if creative use of RelStorage might help, although I’ve no idea how you’d know where to split the partitions.

Optimizing is Weird

When addressing bottlenecks in an app written against a given RDBMS, there are typically pretty decent docs available that help even a novice get started with tuning their queries and setting up the proper indexes. If I google “mysql query optimization”, I find a lot of useful results on the first page. With ZODB, there are some general strategies that typically are learned the hard way. Good luck googling for docs or tips. One of the few things I found was from a presentation (PDF) that Chris McDonough made about ZODB: “The most important optimization you can perform is to write efficient code. Unfortunately, this is also the hardest way to optimize, because you need to manage all the details.”

Finally: You Can’t Take It With You

You may have noticed a theme running through some of the above.

I’m tired of feeling like I’m in a programming ghetto, and frankly ZODB feels rather marginalized. Not because of any of the things that I think are wrong with it, but just because almost nobody uses it. This has a lot of implications - lost opportunities for re-using innovative work and so forth. This is the danger that Mark Ramm recently warned the Django world about (video link, sorry; there’s no transcript anywhere AFAICT).

But more personally, I just don’t feel that I’m young enough to waste much more of my career on dead-end tech. Python is a plenty big pond for me to swim in; but most of the fish in that pond wouldn’t touch ZODB with a ten-foot pole. That’s a shame, maybe, but perception is reality, and if this was going to change, it would have changed by now. It’s been over 10 years now.

Given that, the time that I spend using ZODB could be better spent learning skills that I can more realistically apply on more future projects. Maybe even (gasp) non-Python projects.

It may seem selfish to harp on how this affects an individual’s career, but sometimes the events in your life push you in that direction. Having ZODB and Zope on my resumé gives me a certain amount of hireability right now, I think because I’m a relatively experienced fish in this tiny (and, AFAICT, not growing) pond where demand (for now) seems to slightly outstrip supply. Will ZODB be even that relevant in, say, 10 years?

Let’s ask:

Okay, I’m putting my flame suit on now

(For a more generally enthusiastic take on ZODB, you should have a look at Chris McDonough’s blog post on ZODB compared to CouchDB. )

FeedBacker: An Introduction

For the last few weeks I’ve been working on FeedBacker, a small REST application that stores and serves up feed items.   FeedBacker also provides a rudimentary query interface to the stored feed items, serving up the result set as a feed.  The idea is that feed items will be stuffed into the FeedBacker database whenever activity happens on an Opencore site, allowing us to easily generate feeds of arbitrary activity history simply by constructing the appropriate FeedBacker queries.  Seeing David Turner’s update and debrief on his Henge efforts yesterday reminded me that I wanted to write a similar post about FeedBacker, so here I am.

Conceptually, FeedBacker is very simple.  It defines a simple FeedItem data model, based on the Atom syndication format, and lets you use POST requests to create them.  GET requests can return either a collection of stored FeedItems or a single FeedItem, in either HTML (using the hAtom microformat) or XML (using Atom)  RSS and/or other formats later may be added later.  There are, however, a few additional features that are worth a closer look.

Metadata

Our use cases require us to support fairly arbitrary data on our FeedItems.  These data points are needed for later retrieval and also for querying purposes.  Since our ultimate goal is to generate feeds of site activity, we need to be able to store such values as the type of action that happened (e.g. ‘created’, ‘edited’, ‘joined’), the type of object on which the action was performed (e.g. ‘wiki page’, ‘discussion’, ‘blog post’), and the project where the action took place, if any.  So FeedBacker allows you to define extra fields on your FeedItems, either through HTTP or by setting an ‘extra_fields’ value in the PasteDeploy ini file.  Each extra field is typed, either as int, string, unicode, datetime, or a binary BLOB.  Any non-BLOB extra field can be used in the queries.  Also, any non-BLOB extra field can be specified as ‘in_results’, i.e. the contents of that field will be included in the FeedItem’s rendering when you make queries or visit the FeedItem resource directly.  When results are returned in Atom format, the extra fields are included in the results as custom tags in the ‘feedbacker’ namespace, e.g:

 <feedbacker:project>StreetsBlogNet</feedbacker:project>

Filters

The querying facility will get us most of the way towards the feeds that we want, but there are some types of feed parameters that can’t be easily represented as a query against the result set.  For this reason ‘feed filters’ were introduced.  Feed filters are just callable functions that expect to receive the results of a feed query and will return a filtered result set on the other side.  Feed filters will help us handle cases like removing near-duplicates (e.g. multiple small page edits in rapid succession) and enforcing security policies so as not to expose a closed project’s activity to anyone not on the project.  Feed filters can be registered programmatically, or by use of a ‘feed_filters’ setting in the PasteDeploy ini file.  There is no support for using HTTP to register feed filters at this time.

Technical Debrief

FeedBacker was written in Python using the repoze.bfg web framework, with SQLAlchemy for persistence.  Since I’ve been working in Zope-land for so long, this was the first project in quite some time I’ve done using a relational database as a back end.  The expressiveness of SQL for a query language is very nice, and SQLAlchemy provides a surprisingly good abstraction layer, but you have to jump through so many hoops to model more complicated data structures in a queryable manner.  It’s quite nice to be able to just stick a dictionary (or a rich object, even) in a pickle and not have to worry about mapping all of the keys and values to relational tables.

Object-Relational Mapping

A significant amount of FeedBacker’s complexity right now is in the object-relational mapping.  To it’s credit, SQLAlchemy was able to do exactly what I wanted it to do, so the complexity of the mapping is absolutely hidden to the front end code.  All of the data points, even the arbitrarily defined extra fields, show up transparently as attributes on the FeedItem objects.  But it took me a few days to get it all dialed in correctly, and then a few later tweaks to make sure everything works correctly even when there are multiple values for a single field.  It remains to be seen how well this will scale with very large data sets or high traffic volumes.

HTML to SQL Query Translation

The other place where there is a bit of complexity is in the translation of the HTTP GET requests to the SQL queries that actually do the work.  FeedBacker provides a hand-rolled syntax to allow you express fairly arbitrary queries against the storage (nice HTML docs coming soon, for now see the docstring for details) which needs to be parsed and converted to a SQL query that can actually be run.  Again, I was very impressed w/ SQLAlchemy’s help here; the ability to construct query clauses piecemeal, passing them around and nesting them further within other clauses, made it relatively easy to pick apart the HTTP query arguments and iteratively construct a query in the process.  Again, we’ll see what the performance ends up looking like under high load, after the right indexes have been applied.

Current Status

For this week I’ve been working on tying everything together by actually implementing a FeedBacker-backed page in the Livable Streets stack.  Things have been coming quite well, although I of course had to revisit a few things when the truth of the requirements butted up against some of FeedBacker’s original assumptions.  The page I’ve been working on is the member account page, based on what is proposed in a mock-up Rollie put together a couple of months back.  Not everything you can see in the mock-up will be making it into the first deployment (there’s not yet support for being a ‘fan’ of a project, for instance), but it gives you a good idea what type of functionality we’re trying to support.  Ultimately, the hope is that the account page will become similar to a user’s home page on FaceBook, where all of the activity that the user cares about will roll by in one place.

The page works by making an HTTP query request of FeedBacker, where the query represents what should show up on the page.  It asks for the results in atom format and then uses Python’s universal feedparser to parse the results and construct the output.  The page is nearly usable; at this point I’m just working on getting all the links and formatting correct, getting the icons to show up, etc.  I’m also working on making sure that as much info as we need is embedded in the feed item data itself.  If we have to visit a large number of Zope objects in order to fill in the details of the page, then we’ve lost, the page will be very slow.  I haven’t hit anything that I’d call a roadblock, however, so I’m feeling pretty confident that this is going to be a good starting point.

My goal is to get the implementation of the page finished up today, and to deploy it to a dev server so we can start comprehensive testing tomorrow.  And, if I wrap this blog post up and get back to it, I just might.

I started working on a simple calendar for opencore called Henge. The idea is that projects could post a calendar of events. My initial version doesn’t even let you describe events — each event just has a link field that you can use.

I’ve decided to put the project on hiatus, because we’re considering rewriting opencore. I think we’re finally starting to figure out how to write applications in the dvhoster, openplans_hooks, transcluder, cookieauth, etc framework. But being able to do things doesn’t mean being able to do them well or efficiently. While debugging henge, I did run into various issues caused by the stack: deliverance hides error messages; I couldn’t figure out what file I had to change to map URLs to project/x/calendar; debugging the project member http call was hassle. And I didn’t even get to the opencore featurelet.

Also, I was starting to hit the limits of SQLObject pretty hard. I wasn’t doing anything especially complicated but I was trying to write an undo mode that made sense. The data model is:

Projects have many Events.

Events have many EventDates.

When an event is deleted, its associated dates should be deleted — this is trivial. But when an event is restored, its EventDates should be restored — and only those EventDates that it had when it was deleted should be restored (not any that might have been deleted earlier). I tried a bunch of strategies to make this work well, and ended up having to do both metaprogramming and hand-coding to make it work. I think a better programmer could have done it all through metaprogramming, but it would have taken me more time than it was worth. In particular, I wanted objects to have uuids as primary keys, and that was impossible. As it turned out, that wouldn’t have gotten me what I needed, but I still should have been able to do it.

What’s actually going on is that an Event is a composite object composed of some general data and a bunch of dates. But SQLObject doesn’t natively have a way to talk about a row and its associated rows from other tables. You can walk through the metadata and find one-to-many relationships, but that’s not the same thing as ownership. If there were a way to talk about this naturally, it would be easier to support things like undo and versioning of composite objects.

I think the lesson is that SQLObject is not really powerful enough for applications which want to support complex interactions with data. This isn’t a complaint about SQLObject — it’s not meant for these sorts of complex cases. I’m not sure if SQLAlchemy is the solution, but I’m certainly interested to find out.

Anyway, there’s a few weeks of work left before I could put Henge live, and I find myself without the time to do them.

• the aforementioned featurelet
• events should have descriptions
• there should be an agenda view
• a javascript date picker
• event permalinks which survive event deletion

Of course, there are a ton of ways to complexify things, many of which are obvious — presently, there is no interface to add multiple dates to an event. But really, the above is all that’s needed to launch.