OpenCore Software

I’m going to be playing Opencore Release Manager, starting now. Let me explain what that means, and relay some plans.

Scope

I’m just talking about opencore, the Plone extension at the heart of the openplans software stack.

I’m not talking about any other part of the openplans software stack (tasktracker, wordpress, deliverance, …)

I’m definitely not talking about openplans.org the website.

Opencore needs a release manager?

Opencore was originally created by TOPP for www.openplans.org. So opencore releases have historically been driven by the needs of www.openplans.org’s deployment schedule. Since our deployment process doesn’t require a formal opencore release, releases tend to be a rare afterthought.

But there are at least two things that more formal releases would help with:

1. Lower the barrier for other community members to get started using opencore.
2. Let the world know about what we’re doing.
   

My overarching goal is to promote a lively community of developers and users for opencore, and provide some balance to TOPP’s path of least resistance–to only worry about our own deployments.

Why me?

Ethan asked me to do it and I was happy to volunteer. (If you don’t know who Ethan is either, this picture should suffice: http://www.openplans.org/people/ejucovy )

Who?

I’m Paul Winkler, a Zope user and developer since 1999, a TOPP employee since October 2007, and Opencore’s release manager since right now. http://www.openplans.org/people/slinkp

What will determine the release schedule?

I want to decouple Opencore’s release cycle from the TOPP deployment cycle. Douglas Mayle has volunteered to act as the openplans.org deployment manager; he and I have tentatively agreed on this plan:

• Opencore releases will be on a time-based schedule, TBD.
• Features that aren’t ready in time for a release can just wait until the next release.

  Rationale: Feature-driven releases inevitably get delayed and delayed, and the features don’t get done any faster.

• Openplans.org deployments will be on an independent time-based schedule. (I believe the plan is to deploy weekly.)
• “Must-have” openplans.org features and fixes can be deployed independently of the opencore release cycle by putting them in a new package that layers on top of opencore.
• Openplans-specific code and text (like the current copyright notice) will be gradually moved out into this new package.
• The aforementioned openplans.org customization package does not exist yet; we need to create one ASAP.
  

Of course TOPP’s own needs will continue to drive a lot of opencore development, which is a good thing; but that shouldn’t be a burden on you, the wider opencore user/developer community.

Does the Release Manager have to be a TOPP employee?

No. I’m just the first. It should be a rotating job. It’s my hope that before long, people outside TOPP will be willing and able to play this role.

When will the job rotate?

Don’t know yet.

What does the Opencore Release Manager do?

• Manage the release schedule (not TOPP’s deployment schedule!). This does not mean I’m the dictator, just the point man and coordinator.
• Shepherd release candidates through a stabilization process.
• Create and announce final release packages.
• Document upgrade process.
• Document any backward incompatibilities.
• Draft and enforce relevant dev process standards (e.g. which branches or plugins should you commit bugfixes or features to).
• Document this process for the next release manager.

At this stage, I’ll also be acting as a sort of ambassador between TOPP and the wider community. Which is a bit silly, as every opencore contributor does that to whatever extent they wish. But I think for now it’s useful to consciously formalize it as part of my job. I will make an effort to:

• Solicit community input about decisions we need to make for opencore.
• Remind TOPP employees to discuss and make decisions about opencore out in the open. If you have a real-life relevant conversation I’ll try to remind you to post a summary to the mailing list.
• Keep the community updated on changes to our build process. If we break your builds, you can complain to the opencore-user or opencore-dev mailing list, and you should expect a response from me.
• Lobbyist for the community: e.g. prioritizing and scheduling feature requests and bugfixes.

There will of course be other forces at TOPP driving things with other priorities, but *my* role will be to ensure that the community’s needs don’t get ignored. This is more about release management than development management: I’m not in charge of deciding which features get done when. But whenever good code comes from the community, we have a responsibility to get that code released in a timely manner, and generally be as responsive as we can.

What about the rest of the stack?

We realize that most of the non-TOPP users of opencore are using some or all of the entire www.openplans.org stack. We’d eventually like to be making formal releases of all the packages, and possibly some kind of “batteries included” big meta-package; but that’s in the future. For now, opencore is our fastest-moving target and the most in need of release management.

Release Numbers

We supposedly have a policy now (Ethan proposed it here: http://tinyurl.com/4ybpro) … but we haven’t been sticking with it. I’m going to draft a proposal for a simpler convention and process. More on this in a later message.

Releases, Eggs, and Builds

I’d like to be releasing versioned eggs of opencore to PyPi. These should be installable into a suitable Zope/Plone instance using nothing more than easy-install.

(We have made a few releases to PyPi, but they aren’t actually usable … some files are missing.)

For bootstrapping a full stack including Zope, Plone, and other non-egg stuff, TOPP will continue to use and develop our Fassembler build tools. But it should become easier to do without that, if you want.

Branch Policy

We are currently trying to follow a “Stable Trunk” practice. Experimental and risky code should not be on the trunk.

Developer Infrastructure

This is only tangentially related to release management. I just wanted to note in passing that we’re planning a more thought-out integrated home on the web for opencore development. We’re calling this the “TOPP Dev Center” for now. Other open-source packages from TOPP will live there too.

In the meantime:

Wiki and mailing lists are at: http://www.openplans.org/projects/opencore

Trac (bug tracking) is at: http://trac.openplans.org/openplans

If you need a Trac account, just ask. (I wish we could take anonymous bug reports, but we got trac spam. There is a guest account, but I’m not sure of the login details.)

If you want SVN commit access, first you should send some bugfixes and features as patches (either in trac or on the dev list). We’ll happily give commit access to people who have a history of submitting good patches. There will be a contributor agreement you’ll have to sign, somewhat similar to Zope’s.

Current Road Map

Some things coming up in the short term:

• Plone 3

  We will soon be releasing a version of opencore that runs on Plone 3. Rob Miller has done all the hard work and we just need to QA this branch and merge it back to the trunk.

  Currently, the plan is:
  

  1. TOPP will create a release candidate from the plone3 branch.
     
  2. TOPP will QA this branch.
  3. TOPP will deploy this branch to www.openplans.org.
     
     When that goes well, I will:
  4. Branch off the current opencore trunk to an opencore-plone2.5 branch
  5. Make a final stable release from the opencore-plone2.5 branch (to be followed by bugfix releases as necessary)
  6. Merge the plone 3 branch back to the trunk.

  This will definitely happen this summer; hopefully even in July. More concrete dates will be forthcoming.

  If people in the community expect to continue using the Plone 2.5 version for a while, I would like to hear about it so I can serve you better.

• Easier “plugin” management using z3c.autoinclude. This will replace our usage of zcmlloader, doing the same job but better (and other people are actively developing it).
  

Some ongoing maintenance tasks I’d love community help with:

• Opencore is slow. Let’s make things faster. (But let’s get plone 3 merged to trunk first, it already helps substantially.)
• The test suite has a lot of problems. More on this in a later message.
• Dependency cleanup. I suspect there is cruft in the Products “bundle” that we don’t actually use.
• What do you guys want?

Going forward, this road map should have a permanent home prominently linked in the aformentioned dev center. There is already such a page at http://www.openplans.org/projects/opencore/planning but it badly needs an update. I’ll be looking to revive that.

Questions?

Whew. Sorry for the length. Feedback would be most welcome.

- PW

Rob Miller has been thinking about zc.buildout lately, and providing a buildout-based bootstrap for our stack (even if it only calls out immediately to fassembler).  This got me to thinking, and I finished a reflection post about fassembler.

At the end I almost talked myself into refactoring stuff to make the good parts of fassembler usable in zc.buildout, so that we could move over to that in the future if we want to.  Then… right at the end… pow!  Files!  All files, only files!

What if our build process just built files?  Relocatable files.  Not-system-dependent files.  You could tar the whole thing up, copy it to another location or server, unpack it, and get the exact same site.  That tarball would basically be equivalent to the build itself.

It’s not that App Engine seems particularly awesome.  More that it seems like it might be Just The Right Kind Of Dumb.  It’s that this makes it so easy to think about the build, about what it does, about stuff like the workflow for testing and deployment and rolling back.  Easy to explain to other people, easy to audit, easy to debug.

The problem of course is that it requires pushback into other products.  Lots of other products.  But we can do that.

Thoughts?

This past Tuesday I attended the NYC Python User’s Group meeting at the offices of DayLife.

The presentation this week was by Peter Fein about GrassyKnoll, a text search engine written in Python.

From TOPP’s point of view there are several interesting things about it:

• It could provide us with a sitewide search solution for openplans.org and livablestreets.com.
  
• It can use any of a number of pluggable back ends (notably PyLucene).
  
• You interact with it entirely via a simple REST API.
  
• A GrassyKnoll client can also trivially be a GrassyKnoll server. (Peter gave a live demo of this.) In theory this could allow for fun things like smart clustering, where one server gets your query and dispatches queries to N other servers, and then merges the results appropriately. The end client still sees the same simple rest API.
• It’s written entirely in Python and is Free.
• They’re looking for more people to help out and would love for us to get involved.

The major down side is that they’re still some months away from a production-ready release. (There were a few glitches in the live demo.) When I pressed him about this, Peter said “definitely by the end of the year.” I got the impression that more hands helping would speed that up.

Also, it doesn’t define any kind of common query language; it just passes them along to the back end. So you do have to know what you’re really talking to.

Peter said he was hopefully coming to the next NYCPUG meeting that we’re hosting here at TOPP on July 15.

So, Zope’s handling of XML page templates is kind of broken.

As a result, all of opencore’s page templates are parsed in ZPT’s “html mode”. This makes for some surprising validation behavior (e.g. unclosed tags is not an error, but putting content in a “link” tag is an error… which sucks if you happen to be trying to generate georss and not HTML …)

I ended up not even using ZPT for my georss feed, but for posterity, there are at least two possible workarounds:

1. Put an xml declaration at the top of your template (this makes the parser to use XML mode). If  your template is only for a snippet and not for a whole document, you don’t want the declaration showing up in the middle of the final document. To avoid that, use METAL macros.  It’s a bit tricky to do this from python, so we have a handy render_tal function in opencore.browser.tal.  And you’ll need to manually encode to your output encoding if you want to use this with response.write().  An example of this that I later reverted can be seen here.

2. You could, if you don’t want to use macros and can’t have the xml declaration in your output, instead subclass Five’s ZopeTwoPageTemplateFile like so to force the TAL parser to use the right mode. (Thi sis also in the example linked above.)

 class XmlPageTemplateFile(ZopeTwoPageTemplateFile):

    """This class exists because the process whereby PageTemplateFile
    decides whether to use HTML or XML mode for parsing and validation
    is completely stupid.
    """

    content_type = 'text/xml'

    def _cook_check(self):
        # In PageTemplateFile, this gets called potentially multiple
        # times (if you're in dev mode), sniffs the file's content,
        # and if it doesn't start with an XML declaration, uses HTML
        # mode.  We don't want that.  So here, we duplicate the same
        # code without the sniffing.
        if self._v_last_read and not DevelopmentMode:
            return
        __traceback_info__ = self.filename
        try:
            mtime = os.path.getmtime(self.filename)
        except OSError:
            mtime = 0
        if self._v_program is not None and mtime == self._v_last_read:
            return
        t = self.content_type  # Take my word for it, dammit!
        f = open(self.filename, "rb")
        if t != "text/xml":
            # For HTML, we really want the file read in text mode:
            f.close()
            f = open(self.filename, 'U')
            text = ''
        text = f.read()
        f.close()
        self.pt_edit(text, t)
        self._cook()
        if self._v_errors:
            logger.error('Error in template: %s' % 'n'.join(self._v_errors))
            return
        self._v_last_read = mtime

For my georss feed, I ended up deciding this was all too much bother and I just went back to using Python string interpolation. There just wasn’t enough to the “template” to justify the pain of getting TAL to work properly under these constraints.