OpenCore Software

Should I upgrade?

This release includes several small usability enhancements and bug fixes, with a focus on fixing internationalization and character encoding issues. If you’re running an Opencore site with significant non-English content, you will want to upgrade.

If you are still running Opencore 0.12 or earlier on top of Plone 2.5, this is a fairly big upgrade; see the 0.13 announcement for more info.

What’s New Since 0.14.1

Data Migrations: You will need to use the portal_setup tool in the ZMI to run all the available upgrades for version 0.15.

Features:

* Partial translations of the UI into Spanish, French & Portuguese (dimo)

* Member objects now have a _change_member_id() method. This is not exposed by UI yet, but admins can use it in eg. zopectl debug.

* Member profile now has a “contact” button which links to a mail form. The member’s email address is not exposed. (slinkp, rafrombrc)

* We now use the member’s full name (by way of Title()) in most places that conceptually refer to a person, and the member’s ID only in places that refer to the account. Fixes openplans:#2740 (slinkp)

* Mailing lists now have a /membership view that non-Managers can see. Includes a link to each member’s contact form. Email addresses are not exposed. Sorted case-insensitively by member Title or email. (closes openplans:#2752; partially closes openplans:#2751) (slinkp)

* Mailing lists’ manage_membership page now includes links to members’ contact form, and is sorted case-insensitively by member Title or email address. (Partially closes openplans:#2751) (slinkp)

* When inviting a site member to join a project, it is now possible to customize the email message sent (closes openplans:#2741) (egj)

* Marked up a few more strings for translation (slinkp)

* There is now a rebuild_i18n script for conveniently merging new message ids into all the .po files, and generating the test az “translation”. See for more info: http://www.openplans.org/projects/opencore/i18n-usage-in-opencore (slinkp)

* opencore.utils now includes a simple in-memory timestamp_memoize() for caching expensive calls (rafrombrc)

* Added a manual.pot file to manually manage strings from zcml files which can’t be parsed automatically by i18ndude (dimo)

* Added viewlet for analytics JS snippets (rafrombrc)

Bug fixes:

* Fix a longstanding point of user confusion, openplans:#1474. The member search in team management view no longer ignores existing team members but instead displays them with a contextual note that they are already in the project. (egj)

* Account creation forms no longer explode when the id contains ‘(’ or ‘)’. (egj)

* “Delete” button on project contents page now is shown or hidden based on correct permission check. There is a GS upgradeStep to fix existing workflows. (rafrombrc)

* The ‘addable types’ setting for the OpenTeam types was incorrect; it was set to all implicitly allowable types instead of explicitly allowing OpenMembership (and nothing else). Edited the GS profile to reflect the new setting and added an upgradeStep to automate its application. (rafrombrc)

* The create-az.py script now reliably finds all message ids, is less likely to produce invalid output, and doesn’t care what directory you’re in. (slinkp)

* Many flunc tests are slightly less fragile (slinkp)

* Unit tests no longer fail when VerboseSecurity is enabled (slinkp)

* Fix unicode errors in project preferences view (dimo, novalis)

* Fix unicode errors in request-membership view (dimo)

* Wiki attachments now have unique ids based on full path (closes http://trac.openplans.org/openplans/ticket/2728) (novalis)

* Fixed AttributeError when anonymous user requests project membership (http://trac.openplans.org/openplans/ticket/2735) (dimo)

* Slightly more useful error message on project creation when URLdoes not contain enough ASCII characters (http://trac.openplans.org/openplans/ticket/2733) (egj)

* opencore.utility.email_sender now does quoted-printable encoding of headers when it gets anything that can’t be encoded as ascii. (slinkp)

* opencore.utility.email_sender is now a bit smarter about addresses. (slinkp)

How to Install

As usual, I recommend building using our usual build process. Whether you use the newbuild.sh wrapper script from the fassembler-boot package, or just use fassembler directly, you have a choice of two requirements sets:

• opencore-minimal/tags/0.15.0 is a requirements profile for a fairly stripped down opencore build. It includes supervisord and zeo, but does not include tasktracker, wordpress, or deliverance.
  
• openplans/branches/opencore-0.15 builds a more complete stack based on the 0.15 branch, including tasktracker and wordpress blogs. We are in process of vetting this for deployment to www.openplans.org (this site). Once that’s done, we’ll put a tag of these requirements in openplans/releases.

Support

As always, if you have any problems, contact us on the opencore-users mailing list or look us up on the #openplans IRC channel on freenode.

What’s next?

We’re planning to do another release in January; the plan is to get Opencore 0.16 out in time for our friends at http://openfsm.net/ to have it in place before their conference in Belem at the end of the month. A lot of the remaining high-priority issues are still to do with internationalization and translation. Feel like helping out? Get in touch on the opencore-dev mailing list

Should I upgrade?

This is a bugfix-only release. If you are running opencore 0.13 or 0.14, this should be a safe upgrade with a few useful fixes.

If you are still running opencore 0.12 or earlier on top of Plone 2.5, this is a fairly big upgrade; see the 0.13 announcement for more info.

What’s New Since 0.14.0

Data Migrations: You will need to run this migration only if you’re upgrading from 0.13 or earlier. You can run the migration via the portal_setup tool in the ZMI; go to the upgrades tab and run the upgrade registered for version 0.14.0.

Bug fixes:

  * List manager address was showing up as escaped HTML.  (slinkp) (http://trac.openplans.org/openplans/ticket/2706)

 * Wiki version view doesn’t error when no version_id is specified  (slinkp) (http://trac.openplans.org/errors-openplans/ticket/30)

 * Error page doesn’t barf on 404s if the catalog has a problem coming up with suggestions based on the current URL (slinkp)  (http://trac.openplans.org/errors-openplans/ticket/23)

 * Fixed rare error in recataloging projects, raised by the project’s _default_img_data method. (slinkp)

There are also a bunch of upstream bugfixes in Listen, our mailing list component. You will want to upgrade if you’ve had any trouble with non-ASCII characters in mail messages.

How to Install

As usual, I recommend building using our usual build process.  Whether you use the newbuild.sh wrapper script from the fassembler-boot package, or just use fassembler directly, you have a choice of two requirements sets:

• opencore-minimal/tags/0.14.1 is a requirements profile for a fairly stripped down opencore build. It includes supervisord and zeo, but does not include tasktracker, wordpress, or deliverance.
  
• openplans/releases/opencore-0.14.1-rc builds a more complete stack including tasktracker and wordpress blogs; this is equivalent to what we’d deploy on www.openplans.org.   We haven’t fully vetted this build for deployment yet, but it passes all functional and unit tests.
  

Support

As always, if you have any problems, contact us on the opencore-users mailing list or look us up on the #openplans IRC channel on freenode.

What took so long?

I realize it’s been over a month since Opencore 0.14.0.  One thing that added to the time was that, this time I was very careful to create a frozen list of all python packages that opencore depends on, directly or indirectly, so that building this release should be reliable and repeatable. No more depending on whatever happens to currently be in SVN.  And since I’d never done this for 0.13 or 0.14.0, it took longer and I ran into more problems than I expect to for future bugfix releases.

Pip was very useful for this job, although I had to massage the output a little bit to make fassembler happy.

I also took the time to edit my notes and document exactly how I make a release.  I’m using this as a checklist for now.  In the future I’d like to automate some of the more mechanical parts of the process.

Get it from https://svn.openplans.org/svn/opencore/tags/0.12.0

I am not uploading a package to Pypi because we still have some work to do before opencore works as an egg. I expect to be working on this over the next few releases. For now, the most reliable way to install is still setup.py develop (or use Fassembler which does it for you).

This is as good a time as any to announce that I hope to release about once a week for the rest of the summer! I will miss a few due to vacations. But hopefully 0.13 will be released by July 21.

What’s new in 0.12.0

This is a summary of user-facing changes. For a complete list, see CHANGES.txt in the source code.

* New form for deleting accounts. This is not linked to from anywhere yet, but if you need to delete a spammer, or if a user wants to leave the site, point your browser at people//delete and fill out the form.

* Zip file export view of project wiki pages. This is not linked to from anywhere yet; you have to visit projects//project-wiki-export.zip. Page attachments are not included.

* Email obfuscation for discussion list views

* fix team AttributeError reported by user on summary page

* fix site_url AttributeError reported by user on password reset

* fix user reported UnboundLocalError on pending page

* possibly resolve user reported error on attachment delete

A decree from the release manager:

Until further notice, we will use the common three-component major.minor.patch version numbering scheme, as popularized by Apache and many other projects.

• Major releases (e.g. 1.0.0, 2.0.0) indicate major featureset milestones or large backward incompatibilities.

  Our major release number will remain 0 until further notice (i.e. until we figure out WTF a 1.0 release would mean).

• Minor releases (e.g. 0.12.0, 0.13.0) indicate new features.

  Minor releases may also require data migrations and will generally also include all the bugfixes since the previous minor release.

• Patch (aka “point” or “micro”) releases (e.g. 0.12.1, 0.12.2) may only contain bugfixes, and must be 100% backward compatible.

  No new features are allowed in a patch release. No data migrations may be required by a patch release except as strictly required for a critical bugfix, in which case I might decide to bump the minor version anyway.

You might notice with this policy that minor numbers will get bumped quite often. That’s okay. There’s nothing wrong with numbering a release eg. 0.87.0, except that it would strongly suggest a need to re-evaluate the 1.0 milestone

This is basically the same numbering scheme that we’ve had since we broke out of the infinite 0.9.7.x loop. The main difference is organizational - one person (me) will be paying attention to keeping it consistent. That was the only real problem with Ethan’s previous plan.

I’m going to be playing Opencore Release Manager, starting now. Let me explain what that means, and relay some plans.

Scope

I’m just talking about opencore, the Plone extension at the heart of the openplans software stack.

I’m not talking about any other part of the openplans software stack (tasktracker, wordpress, deliverance, …)

I’m definitely not talking about openplans.org the website.

Opencore needs a release manager?

Opencore was originally created by TOPP for www.openplans.org. So opencore releases have historically been driven by the needs of www.openplans.org’s deployment schedule. Since our deployment process doesn’t require a formal opencore release, releases tend to be a rare afterthought.

But there are at least two things that more formal releases would help with:

1. Lower the barrier for other community members to get started using opencore.
2. Let the world know about what we’re doing.
   

My overarching goal is to promote a lively community of developers and users for opencore, and provide some balance to TOPP’s path of least resistance–to only worry about our own deployments.

Why me?

Ethan asked me to do it and I was happy to volunteer. (If you don’t know who Ethan is either, this picture should suffice: http://www.openplans.org/people/ejucovy )

Who?

I’m Paul Winkler, a Zope user and developer since 1999, a TOPP employee since October 2007, and Opencore’s release manager since right now. http://www.openplans.org/people/slinkp

What will determine the release schedule?

I want to decouple Opencore’s release cycle from the TOPP deployment cycle. Douglas Mayle has volunteered to act as the openplans.org deployment manager; he and I have tentatively agreed on this plan:

• Opencore releases will be on a time-based schedule, TBD.
• Features that aren’t ready in time for a release can just wait until the next release.

  Rationale: Feature-driven releases inevitably get delayed and delayed, and the features don’t get done any faster.

• Openplans.org deployments will be on an independent time-based schedule. (I believe the plan is to deploy weekly.)
• “Must-have” openplans.org features and fixes can be deployed independently of the opencore release cycle by putting them in a new package that layers on top of opencore.
• Openplans-specific code and text (like the current copyright notice) will be gradually moved out into this new package.
• The aforementioned openplans.org customization package does not exist yet; we need to create one ASAP.
  

Of course TOPP’s own needs will continue to drive a lot of opencore development, which is a good thing; but that shouldn’t be a burden on you, the wider opencore user/developer community.

Does the Release Manager have to be a TOPP employee?

No. I’m just the first. It should be a rotating job. It’s my hope that before long, people outside TOPP will be willing and able to play this role.

When will the job rotate?

Don’t know yet.

What does the Opencore Release Manager do?

• Manage the release schedule (not TOPP’s deployment schedule!). This does not mean I’m the dictator, just the point man and coordinator.
• Shepherd release candidates through a stabilization process.
• Create and announce final release packages.
• Document upgrade process.
• Document any backward incompatibilities.
• Draft and enforce relevant dev process standards (e.g. which branches or plugins should you commit bugfixes or features to).
• Document this process for the next release manager.

At this stage, I’ll also be acting as a sort of ambassador between TOPP and the wider community. Which is a bit silly, as every opencore contributor does that to whatever extent they wish. But I think for now it’s useful to consciously formalize it as part of my job. I will make an effort to:

• Solicit community input about decisions we need to make for opencore.
• Remind TOPP employees to discuss and make decisions about opencore out in the open. If you have a real-life relevant conversation I’ll try to remind you to post a summary to the mailing list.
• Keep the community updated on changes to our build process. If we break your builds, you can complain to the opencore-user or opencore-dev mailing list, and you should expect a response from me.
• Lobbyist for the community: e.g. prioritizing and scheduling feature requests and bugfixes.

There will of course be other forces at TOPP driving things with other priorities, but *my* role will be to ensure that the community’s needs don’t get ignored. This is more about release management than development management: I’m not in charge of deciding which features get done when. But whenever good code comes from the community, we have a responsibility to get that code released in a timely manner, and generally be as responsive as we can.

What about the rest of the stack?

We realize that most of the non-TOPP users of opencore are using some or all of the entire www.openplans.org stack. We’d eventually like to be making formal releases of all the packages, and possibly some kind of “batteries included” big meta-package; but that’s in the future. For now, opencore is our fastest-moving target and the most in need of release management.

Release Numbers

We supposedly have a policy now (Ethan proposed it here: http://tinyurl.com/4ybpro) … but we haven’t been sticking with it. I’m going to draft a proposal for a simpler convention and process. More on this in a later message.

Releases, Eggs, and Builds

I’d like to be releasing versioned eggs of opencore to PyPi. These should be installable into a suitable Zope/Plone instance using nothing more than easy-install.

(We have made a few releases to PyPi, but they aren’t actually usable … some files are missing.)

For bootstrapping a full stack including Zope, Plone, and other non-egg stuff, TOPP will continue to use and develop our Fassembler build tools. But it should become easier to do without that, if you want.

Branch Policy

We are currently trying to follow a “Stable Trunk” practice. Experimental and risky code should not be on the trunk.

Developer Infrastructure

This is only tangentially related to release management. I just wanted to note in passing that we’re planning a more thought-out integrated home on the web for opencore development. We’re calling this the “TOPP Dev Center” for now. Other open-source packages from TOPP will live there too.

In the meantime:

Wiki and mailing lists are at: http://www.openplans.org/projects/opencore

Trac (bug tracking) is at: http://trac.openplans.org/openplans

If you need a Trac account, just ask. (I wish we could take anonymous bug reports, but we got trac spam. There is a guest account, but I’m not sure of the login details.)

If you want SVN commit access, first you should send some bugfixes and features as patches (either in trac or on the dev list). We’ll happily give commit access to people who have a history of submitting good patches. There will be a contributor agreement you’ll have to sign, somewhat similar to Zope’s.

Current Road Map

Some things coming up in the short term:

• Plone 3

  We will soon be releasing a version of opencore that runs on Plone 3. Rob Miller has done all the hard work and we just need to QA this branch and merge it back to the trunk.

  Currently, the plan is:
  

  1. TOPP will create a release candidate from the plone3 branch.
     
  2. TOPP will QA this branch.
  3. TOPP will deploy this branch to www.openplans.org.
     
     When that goes well, I will:
  4. Branch off the current opencore trunk to an opencore-plone2.5 branch
  5. Make a final stable release from the opencore-plone2.5 branch (to be followed by bugfix releases as necessary)
  6. Merge the plone 3 branch back to the trunk.

  This will definitely happen this summer; hopefully even in July. More concrete dates will be forthcoming.

  If people in the community expect to continue using the Plone 2.5 version for a while, I would like to hear about it so I can serve you better.

• Easier “plugin” management using z3c.autoinclude. This will replace our usage of zcmlloader, doing the same job but better (and other people are actively developing it).
  

Some ongoing maintenance tasks I’d love community help with:

• Opencore is slow. Let’s make things faster. (But let’s get plone 3 merged to trunk first, it already helps substantially.)
• The test suite has a lot of problems. More on this in a later message.
• Dependency cleanup. I suspect there is cruft in the Products “bundle” that we don’t actually use.
• What do you guys want?

Going forward, this road map should have a permanent home prominently linked in the aformentioned dev center. There is already such a page at http://www.openplans.org/projects/opencore/planning but it badly needs an update. I’ll be looking to revive that.

Questions?

Whew. Sorry for the length. Feedback would be most welcome.

- PW

Rob Miller has been thinking about zc.buildout lately, and providing a buildout-based bootstrap for our stack (even if it only calls out immediately to fassembler).  This got me to thinking, and I finished a reflection post about fassembler.

At the end I almost talked myself into refactoring stuff to make the good parts of fassembler usable in zc.buildout, so that we could move over to that in the future if we want to.  Then… right at the end… pow!  Files!  All files, only files!

What if our build process just built files?  Relocatable files.  Not-system-dependent files.  You could tar the whole thing up, copy it to another location or server, unpack it, and get the exact same site.  That tarball would basically be equivalent to the build itself.

It’s not that App Engine seems particularly awesome.  More that it seems like it might be Just The Right Kind Of Dumb.  It’s that this makes it so easy to think about the build, about what it does, about stuff like the workflow for testing and deployment and rolling back.  Easy to explain to other people, easy to audit, easy to debug.

The problem of course is that it requires pushback into other products.  Lots of other products.  But we can do that.

Thoughts?

I’ve stated before one thing I like about openplans is the “simple” permission simple.  I use the quotes because there are several caveats, one of which I’ll address here.  For a particular project, the permission system is essentially a 3×4 table:

  This is pretty simple and intuitive I think.  We also have a status of being listed on a project.  That is, can people see you’re part of the team.  If you’re a whistleblower at exxon, you might not want your company to see that you’re part of the ExxonIsSatan project, for instance.  For a particular project, other members can see information about your affiliation according to the following rubrik:

 I’m being intentionally over-verbose here for illustration.  If user A wants to see information concerning member B about his affiliation with project C, how does this work?

- can user A view project C?

- can user A view member B’s affiliation with project C?

So its a cascaded permission system, if you will (or something like that, anyway).  My main reason for posting this is to put this in programmatic terms and to see what others thought about this. We’re not really taking advantage of as much as we can here, and I’m tempted to say that intelligent use of roles could make this all happen behind the curtain, which would be my preference.  Also, why not have a reference on line?

 [Disclaimer:  no exxon executives, stakeholders, or stock prices were harmed either financially or materially in the writing of this post]

Looking at our code (that’s how it always starts) I was again bitten by that disgust for DRY code. Some of this is inevitable in a web framework, I think. Web frameworks are by their nature complex programs. Should a web framework handle authentication and permissions? Almost definitely. Should a web framework handle unicode and i18n and localization issues? One would hope so.

Python has a bunch of these frameworks and I think this is a good thing. What I do question is how much functionality lives in the framework that could be abstracted outside of the framework.  What is a framework but a tool kit that you want to apply to HTTP requests and responses?  Thinking that way, issues like handling unicode, HTML escaping, authentication, etc are really library type functions.  If Bob doesn’t like pylons but likes how they do auth, that part should be pretty libraried-out (well, there’s authkit, which is actually a good example of what I consider *bad* library code).

This isn’t a magic bullet, not do I encourage programmers to prematurely make their code library-like.  I’ve been bitten by that too.  But once one figures out process and what one *really* wants to do, its easy to figure out the pattern, figure out which parts are actually *part* of the web framework and which parts are better *consumed* by the web framework.

Robert and I spent much of the day optimizing the mailing list feed.  Formerly, it was sorted by date of latest reply, with the number of replies displayed and a link to the original post and the latest reply.  The idea was/is that the summary page should just give a brief overview and therefore show all the threads separately (if two of the most recent replies are to the same original message, this is shown only once in the feed).  Unfortunately, the way listen currently is, doing this meant getting all of the top-level threads and traversing them (only brains, but still).  Correction of a simple oversight on my part brought the time for the opencore project down from about 110s to 30s (there are tens of thousands of messages on these lists), but this is still too slow and will cause a proxy error.

 So we took out the traversal to find the thread beginning/end and just display the 5 most recent messages regardless of thread.  This brought things down to 5s, but now its different from the design spec of how its displayed.

 Which got me to thinking…

What I was given was a definitive but tentative plan for how the feeds would work.  The blog feed displays the top 5 messages regardless of the most recent comments (the number of which are displayed).  The wiki feed displays the five most recently changed/created pages (there are no “responses” to a wiki page, at least yet).  The listen feed is speced to display the activity of the five most recent threads across all mailing lists (making it a feed aggregator as well as a feed provider).  The team feed displays the members sorted by whether they have a portrait or not, though debate has occurred whether additionally the admins should be displayed on top, or sorting users randomly, or…

It would be nice to have the adaptation controllable with parameters.  How the feed is sorted could be such a parameter, or maybe several parameters if sorting by multiple fields (”i want all admins on top, then members with pictures, with some randomness thrown in”).

I hope we can soon figure out our feed story going forward.  Its come a long way, but still lots left to do.

A more accurate picture of the oc-feed model: