­ ­

­”ZopeSkel is a way of sharing our best practices.­” - Godefroid Chapelle­

Lesson learned: don’t blog about the sprint during the sprint. Sprint instead.

So this post comes after the No-fun ZopeSkel BBQ Sprint to inform you of the results. The results were impressive. So this post will be hefty. To get the executive summary, see the list of accomplishments.

There are two branches and a new namespace package for ZopeSkel from the sprint. Most of the work is in a branch produced primarily by Joel Burton and Cris Ew­ing:

http://dev.plone.org/collective/browser/ZopeSkel/branches/bbq-sprint-cewing

­

­http://svn.plone.org/svn/collective/ZopeSkel/branches/bbq-sprint-cewing­

See the ­Try It Out section of this report for how to play with this branch.

­­The bin/zopeskel­ Wrapper

If you easy_install that branch to get ZopeSkel, you will not only notice the usual bin/cheetah and bin/paster executables added to your Python, but a new bin/zopeskel as well. This new executable is the ZopeSkel wrapper product which had previously been code named “button.” And it makes ZopeSkel as easy as pushing a button for Plone themers and integrators. Running bin/zopeskel gives a lot of the same output as running bin/paster create --list-templates, except you don’t have to remember both the paster create command and its --list-templates option: see the output of ­bin/zopeskel­.

The template summaries themselves were also improved.

It was decided to call the wrapper “zopeskel” instead of “button” in order to not complicate what Plone themers and integrators have already learned from reading documentation about what they need to install without adding yet another new product. Plus, the name is simply intuitive: just run the same name you have just installed. The zopeskel wrapper allows us to customize the ZopeSkel experience for Plone themers and integrators without disturbing what Plone developers have been doing with ZopeSkel already. Everything works with bin/paster just as it did before. But now there is also a simplified bin/zopeskel way to use ZopeSkel.

bin/zopeskel --hel­p­

There are a few things to notice about the bin/zopeskel output. You didn’t need to ask for help to get help. The template listing comes after some usage notes, and the template listing is categorized. That is, all ZopeSkel templates now come with a category attribute. Following along with the usage notes, extended help is available with the --help option, and that option reveals substantial usability improvements: see the output of bin/zopeskel –help­­.­ ­

­

After analyzing tens of thousands of attempted ZopeSkel uses at Plone Bootcamps, the most common user error, omitting the -t from paster create -t some_template_name, has been eliminated by simply having bin/zopeskel take the name of a template. This seems like such a small thing, and maybe something at which some developers might scoff. But when given to thousands of new Plone themers and integrators, the ease of use buy-in can be considerable.

bin/zopeskel --­list­

In addition to the already available short template summary,­ a bin/zopeskel --list option now gives displays the rich contents of a new help attribute for each template: see the output of bin/zopeskel --list­.­ ­

The template help attributes are designed to forestall common user errors in template use and selection as illustrated by the help attribute for the plone template:

plone: A project for Plone products

   This creates a Plone project (to create a Plone *site*, you
   probably want to use the one of the templates for a buildout).

   To create a Plone project with a name like 'plone.app.myproject'
   (2 dots, a 'nested namespace'), use the 'plone_app' template.

­

Template Pre-Run Help

Even if the ZopeSkel user misses the --list­ option, the help attributes are also played when the templates are first run and before any template questions are answered. After the template help attribute content, additional “pre-run” help information is displayed which is designed to answer questions about templates commonly heard on the #plone IRC channel, especially when the Plone newcomer is reading Martin Aspeli’s Professional Plone Development: ­

damon:cewing cbc$ bin/zopeskel plone3_buildout

plone3_buildout: A buildout for Plone 3 installation­

This template creates a Plone 3 buildout (for most users, a preferred
way to get an installation of Plone 3 is to use the appropriate installer,
as these are all buildout-based since Plone 3.1)

If at any point, you need additional help for a question, you can enter
'?' and press RETURN.

Enter project name:

Template Variable Help

Note that each template question now has extended help stored as an attribute on each template variable and accessed by typing a question mark as a response to any template question. That extended help is again designed to route around common Plone newbie mistakes:

Enter project name: ?

**************************************************************************
**   *** NOTE: You probably don't want to use this template!

**  Since Plone 3.1, the preferred way to get a buildout-based setup
**  for Plone is to use the standard installer for your operating
**  system (the Windows installer, the Mac installer, or the Unified
**  Installer for Linux/Unix/BSD). These give you a best-practice,
**  widely-used setup with an isolated Python and a well-documented
**  buildout. This template is here for older versions of Plone and
**  for experts who explicitly want a raw, non-installer-based
**  installation.

**  (This message is particularly aimed at people following out-of-
**  date books/documentation that suggest this is the right way to get
**  a Plone-based buildout. This is no longer the case.)
**************************************************************************

Enter project name:

Of course, most of the time, the extended help information for a question is not that extreme and more about the question at hand:

damon:cewing cbc$ bin/zopeskel archetype     

archetype: A Plone project that uses Archetypes content types

This creates a Plone project that uses Archetypes content types.
It has local commands that will allow you to add content types
and to add fields to your new content types.

This template expects a project name with 1 dot in it (a 'basic
namespace', like 'foo.bar').

Enter project name: ?

This template expects a project name with 1 dot in it (a 'basic
namespace', like 'foo.bar').

­Enter project name: 

Expert/Easy Mode

All templates now start with a question concerning “Easy” or “Expert” mode, for which the default is Easy. In Easy mode, sensible defaults are pre-selected for questions which often confuse Plone themers and integrators. This results in fewer questions to answer:

Expert Mode? (What question mode would you like? (easy/expert/all)?) ['easy']: ?

|  In easy mode, you will be asked fewer, more common questions.

|  In expert mode, you will be asked to answer more advanced,
|  technical questions.

|  In all mode, no questions will be skipped--even things like
|  author_email, which would normally be a default set in a
|  $HOME/.zopeskel file.

Expert Mode? (What question mode would you like? (easy/expert/all)?) ['easy']:

Easy mode is currently of most use in the Basic Namespace, Nested Namespace, and Plone 2 Theme templates, but the capability is now present on all questions in all templates through a ­modes­ attribute on template variables.

Naming Nested Packages

You can really see the benefit of easy mode in the Nested Namespace template where the outer and inner namespaces are derived from the project name, eliminating three questions. In expert mode:

This template expects a project name with 2 dots in it (a 'nested
namespace', like 'foo.bar.baz').

Enter project name: plone.app.example

If at any point, you need additional help for a question, you can enter
'?' and press RETURN.

Expert Mode? (What question mode would you like? (easy/expert/all)?) ['easy']: expert
Namespace Package Name (Name of outer namespace package) ['plone']:
Namespace 2 Package Name (Name of inner namespace package) ['app']:
Package Name (Name of the inner namespace package) ['example']:
Version (Version number for project) ['1.0']:

In easy mode:

Enter project name: plone.app.example

If at any point, you need additional help for a question, you can enter
'?' and press RETURN.

Expert Mode? (What question mode would you like? (easy/expert/all)?) ['easy']:
Version (Version number for project) ['1.0']:

The ndot­ A­ttribute

The ndot attribute of templates now specifies how many dots are required in a project name so that project names are validated before splitting. Here is ndot at work on the Nested Namespace template:

This template expects a project name with 2 dots in it (a 'nested
namespace', like 'foo.bar.baz').

Enter project name: plone.app

ERROR: Project name expected 2 dots, supplied 'plone.app' has 1 dots

This template expects a project name with 2 dots in it (a 'nested
namespace', like 'foo.bar.baz').

Enter project name:

Template Post-Run Help

Like the pre-run help, new “post-run” help is displayed at the conclusion of answering all template questions. Confusing and excessive post-run diagnostics were also significantly trimmed:

damon:cewing cbc$ bin/zopeskel plone3_buildout sexy

plone3_buildout: A buildout for Plone 3 installation

This template creates a Plone 3 buildout (for most users, a preferred
[...snip...]
Verbose Security? (Should verbose security be "on" or "off"?) ['off']:

**************************************************************************
**   Generation finished.

**  You probably want to run python bootstrap.py and then edit
**  buildout.cfg before running bin/buildout -v".

**  See README.txt for details.
**************************************************************************

­­damon:cewing cbc$

Template Dependency Graph Tool

A second branch contains a new ZopeSkel real-time dependency and inheritance graphing tool:

­

­http://dev.plone.org/collective/browser/ZopeSkel/branches/bbq-sprint-cbcunc

­­http://svn.plone.org/svn/collective/ZopeSkel/branches/bbq-sprint-cbcunc­­

Running the zopeskel/doctools/graph.py­ tool in the branch against the current ZopeSkel 2.13 codebase produces GraphViz “dot” commands which can be redirected to a file and processed by popular tools such as GraphViz and OmniGraffle or one of many Python libraries:

­ZopeSkel 2.13 Dependency Graph

The light goldenrod colored shapes depict ZopeSkel entry point templates (the template names) while the pale salmon colored shapes depict the template inheritance from classes in PasteScript and Cheetah.

There are many as yet undocumented options for the graphing tool. The default options produce a simplified graph suitable for inclusion in ZopeSkel documentation. ­Documentation for ­zopeskel.doctools.graph as well as test cases for the tool are to be the focus of some further sprinting on ZopeSkel at Plone Conference 2009.

The purpose of zopeskel.doctools.graph is to insure ZopeSkel developers have a means both to inspect changes in the ZopeSkel dependency tree and to automatically produce up-to-date visualizations for any changes made.

Abstract Base Classes

The graphing tool is necessary to illustrate one of the fundamental changes in ZopeSkel from the sprint, the factoring of some abstract base classes from the inheritance tree of template classes so that common Cheetah variables common to several templates could be maintained in one place more easily. Here is the dependency graph of ZopeSkel in the bbq-sprint-cewing­ branch:

ZopeSkel BBQ Sprint Branch Dependency Graph­­

You can see three new classes, AbstractZope, AbstractNestedZope, and AbstractBuildout, which are not ZopeSkel entry points, but which factor out many common characteristics of their child templates. These classes and others like them provide the basis for future best practice maintenance of ZopeSkel.

Command Line Template Answers

­bin/zopeskel in the ­bbq-sprint-cewing­ branch also allows the Plone themer or integrator to provide template answers directly on the command line. Any answers provided on the command line will cause the interactive template question to be skipped. In fact, there is no reason you cannot use this to script the entire run of a template:

damon:cewing cbc$ bin/zopeskel plone3_buildout sexy expert_mode=expert plone_version=3.3.1 zope2_install='' plone_products_install='' zope_user=admin zope_password=admin http_port=8080 debug_mode=on verbose_security=on

plone3_buildout: A buildout for Plone 3 installation

This template creates a Plone 3 buildout (for most users, a preferred
way to get an installation of Plone 3 is to use the appropriate installer,
as these are all buildout-based since Plone 3.1)

If at any point, you need additional help for a question, you can enter
'?' and press RETURN.

**************************************************************************
**   *** NOTE: You probably don't want to use this template!

**  Since Plone 3.1, the preferred way to get a buildout-based setup
**  for Plone is to use the standard installer for your operating
**  system (the Windows installer, the Mac installer, or the Unified
**  Installer for Linux/Unix/BSD). These give you a best-practice,
**  widely-used setup with an isolated Python and a well-documented
**  buildout. This template is here for older versions of Plone and
**  for experts who explicitly want a raw, non-installer-based
**  installation.

**  (This message is particularly aimed at people following out-of-
**  date books/documentation that suggest this is the right way to get
**  a Plone-based buildout. This is no longer the case.)
**************************************************************************

Creating directory ./sexy

**************************************************************************
**   Generation finished.

**  You probably want to run python bootstrap.py and then edit
**  buildout.cfg before running bin/buildout -v".

**  See README.txt for details.
**************************************************************************

damon:cewing cbc$

The .zopeskel Configuration File­

But really, who want to script ZopeSkel beyond just the occasional template answer? It would be much more convenient to keep common answers to ZopeSkel templates in a configuration file which persists defaults for your preferences. So the sprint results provide just that. Put your preferences for answers to template questions in a .zopeskel file in your home directory, and bin/zopeskel­ will use those answers as its defaults.­ The format of .zopeskel is like a standard “INI” file read by Python’s ConfigParser standard module. It has a section for every ZopeSkel template and a [DEFAULT] section as well. Answers in the [DEFAULT] section apply to all templates while answers in any particular template section override answers in the [DEFAULT] section for that template only.­

bin/zopeskel has a handy option for making the .zopeskel file. You simply redirect the output of bin/zopeskel --make-config-file to ${HOME}/.zopeskel (like so for Unix-like OSes):

damon:cewing cbc$ bin/zopeskel --make-config-file > ~/.zopeskel
damon:cewing cbc$ 

The .zopeskel file generated by the --make-config-file option has a section for every ZopeSkel template with every possible template question commented out. See the output of bin/zopeskel --make-config-file­.­

You simply edit the generated .zopeskel file to uncomment the answers you want for defaults and provide your answers. Those answers then become the defaults the next time you run bin/zopeskel. You can simply hit the enter key when a question comes up for which you have already supplied your preferred default.

In the following example, my .zopeskel­ file shows I would like to use expert mode for every template:

­damon:cewing cbc$ cat ~/.zopeskel

# This file can contain preferences for zopeskel.
# To do so, uncomment the lines that look like:
#    variable_name = Default Value

[DEFAULT]
expert_mode = expert

[archetype]
# Expert Mode? (What question mode would you like? (easy/expert/all)?)
# expert_mode = easy

# Project Title (Title of the project)
# title = Example Name

[...snip...]
damon:cewing cbc$

­­So when I run bin/zopeskel, my expert mode default is expert instead of easy:

­damon:cewing cbc$ bin/zopeskel plone3_buildout

plone3_buildout: A buildout for Plone 3 installation

[...snip...]

Expert Mode? (What question mode would you like? (easy/expert/all)?) ['expert']:

­­­[...snip...]

ZopeSkel Generated Documentation

Documentation was a major thrust of the sprint. With so many people maintaining templates in ZopeSkel, not only do we need automatically generated diagrams showing the template dependencies, we need automatically generated documentation about the templates themselves as well. We decided we wanted ZopeSkel documentation which would not go out of date easily and thus created documentation generators. The sprint created a zopeskel.doctools subpackage (used in the bbq-sprint-cbcunc branch f­or zopeskel.doctools.graph) and added the html_doc module to the subpackage in the bbq-sprint-cewing branch. The html_doc module generates an HTML document of all templates, their summaries, their help information, their fields (the template question/­variable names), and their local commands (as well as the summaries, help information, and fields for each local command).

Behold the documentation.

Could be prettier. I’m sure it will be in time.

ZopeSkel as WSGI Application: Pasteweb

Part of the motivation behind the sprint has been not only to simplify ZopeSkel for Plone themers and integrators, but to integrate those simplifications into http://paster.joelburton.com. Thanks for Chris Rossi, this goal really flowered. Now you can run ZopeSkel in your browser served from your own laptop. A new ­zopeskel.webui package enables running a new executable, bin/pasteweb­, which serves up ZopeSkel on localhost port 6666 by default.­­

See the ­Try It Out section of this report for how to play with this package. 

The front page of the application allows your selection of a template:

Pasteweb Front Page

Clicking on the help link renders up the same new help content provided by bin/zopeskel

Pasteweb Help Widget

Clicking on a template name takes you to the “pre-run” page where the template is described and you enter your project name:

Pasteweb Begin Page

Clicking on the Continue button after entering the project name brings you to the main template question and answer page. A great advantage of this approach is you see all the template questions and default answers at once:

Pasteweb Main Page

The template widgets all correspond to the field types of the template answers. This is because the sprint ­subclassed the template variables into different types: strings, booleans, and multi-choice. These subclasses all normalize their own values and provide validation for earlier and better feedback on inappropriate values.­

Clicking on the Continue button after after editing the template defaults takes you to a Summary page where you can examine your answers all at once before accepting them and generating your product:

Pasteweb Summary Page

When you are satisfied with your answers, click the Finish button. Your new product is written to disk. Currently the new “post-run” messages are displayed in an unstylized fashion for a final page:

Pasteweb Finish Page

Our intention, of course, is to give the entire bin/pasteweb application a little more visual flair. Possibly even several pieces of flair. Certainly more than the minimum number of pieces flair.

Pasteweb is now available online at http://pasteweb.joelburton.com/ for your experimental pleasure.

Details, Details, and More Details

There are many more features and improvements added to ZopeSkel during the sprint:

  • Template questions now have a “title” attribute which is a bit friendlier than the question identifiers themselves.
  • All the tests now run.
  • A whole bunch more tests were added, especially for new features.
  • A proposal was written to split ZopeSkel into several eggs. ­This proposal requires discussion with a wider community. But it will allow less disruptive integration of new template bundles into ZopeSkel, such as Matthew WIlkes’s wsgitemplates, as well as project specific template bundles:
    • zopeskel.base
    • zopeskel.zope
    • zopeskel.plone
    • zopeskel.silva
  • A new ZopeSkel email list was created on plone.org in order to better coordinate work on ZopeSkel and ZopeSkel releases. At least some of the discussion concerning the splitting proposal will take place on the new email list.

Try It Out

To experiment with bin/zopeskel in the short term:

cd /tmp
virtualenv --no-site-packages -p /usr/bin/python2.4 test-new-zopeskel
cd test-new-zopeskel/
source bin/activate
svn co http://svn.plone.org/svn/collective/ZopeSkel/branches/bbq-sprint-cewing
cd bbq-sprint-cewing/
python setup.py develop
python setup.py test
zopeskel

­To experiment with bin/pasteweb in the short term:

cd ..
svn co http://svn.plone.org/svn/collective/zopeskel.webui/trunk/zopeskel.webui
cd zopeskel.webui
python setup.py develop
python setup.py test
pasteweb
­­­­­­­­

Credits, Lessons Learned, and the Road Ahead

The No-Fun ZopeSkel BBQ Sprint accomplished 23 major tasks in four days primarily by four sprinters.

We are very excited by the productivity and usefulness of the sprint and feel there are some lessons to impart:

  • Smaller sprints are by far more productive.
  • Ruthlessly focused sprints are more productive. Having super-clear goals and not wavering from them is key.
  • Excluding topics which don’t exactly fit goals is not a bad idea.
  • Design discussion and documentation ahead of the sprint make for a more productive sprint.
  • Inviting capable sprinters with strong motivations and undivided attention is absolutely necessary.
  • Make sure the coding robot named Joel Burton is on your team.
  • Bounties are not all they are cracked up to be. They take a lot of work. There may be easier ways to raise travel expenses.
  • A work environment geared towards serious concentration with no interruptions or distractions is extremely helpful.
  • Starting as early as feasible each day and working for about ten hours is most productive.
  • A lunch break which involves walking to a location away from the work environment refreshes the afternoon’s work.
  • IRC, Twitter, UStream and other open communication channels are distractions while sprinting. Help yourselves before helping others outside the sprint while it is sprint-time. There will be time to help others after the sprint and a sprint which doesn’t produce helps nobody.
  • Sprint now, report out later. Blogging is another distraction while sprinting. Help the sprint first.
  • Photographing whiteboards is a nice security blanket which doesn’t take much time.
  • Have the network set up the day before. Don’t go wireless. Have a high speed switch on a fat pipe.
  • Have a couple of nice dinners in the middle of the sprint. Make lunch fun. Eat BBQ every day. Have BBQ on your pizza. People who have fun together work together better.
  • Get plenty of sleep. Don’t stay out all night.
  • Get the nicest possible accommodations. Private accommodations entirely taken over by the sprinters are best.
  • Do not fit three people in the front seat of a pick-up truck.
  • More Deerhoof.

There’s a lot of clean-up work left over from this sprint. We could have used an extra day. It would have been wrong to cut short the work being completed on the final day in order to make a second ZopeSkel release in four days. Plus, some clean-up work depends on the outcome of discussions regarding the previously mentioned splitting proposal. Suffice to say, there will be at least a couple of people merging branches into trunk at the Plone Conerence 2009 Sprint.

Thanks

Once again, thanks to all who contributed to defray the air fares of sprinters:

  • Chris Calloway
  • Matthew Bowen
  • Jan-Jaap Driessen
  • Michael Revoir
  • Justin Bennett
  • Stephen McMahon
  • Andrzej Mleczko
  • Stephen Compall
  • Fabian Reinhard
  • Robert Gimlich
  • Francis Ridder
  • Calvin Hendryx-Parker
  • Clayton Parker
  • Jon Stahl
  • David Glick
  • Aaron VanDerlip
  • Don Fick
  • Nath Aune
  • Ken Wasetis
  • Don Ross
  • Karl Johan Kleist
  • Dorneles Treméa
  • Joel Burton
  • Alex and Amy Clark

The accommodations were provided free of charge by friends of TriZPUG and the sprinters themselves took care of their meals and catering expenses. Thanks to Josh Johnson for continuing to preach ZopeSkel and especially thanks to Joel Burton for pursing the idea of this sprint for the last year.

Finally, thanks to Daniel Nouri for inventing ZopeSkel, and to hannosch, optilude, davconvent, wichert, fschulze, mustap, MatthewWilkes, and tarek for keeping it maintained for us. We had quite a nice place to start.

­­­­­

Filed October 13th, 2009 under Uncategorized
  1. Wow! Absolutely fantastic work, sprinters, and nicely reported, Chris. ZopeSkel as WSGI Application? You just blew my mind.

    My only nit would be that the diagrams be more UML: rectangles for classes, arrows point to superclasses.

    Comment by Sean Kelly on October 13, 2009 at 10:16 am

  2. Thanks, Sean. Apparently, there are people doing UML-ish looking diagrams with GraphViz: http://www.ffnn.nl/pages/articles/media/uml-diagrams-using-graphviz-dot.php. Right now, it’s just a standard out of the box GraphViz “directed graph.” We’ll see. There are options in zopeskel.doctools.graph to put a lot more information on the dependency visualization. And when we use them, the graph gets huge fast. Too big to look at on the web. Too big to print. Too big for most monitors without scrolling. So putting things like all the template fields and local commands in a UML class diagram might be kind of outsized. What we were looking for was something to keep documentation up to date. I’m most interested in having the output of zopeskel.doctools.graph hotlink to the output of zopeskel.doctools.html_doc. That would take care of what a class diagram would tell you, but in a more readable fashion. The object is to generate documentation in support of plone.org tutorials.

    Comment by Chris Calloway on October 13, 2009 at 3:52 pm

  3. Very impressive work that you guys accomplished in just a few days! Kudos to Chris and everyone who participated. What is the timeline for merging these changes back to trunk, or making a ZopeSkel release on the cheeseshop? I’d like to introduce the new ZopeSkel at my training next week, but would prefer to tell students to type “easy_install ZopeSkel” instead of the svn checkout steps described above.

    The current ZopeSkel command to create a new plone3 buildout is still using plone.recipe.plone instead of the Plone egg.
    http://svn.plone.org/svn/collective/ZopeSkel/trunk/zopeskel/templates/plone3_buildout/buildout.cfg_tmpl

    Comment by Nate Aune on October 20, 2009 at 9:46 am

  4. You can easy_install ZopeSkel now. But discussion about merging and releasing a new ZopeSkel probably won’t even start until the conference sprint. I wouldn’t recommend having anyone install out of branches for a class. That’s for inspecting the work of the sprint.

    The sprint wasn’t about templates. That’s where Plone developers put their best practices. The sprint was to improve the usability of ZopeSkel itself. In fact, one of the things to discuss at the conference sprint is refactoring ZopeSkel to put the templates in subpackages to separate the idea of what a template is from what ZopeSkel is.

    ZopeSkel has previously been viewed as a collection of developer templates rather than an easy to use interface to best practice templates for integrators and theres. We’re trying make ZopeSkel an easy to use interface to any collection of best practice templates, even dropping in your own subpackage of templates.

    Comment by Chris Calloway on October 23, 2009 at 8:47 pm

Leave a comment

To comment on this blog you will need to log in or create an account first.