http://trisquel.info/

http://www.thebaffler.com/past/the_meme_hustler

http://en.wikipedia.org/wiki/Psychogeography

http://www.patternlanguage.com/ 

Filed May 17th, 2013 under Uncategorized

A few notes on problems I ran into while running the export_all_projects script on a real server with real content.

1) The wiki export uses bazaar, which requires a Python compiled with bz2 support.  This means running something like sudo apt-get install libbz2-dev before installing Python or compiling it with ./configure –enable-bz2 && make

2) On some projects I got this error while exporting mailing list archives:

Traceback (most recent call last):
  File "“, line 1, in ?
  File “src/opencore/opencore/scripts/export_all_projects.py”, line 71, in ?
    path = qview.export(proj_id, status)
  File “opencore/src/opencore/opencore/export/export_utils.py”, line 274, in export
    exporter.save()
  File “opencore/src/opencore/opencore/export/export_utils.py”, line 315, in save
    self.save_list_archives()
  File “opencore/src/opencore/opencore/export/export_utils.py”, line 550, in save_list_archives
    tmpfd, tmpname = em.export_messages_to_tempfile()
  File “opencore/src/opencore-listen/Products/listen/extras/import_export.py”, line 179, in export_messages_to_tempfile
    temp_outfile.write(self._convert_to_mbox_msg(msg.getObject()))
  File “opencore/lib/zope/lib/python/Products/ZCatalog/CatalogBrains.py”, line 77, in getObject
    parent = parent.unrestrictedTraverse(path[:-1])
  File “opencore/lib/zope/lib/python/OFS/Traversable.py”, line 259, in unrestrictedTraverse
    next = queryMultiAdapter((obj, self.REQUEST),
AttributeError: REQUEST

I don’t know what causes it, but a patch in Traversable.py seems to make the error go away without losing any content from the export.  I just check for the existence of self.REQUEST prior to that line, and set next = None if the request doesn’t exist.

3) And sometimes I get this error when exporting wiki history:

Traceback (most recent call last):
  File ““, line 1, in ?
  File “src/opencore/opencore/scripts/export_all_projects.py”, line 71, in ?
    path = qview.export(proj_id, status)
  File “opencore/src/opencore/opencore/export/export_utils.py”, line 274, in export
    exporter.save()
  File “opencore/src/opencore/opencore/export/export_utils.py”, line 323, in save
    self.save_wiki_history()
  File “opencore/src/opencore/opencore/export/export_utils.py”, line 372, in save_wiki_history
    clonedir = converter.convert()
  File “opencore/src/opencore/opencore/nui/wiki/bzrbackend.py”, line 88, in convert
    self.sort_checkins()
  File “opencore/src/opencore/opencore/nui/wiki/bzrbackend.py”, line 130, in sort_checkins
    for version in versions:
  File “opencore/zope/Products/CMFEditions/CopyModifyMergeRepositoryTool.py”, line 785, in next
    return self._getItem(self._pos)
  File “opencore/zope/Products/CMFEditions/CopyModifyMergeRepositoryTool.py”, line 762, in __getitem__
    self._countPurged)
  File “opencore/zope/Products/CMFEditions/CopyModifyMergeRepositoryTool.py”, line 474, in _retrieve
    countPurged=countPurged)
  File “opencore/zope/Products/CMFEditions/CopyModifyMergeRepositoryTool.py”, line 533, in _recursiveRetrieve
    preserve, countPurged)
  File “opencore/zope/Products/CMFEditions/ArchivistTool.py”, line 314, in retrieve
    return history[selector]
  File “opencore/zope/Products/CMFEditions/ArchivistTool.py”, line 470, in __getitem__
    self._preserve)
  File “opencore/zope/Products/CMFEditions/ModifierRegistryTool.py”, line 254, in afterRetrieveModifier
    to_be_del, attrs, preserve = mod.afterRetrieveModifier(obj, repo_clone)
  File “opencore/zope/Products/CMFEditions/StandardModifiers.py”, line 647, in afterRetrieveModifier
    if ref.getId() not in repo_clone_ref_ids:
  File “opencore/lib/zope/lib/python/ZODB/Connection.py”, line 761, in setstate
    self._setstate(obj)
  File “opencore/lib/zope/lib/python/ZODB/Connection.py”, line 819, in _setstate
    self._reader.setGhostState(obj, p)
  File “opencore/lib/zope/lib/python/ZODB/serialize.py”, line 604, in setGhostState
    state = self.getState(pickle)
  File “opencore/lib/zope/lib/python/ZODB/serialize.py”, line 597, in getState
    return unpickler.load()
  File “opencore/lib/zope/lib/python/ZODB/serialize.py”, line 471, in _persistent_load
    return self.load_oid(reference)
  File “opencore/lib/zope/lib/python/ZODB/serialize.py”, line 537, in load_oid
    return self._conn.get(oid)
  File “opencore/lib/zope/lib/python/ZODB/Connection.py”, line 214, in get
    obj = self._reader.getGhost(p)
  File “opencore/lib/zope/lib/python/ZODB/serialize.py”, line 569, in getGhost
    klass = unpickler.load()
UnpicklingError: invalid load key, ‘>’.

Here I don’t think there’s a way to work around the error without losing content — this error apparently indicates damage to the physical disk which is unsurprising since that’s the reason I’m doing this whole export/import process in the first place.  So I think I just need to catch all instances of (pickle.UnpicklingError, cPickle.UnpicklingError) while unfolding the versions list in bzrbackend.sort_checkins, and just discard the version history of any page where that occurs.  I should take note of everywhere that this happens on the openfsm. export — that way we can decide what to do with the information (e.g. notify all users whose content was lost)

Here’s the patch I’m using (which I’ll commit later) – https://gist.github.com/ejucovy/5563378 

4) This error came up when trying to write the bzr history of a file with a really long name:

Traceback (most recent call last):                                                                                                                                          
  File "<string>", line 1, in ?
  File "src/opencore/opencore/scripts/export_all_projects.py", line 74, in ?
    path = qview.export(proj_id, status)
  File "opencore/src/opencore/opencore/export/export_utils.py", line 274, in export
    exporter.save()
  File "opencore/src/opencore/opencore/export/export_utils.py", line 323, in save
    self.save_wiki_history()
  File "opencore/src/opencore/opencore/export/export_utils.py", line 372, in save_wiki_history
    clonedir = converter.convert()
  File "opencore/src/opencore/opencore/nui/wiki/bzrbackend.py", line 92, in convert
    self.port_checkins()
  File "opencore/src/opencore/opencore/nui/wiki/bzrbackend.py", line 202, in port_checkins
    timestamp=timestamp)
  File "opencore/src/sven/sven/bzr.py", line 450, in write
    f = file(absolute_uri, mode)
IOError: [Errno 36] File name too long: '/tmp/tmpwdq2h3/bzr_checkouts/projects/project-name/main-site/really-long-name-that-looks-sort-of-buggy-because-it-ends-with-some-html-tags-and-style-declarataions'

To deal with this, I catch the IOError 36 and hash the filename if necessary.  So now I also need to maintain and export a mapping of page ids <=> filesystem names.  I’ll export it as JSON.

Filed May 12th, 2013 under openfsm

Last week I got through these tasks:

  • Finish copying openfsm’s Data.fs to the backup site.  When this is done, I’ll restart zeo, wait a while for its indexes to be rebuilt, restart zope, and continue debugging the build — I’m sure I don’t have ZCMLLoader properly set up for example, so the opencore plugin packages won’t be properly installed yet.
  • Finish debugging the build until I have a fully working openfsm.net clone
  • Fix the fassembler build so that I can reliably build a new site without doing all this work by hand — I’m going to need to build a lot of fresh opencore sites to test the import process, which will be much slower if I don’t have a working automated build
  • Build a clean opencore site with the working automated fassembler build

I haven’t yet done this: “Run the master export script to produce a “first draft” of a complete export, and fix any errors that spring up along the way”

So, my goal for this week is to:

  1. Write the master export script — I’m pretty sure I don’t actually have a script yet that iterates over all people and projects and exports them all
  2. Run the master export script against the backup site
  3. Debug any errors that come up during the export script
  4. Write a master import script
  5. Run the master import script against a new site
  6. Debug any errors that come up during the import script

When that’s done, we’ll be able to move on to a testing phase, checking for missing or incorrect content and broken functionality.

It’s going to take a very long time to run the export and import scripts, so a more realistic goal for this week would be to get through step 3 above — just debugging the export script until I have a full site export file.

Filed May 7th, 2013 under openfsm

The short version is: we need to use virtualenv==1.5.2 and pip==1.1 everywhere.  This includes the environment we create by hand to install opencore’s bootstrapper, and also every environment created by the bootstrapper.

I’ve pushed changes to fassembler to ensure that it creates all internal environments with those versions.

To install opencore-fassembler_boot itself: 

git clone git://github.com/pypa/virtualenv.git 
cd virtualenv
git checkout 1.5.2
./virtualenv.py –python=python2.4 ../openfsm.net
source ../openfsm.net/bin/activate
git clone git://github.com/socialplanning/opencore-fassembler_boot
cd opencore-fassembler_boot
python setup.py develop
cd ..
new-opencore-site backup.openfsm.net  10000
cd backup.openfsm.net 
rebuild-opencore-site -w 

 

Filed May 5th, 2013 under openfsm