Last problem … the build needs to download two large tarballs (Zope, and opencore’s Plone Product bundle) from dist.socialplanning.org, which is a Github Pages site.  It seems that Github has decided to throttle downloads aggressively from the server I’m building on: both downloads get 20-80% complete and then hang indefinitely, whether I’m using the fassembler build code or wget.  So I had to download the tarballs to my local computer, scp them to the server, put them in the root of the build directory, and then convince fassembler not to re-download them.  I haven’t yet cleaned up & committed the code for this one — I just edited it locally to comment out the download logic — so I’ll circle back to it before my next rebuild.

Next rebuild in progress but I haven’t circled back to it yet.  :-(  I need to fix this. 

Filed January 26th, 2014 under openfsm, opencore

The openfsm project export is happily chugging along (it’s been running all night; we’re almost through the “e”s) so this is a good time to circle back and write up some opencore build troubles I ran up against yesterday.  It had been about a month since I last built an opencore site, so, as usual, the build stopped working.  Here’s the problems I ran into:

  1. The setuptools bootstrapping code in fassembler-boot.py decided to fetch setuptools 2.0 (!there’s a 2.0 now!?) which doesn’t support Python 2.4.  Normally we install the venerable setuptools 0.6c11.  The bootstrapping code already has built-in logic to look for an existing setuptools egg in one of several filesystem locations before resorting to the internet.  So I just had to add a step to wget setuptools-0.6c11-py2.4.egg into the root of the build directory before fetching and executing fassembler-boot.py, in the rebuild-opencore-site script.  I added that here.
  2. The opencore build was set up to use an editable SVN checkout of zope.dottedname 3.4.5 for some reason instead of just installing that release from PyPI.  This wasn’t working yesterday, probably because of the recent Zope SVN move.  So I just changed the build requirements to install the release from PyPI instead.
  3. Wait, no, that broke everything!  After building an opencore with that release from PyPI, my Zope environment had gone crazy: attempting to start it up at all failed with an `ImportError: no module interface` during `from zope.interface import Interface`.  At first I assumed I had somehow pulled in some post-3.4 zope.* egg that ended up installing a whole parallel zope universe.  But this opencore-dev thread gave me a useful hint.  It turns out that fassembler’s version of pip deals badly with namespace packages installed non-editably: doing that non-editable install of zope.dottedname caused pip to clobber the entire zope.* namespace.  So instead I changed the build requirements to install the release editably from Github.
  4. Last problem … the build needs to download two large tarballs (Zope, and opencore’s Plone Product bundle) from dist.socialplanning.org, which is a Github Pages site.  It seems that Github has decided to throttle downloads aggressively from the server I’m building on: both downloads get 20-80% complete and then hang indefinitely, whether I’m using the fassembler build code or wget.  So I had to download the tarballs to my local computer, scp them to the server, put them in the root of the build directory, and then convince fassembler not to re-download them.  I haven’t yet cleaned up & committed the code for this one — I just edited it locally to comment out the download logic — so I’ll circle back to it before my next rebuild.
Filed December 15th, 2013 under openfsm

Today I am:

  • Copying the live openfsm site’s Data.fs to a backup location
  • Building a new opencore site (frontdown.openfsm) against the copied Data.fs
  • Running the full export procedure against the newly built site 

And also:

  • https://github.com/socialplanning/opencore/issues/33 (code to import member account notifications)

Next step will be to run the full import procedure:

  • Build a new opencore site with a fresh ZODB
  • Run the import scripts
  • Test
  •  

 

Filed December 14th, 2013 under openfsm

Here’s my current draft of what the openfsm migration procedure will involve:

  1. Build new opencore site
  2. During build, enter the admin password from the live site’s $VAR/admin.txt
  3. After build, manually set new site properties in a debug shell per https://github.com/socialplanning/opencore/issues/34
  4. On old site, run a full export of all projects
  5. Shut off old site, turn on “Down for Maintenance” page 
  6. On old site, export all members (including member wikis and wiki attachments, member logos, and member account notifications)
  7. On old site, export all site news (including wiki history and attachments)
  8. On old site, incrementally re-export all site projects
  9. On new site, import all members
  10. On new site, import all news
  11. On new site, import all projects
  12. On new site, clear and rebuild all catalogs (maybe handled in project import script)
  13. Test
  14. Turn on new site
Filed November 3rd, 2013 under openfsm

Before I can import projects, I need to import users.

I have an export script that will dump all user data to a CSV.  So now I need to write another script to read that CSV and create a new member object for each row.

Apparently it is not possible (at least not easily possible) to create a Plone member through a zopectl script.  Trying to reuse code from opencore’s member join view fails with strange exceptions.  So does this script

The exceptions are things like AttributeError: unicodeEncode on the context object in Archetypes schema validation code, or “Called without a request as the context” messages in Products.Five.i18n calls.

It turns out these are all (one way or another) happening because we’re running it from a zopectl run script.  If you run the same code through a browser view, it works — because there you have a real request and have access to pythonscript content that doesn’t exist outside of an HTTP request.  (The unicodeEncode method is apparently one of these.)

So, I’m going to create a site-manager-only view in opencore/browser/management.py which will trigger the member import code.

Filed June 2nd, 2013 under openfsm

A few notes on problems I ran into while running the export_all_projects script on a real server with real content.

1) The wiki export uses bazaar, which requires a Python compiled with bz2 support.  This means running something like sudo apt-get install libbz2-dev before installing Python or compiling it with ./configure –enable-bz2 && make

2) On some projects I got this error while exporting mailing list archives:

Traceback (most recent call last):
  File "“, line 1, in ?
  File “src/opencore/opencore/scripts/export_all_projects.py”, line 71, in ?
    path = qview.export(proj_id, status)
  File “opencore/src/opencore/opencore/export/export_utils.py”, line 274, in export
    exporter.save()
  File “opencore/src/opencore/opencore/export/export_utils.py”, line 315, in save
    self.save_list_archives()
  File “opencore/src/opencore/opencore/export/export_utils.py”, line 550, in save_list_archives
    tmpfd, tmpname = em.export_messages_to_tempfile()
  File “opencore/src/opencore-listen/Products/listen/extras/import_export.py”, line 179, in export_messages_to_tempfile
    temp_outfile.write(self._convert_to_mbox_msg(msg.getObject()))
  File “opencore/lib/zope/lib/python/Products/ZCatalog/CatalogBrains.py”, line 77, in getObject
    parent = parent.unrestrictedTraverse(path[:-1])
  File “opencore/lib/zope/lib/python/OFS/Traversable.py”, line 259, in unrestrictedTraverse
    next = queryMultiAdapter((obj, self.REQUEST),
AttributeError: REQUEST

I don’t know what causes it, but a patch in Traversable.py seems to make the error go away without losing any content from the export.  I just check for the existence of self.REQUEST prior to that line, and set next = None if the request doesn’t exist.

3) And sometimes I get this error when exporting wiki history:

Traceback (most recent call last):
  File ““, line 1, in ?
  File “src/opencore/opencore/scripts/export_all_projects.py”, line 71, in ?
    path = qview.export(proj_id, status)
  File “opencore/src/opencore/opencore/export/export_utils.py”, line 274, in export
    exporter.save()
  File “opencore/src/opencore/opencore/export/export_utils.py”, line 323, in save
    self.save_wiki_history()
  File “opencore/src/opencore/opencore/export/export_utils.py”, line 372, in save_wiki_history
    clonedir = converter.convert()
  File “opencore/src/opencore/opencore/nui/wiki/bzrbackend.py”, line 88, in convert
    self.sort_checkins()
  File “opencore/src/opencore/opencore/nui/wiki/bzrbackend.py”, line 130, in sort_checkins
    for version in versions:
  File “opencore/zope/Products/CMFEditions/CopyModifyMergeRepositoryTool.py”, line 785, in next
    return self._getItem(self._pos)
  File “opencore/zope/Products/CMFEditions/CopyModifyMergeRepositoryTool.py”, line 762, in __getitem__
    self._countPurged)
  File “opencore/zope/Products/CMFEditions/CopyModifyMergeRepositoryTool.py”, line 474, in _retrieve
    countPurged=countPurged)
  File “opencore/zope/Products/CMFEditions/CopyModifyMergeRepositoryTool.py”, line 533, in _recursiveRetrieve
    preserve, countPurged)
  File “opencore/zope/Products/CMFEditions/ArchivistTool.py”, line 314, in retrieve
    return history[selector]
  File “opencore/zope/Products/CMFEditions/ArchivistTool.py”, line 470, in __getitem__
    self._preserve)
  File “opencore/zope/Products/CMFEditions/ModifierRegistryTool.py”, line 254, in afterRetrieveModifier
    to_be_del, attrs, preserve = mod.afterRetrieveModifier(obj, repo_clone)
  File “opencore/zope/Products/CMFEditions/StandardModifiers.py”, line 647, in afterRetrieveModifier
    if ref.getId() not in repo_clone_ref_ids:
  File “opencore/lib/zope/lib/python/ZODB/Connection.py”, line 761, in setstate
    self._setstate(obj)
  File “opencore/lib/zope/lib/python/ZODB/Connection.py”, line 819, in _setstate
    self._reader.setGhostState(obj, p)
  File “opencore/lib/zope/lib/python/ZODB/serialize.py”, line 604, in setGhostState
    state = self.getState(pickle)
  File “opencore/lib/zope/lib/python/ZODB/serialize.py”, line 597, in getState
    return unpickler.load()
  File “opencore/lib/zope/lib/python/ZODB/serialize.py”, line 471, in _persistent_load
    return self.load_oid(reference)
  File “opencore/lib/zope/lib/python/ZODB/serialize.py”, line 537, in load_oid
    return self._conn.get(oid)
  File “opencore/lib/zope/lib/python/ZODB/Connection.py”, line 214, in get
    obj = self._reader.getGhost(p)
  File “opencore/lib/zope/lib/python/ZODB/serialize.py”, line 569, in getGhost
    klass = unpickler.load()
UnpicklingError: invalid load key, ‘>’.

Here I don’t think there’s a way to work around the error without losing content — this error apparently indicates damage to the physical disk which is unsurprising since that’s the reason I’m doing this whole export/import process in the first place.  So I think I just need to catch all instances of (pickle.UnpicklingError, cPickle.UnpicklingError) while unfolding the versions list in bzrbackend.sort_checkins, and just discard the version history of any page where that occurs.  I should take note of everywhere that this happens on the openfsm. export — that way we can decide what to do with the information (e.g. notify all users whose content was lost)

Here’s the patch I’m using (which I’ll commit later) – https://gist.github.com/ejucovy/5563378 

4) This error came up when trying to write the bzr history of a file with a really long name:

Traceback (most recent call last):                                                                                                                                          
  File "<string>", line 1, in ?
  File "src/opencore/opencore/scripts/export_all_projects.py", line 74, in ?
    path = qview.export(proj_id, status)
  File "opencore/src/opencore/opencore/export/export_utils.py", line 274, in export
    exporter.save()
  File "opencore/src/opencore/opencore/export/export_utils.py", line 323, in save
    self.save_wiki_history()
  File "opencore/src/opencore/opencore/export/export_utils.py", line 372, in save_wiki_history
    clonedir = converter.convert()
  File "opencore/src/opencore/opencore/nui/wiki/bzrbackend.py", line 92, in convert
    self.port_checkins()
  File "opencore/src/opencore/opencore/nui/wiki/bzrbackend.py", line 202, in port_checkins
    timestamp=timestamp)
  File "opencore/src/sven/sven/bzr.py", line 450, in write
    f = file(absolute_uri, mode)
IOError: [Errno 36] File name too long: '/tmp/tmpwdq2h3/bzr_checkouts/projects/project-name/main-site/really-long-name-that-looks-sort-of-buggy-because-it-ends-with-some-html-tags-and-style-declarataions'

To deal with this, I catch the IOError 36 and hash the filename if necessary.  So now I also need to maintain and export a mapping of page ids <=> filesystem names.  I’ll export it as JSON.

Filed May 12th, 2013 under openfsm

Last week I got through these tasks:

  • Finish copying openfsm’s Data.fs to the backup site.  When this is done, I’ll restart zeo, wait a while for its indexes to be rebuilt, restart zope, and continue debugging the build — I’m sure I don’t have ZCMLLoader properly set up for example, so the opencore plugin packages won’t be properly installed yet.
  • Finish debugging the build until I have a fully working openfsm.net clone
  • Fix the fassembler build so that I can reliably build a new site without doing all this work by hand — I’m going to need to build a lot of fresh opencore sites to test the import process, which will be much slower if I don’t have a working automated build
  • Build a clean opencore site with the working automated fassembler build

I haven’t yet done this: “Run the master export script to produce a “first draft” of a complete export, and fix any errors that spring up along the way”

So, my goal for this week is to:

  1. Write the master export script — I’m pretty sure I don’t actually have a script yet that iterates over all people and projects and exports them all
  2. Run the master export script against the backup site
  3. Debug any errors that come up during the export script
  4. Write a master import script
  5. Run the master import script against a new site
  6. Debug any errors that come up during the import script

When that’s done, we’ll be able to move on to a testing phase, checking for missing or incorrect content and broken functionality.

It’s going to take a very long time to run the export and import scripts, so a more realistic goal for this week would be to get through step 3 above — just debugging the export script until I have a full site export file.

Filed May 7th, 2013 under openfsm

The short version is: we need to use virtualenv==1.5.2 and pip==1.1 everywhere.  This includes the environment we create by hand to install opencore’s bootstrapper, and also every environment created by the bootstrapper.

I’ve pushed changes to fassembler to ensure that it creates all internal environments with those versions.

To install opencore-fassembler_boot itself: 

git clone git://github.com/pypa/virtualenv.git 
cd virtualenv
git checkout 1.5.2
./virtualenv.py –python=python2.4 ../openfsm.net
source ../openfsm.net/bin/activate
git clone git://github.com/socialplanning/opencore-fassembler_boot
cd opencore-fassembler_boot
python setup.py develop
cd ..
new-opencore-site backup.openfsm.net  10000
cd backup.openfsm.net 
rebuild-opencore-site -w 

 

Filed May 5th, 2013 under openfsm

My current high level goal is to test and finalize the overall export-and-import process by:

  1. Cloning openfsm.net to a new site with a copy of the database, and running the master export script to produce a complete export
  2. Building a clean opencore site with an empty database, and running the master import script to recreate the content from openfsm.net
  3. Fixing things that break in either the export or the import process
  4. Testing the imported site and taking note of missing content or things that didn’t import cleanly (e.g. logins or permissions not working)
  5. Updating the cloned openfsm.net code with my fixes
  6. Repeating steps 2-5 until the export-and-import process seems to be complete to my satisfaction
  7. Asking Pierre and others to test the imported site and try to spot errors

 

When this is done, we’ll be able to do a final export-and-import of all content from the live site.

 

This week I:

  • Reviewed the open export/import tickets at http://github.com/socialplanning/opencore/issues
  • Built a new-opencore-site on the openfsm server .. ugh this was hard and messy.  Fassembler refuses to adequately pin virtualenv and pip to versions old enough to support py24 so I basically had to throw out fassembler and build opencore by hand.  Lots of problems here but I seem to finally have it more or less working — at least, Zope is installed.  Will find more problems as I go.
  • Started to copy openfsm’s Data.fs to the backup site … but ran out of space on disk.  Need to ask Dimo for help here.
  • Started to copy openfsm’s Data.fs to my local computer.

 

My goals for next week are to:

  1. Finish copying openfsm’s Data.fs to the backup site.  When this is done, I’ll restart zeo, wait a while for its indexes to be rebuilt, restart zope, and continue debugging the build — I’m sure I don’t have ZCMLLoader properly set up for example, so the opencore plugin packages won’t be properly installed yet.
  2. Finish debugging the build until I have a fully working openfsm.net clone
  3. Run the master export script to produce a “first draft” of a complete export, and fix any errors that spring up along the way
  4. Fix the fassembler build so that I can reliably build a new site without doing all this work by hand — I’m going to need to build a lot of fresh opencore sites to test the import process, which will be much slower if I don’t have a working automated build
  5. Build a clean opencore site with the working automated fassembler build

Filed April 27th, 2013 under openfsm