• Mounting portal_catalog in a separate ZODB

last modified July 10, 2010 by egj

What's this about?

  • Zope's in-memory ZODB cache is very important for performance
  • the cache is configured by number of objects to store, not by size
  • Big binary data such as wiki attachments and images tend to thrash the ZODB cache since they are stored as a linked list of 64 kB persistent objects; so, if the cache is full (and it looks like it usually is), loading eg. one 10MB file will cause 160 other objects to be evicted from the cache.
  • portal_catalog also makes very heavy use of ZODB cache, because there tend to be lots of small objects there.  Plone (and opencore) relies VERY heavily on portal_catalog.
  • we have a fair number of binary attachments too
  • therefore, coactivate.org probably doesn't get very good usage out of the ZODB cache
  • as a no-effort-involved workaround, we have our cache size set to 10000 objects, which eats a fair amount of RAM
  • if we were to run the catalog on a separate ZODB from the rest of the content, there would be two ZODB caches, each would not interfere with the other at all, each would be used more optimally, and then each could probably be configured smaller
  • profit!

More docs on tuning ZEO cache size here:

http://wiki.zope.org/zope2/ZEOCache

... which is probably worth reading regardless of whether or not you mount the catalog on a separate ZODB.

­These instructions are based on the ones for generic Plone here: http://plone.org/documentation/kb/mount-portal_catalog-in-separate-zodb

First we make a (temporary) new Zope instance: ­

cd /path/to/opencore/site/build/opencore
source bin/activate
./lib/zope/bin/mkzopeinstance

Fill in any values you like for the zope instance path (I'll call it /newzopepath) and admin credentials.

Add the opencore bundle's Products:

 ln -s ./src/opencore-bundle/* /newzopepath/Products/


Add the following lines to /newzopepath/etc/zope.conf --

<zodb_db empty>

<filestorage>

 path /newzopepath/var/empty.fs

 </filestorage>

 mount-point /empty

</zodb_db>


Now start the Zope site.

 cd /newzopepath
 ./bin/zopectl fg


If you have oc-cabochon installed you might get an error "CabochonConfigError: no cabochon_user_info file specified in zope.conf opencore.nui" -- if so, append these lines (with appropriate substitutions) to /newzopepath/etc/zope.conf and then restart the zope instance:

<product-config opencore.nui>
   cabochon_messages  /path/to/opencore/site/var/opencore/cabochon-messages
   cabochon_user_info /path/to/opencore/site/var/cabochon-password.txt
</product-config>  


Now visit the new Zope instance's ZMI, and add a "ZODB Mount Point" object. Check the "Create new folders if the mounted objects don't yet exist" checkbox and add the "empty" database.

The "/empty" path should now be accessible within the ZMI. Inside that folder, add a new Plone Site.  Name it "openplans".  (I'm not sure if the ID here matters, so to be safe I'm giving it the same ID as my real opencore site.  Yours is probably "openplans" too since that's been hardcoded in opencore for a while.)  Select the "OpenCore default site" extension profile and then click "Add plone site."  (You need to install the OpenCore profile in order to get the right set of indexes on the portal_catalog created inside it.)

Next, copy the new database to your real site's var directory:

cp /newzopepath/var/empty.fs /path/to/opencore/site/var/zeo/catalog.fs  

Now you can shut down the new Zope site.  You can even rm -rf /newzopesite -- it's served its purpose.

­­­­Now we will replace your site's portal_catalog with the one in the catalog.fs database. 

This is the part of the process that requires downtime; from this point on, your site will not work properly until we're done.



Using the ZMI, delete the "portal_catalog" object from your (real) OpenCore site.

Add the following to your /path/to/opencore/site/build/opencore/zeo/etc/zeo.conf --

<filestorage 2>
   path $ZODB_DIR/catalog.fs
</filestorage>    

And add the following to your /path/to/opencore/site/build/opencore/zope/etc/zope.conf --

 <zodb_db catalog>
   mount-point /­openplans/portal_catalog:/empty/openplans/portal_catalog
   # ZODB cache, in number of objects, per thread                                                            
   cache-size 10000
   <zeoclient>
     server $VAR/zeo/zdsock
     storage 2
     name zeostorage
     var $OPENCORE_VAR
     # ZEO client disk cache, in bytes                                                                       
     cache-size 500MB
     # Uncomment to have a persistent disk cache
     #client zeo1                                                                                            
   </zeoclient>
­ </zodb_db>    

Almost done now!  We just need to create the portal_catalog in the new ZODB and rebuild the indexes.

  1. Restart your Zeo and Zope again.

  2. Using the ZMI, add a "ZODB Mount Point" inside your OpenCore Site. The "portal_catalog" database should be available. Dont forget to check "create missing folders...".

  3. Check that you now have a "portal_catalog" object visible in the ZMI.

  4. Using the ZMI, go to the portal_catalog's "advanced" tab and click "clear and rebuild".  This will take a while if you have a lot of content.  When it's done, your site will be working again, and you now have the portal_catalog mounted on its own ZODB.