Mounting portal_catalog in a separate ZODB
What's this about?
- Zope's in-memory ZODB cache is very important for performance
- the cache is configured by number of objects to store, not by size
- Big binary data such as wiki attachments and images tend to thrash the ZODB cache since they are stored as a linked list of 64 kB persistent objects; so, if the cache is full (and it looks like it usually is), loading eg. one 10MB file will cause 160 other objects to be evicted from the cache.
- portal_catalog also makes very heavy use of ZODB cache, because there tend to be lots of small objects there. Plone (and opencore) relies VERY heavily on portal_catalog.
- we have a fair number of binary attachments too
- therefore, coactivate.org probably doesn't get very good usage out of the ZODB cache
- as a no-effort-involved workaround, we have our cache size set to 10000 objects, which eats a fair amount of RAM
- if we were to run the catalog on a separate ZODB from the rest of the content, there would be two ZODB caches, each would not interfere with the other at all, each would be used more optimally, and then each could probably be configured smaller
More docs on tuning ZEO cache size here:
... which is probably worth reading regardless of whether or not you mount the catalog on a separate ZODB.
First we make a (temporary) new Zope instance:
cd /path/to/opencore/site/build/opencore source bin/activate ./lib/zope/bin/mkzopeinstance
Fill in any values you like for the zope instance path (I'll call it /newzopepath) and admin credentials.
Add the opencore bundle's Products:
ln -s ./src/opencore-bundle/* /newzopepath/Products/
Add the following lines to /newzopepath/etc/zope.conf --
Now start the Zope site.
cd /newzopepath ./bin/zopectl fg
If you have oc-cabochon installed you might get an error "CabochonConfigError: no cabochon_user_info file specified in zope.conf opencore.nui" -- if so, append these lines (with appropriate substitutions) to /newzopepath/etc/zope.conf and then restart the zope instance:<product-config opencore.nui> cabochon_messages /path/to/opencore/site/var/opencore/cabochon-messages cabochon_user_info /path/to/opencore/site/var/cabochon-password.txt </product-config>
Now visit the new Zope instance's ZMI, and add a "ZODB Mount Point" object. Check the "Create new folders if the mounted objects don't yet exist" checkbox and add the "empty" database.
The "/empty" path should now be accessible within the ZMI. Inside that folder, add a new Plone Site. Name it "openplans". (I'm not sure if the ID here matters, so to be safe I'm giving it the same ID as my real opencore site. Yours is probably "openplans" too since that's been hardcoded in opencore for a while.) Select the "OpenCore default site" extension profile and then click "Add plone site." (You need to install the OpenCore profile in order to get the right set of indexes on the portal_catalog created inside it.)
Next, copy the new database to your real site's var directory:
cp /newzopepath/var/empty.fs /path/to/opencore/site/var/zeo/catalog.fs
Now you can shut down the new Zope site. You can even rm -rf /newzopesite -- it's served its purpose.
Using the ZMI, delete the "portal_catalog" object from your (real) OpenCore site.
Add the following to your /path/to/opencore/site/build/opencore/zeo/etc/zeo.conf --
<filestorage 2> path $ZODB_DIR/catalog.fs </filestorage>
And add the following to your /path/to/opencore/site/build/opencore/zope/etc/zope.conf --
<zodb_db catalog> mount-point /openplans/portal_catalog:/empty/openplans/portal_catalog # ZODB cache, in number of objects, per thread cache-size 10000 <zeoclient> server $VAR/zeo/zdsock storage 2 name zeostorage var $OPENCORE_VAR # ZEO client disk cache, in bytes cache-size 500MB # Uncomment to have a persistent disk cache #client zeo1 </zeoclient> </zodb_db>
Almost done now! We just need to create the portal_catalog in the new ZODB and rebuild the indexes.
Restart your Zeo and Zope again.
Using the ZMI, add a "ZODB Mount Point" inside your OpenCore Site. The "portal_catalog" database should be available. Dont forget to check "create missing folders...".
Check that you now have a "portal_catalog" object visible in the ZMI.
Using the ZMI, go to the portal_catalog's "advanced" tab and click "clear and rebuild". This will take a while if you have a lot of content. When it's done, your site will be working again, and you now have the portal_catalog mounted on its own ZODB.