• i18n Usage in OpenCore

last modified March 27, 2009 by egj

It's basically the same as the plone i18n system

The Zope 3 i18n architecture is used in opencore, for the following reasons:


  • to facilitate editing of text throughout the site
  • for internationalization
  • to create a shared repository of text for flunc to test against
  • to promote reuse of text snippets


The text managed by i18n originates from py, pt, zcml, and js files. In these files, each text snippet needs to be assigned a message id which is then automatically extracted by i18ndude into opencore/locales/opencore.pot, which is then synched with opencore/locales/en/LC_MESSAGES/opencore.po. Then, when the text snippet needs to be edited, you just need to edit en/LC_MESSAGES/opencore.po by hand or by using a gui like poEdit. To deploy the change you have to svn up the opencore/opencore/locales/ directory.

Here is how you mark up new text snippets in the py and pt files.

py files

At the top of the file make sure this is there:

from opencore.i18n import _

then for a simple message:

self.add_status_message('Welcome! You have signed in.')

# 'self' is any view instance of a subclass of opencore.nui.base.BaseView

change it to:

self.add_status_message(_(u'psm_signin_welcome', u'Welcome! You have signed in.'))

choose a message id ('psm_signin_welcome') that conforms to the established conventions. 'psm' prefix for portal status message, 'err' for error message. See the opencore.po file for examples.

For more complex messages that have values substituted within them like:

self.add_status_message('Thanks for joining OpenPlans, %s!\nA confirmation email has been sent to you with instructions on activating your account.' % mem_id)

you need to pass in a mapping:

self.add_status_message(_(u'psm_thankyou_for_joining', u'Thanks for joining OpenPlans, ${mem_id}!\nA confirmation email has been sent to you with instructions on activating your account.', mapping={u'mem_id':mem_id}))

(It is not possible to use other i18n messages as values in the mapping; translation is not recursive. If you really need to do that, you'll have to call translate() on the inner message before you put it in the mapping.)

BaseView's add_status_message method delegates to opencore.i18n.translate, which you can call directly if you need to translate user-facing text that is not a status message. Generally, though, the translate function doesn't need to be called because text is automatically translated when it is rendered in TAL.

It is not necessary to leave the default value of text strings in the python code. If you don't leave a default value, you must make sure that you set a value for it in the po file. In the case when one message is used in several places, it does make sense to remove any default values from the python code and set a value for the common msgid in the po file.

pt files

At the top of the pt file, in the <html> tag or some other tag that encloses everything else, you need 'i18n:domain="opencore"' like:

<html i18n:domain="opencore" metal:use-macro="here/@@standard_macros/master">

Then for a simple message:

<h2>New users</h2>

you just add a i18n:translate tag with an appropriate message id. The convention here for the message id is to prefix it with the filname of the pt file, so for forgot.pt:

<h2 i18n:translate="forgot_new_users">New users</h2>

for a more complex message:

<a i18n:translate="footer_num_people" tal:attributes="href string:${here/portal_url}/people">OpenPlans has<span i18n:name="num_people" tal:replace="string:${view/nusers}"/> people</a>

note the i18n:name attribute on the span tag which stands as a placeholder for the number of people.

To i18n a string that is an attribute on an html tag:

<input name="forum-update" i18n:attributes="value forum_update_button" value="Update" type="submit"/>

here "value" is the attribute name (default: "Update") and "forum_update_button" is the message id assigned to the string.

zcml and js files

None of these have been i18ned yet and none may need to be.

Rebuilding the Message Catalogs

Every time you change a file as described above, you should update the message catalog files as follows.
First, you'll need i18ndude.

Check if you already have it installed; recent fassembler builds (after 2008-11-21) should install it at opencore/i18ndude/.  If not, installation instructions follow.

Install i18ndude (using fassembler)

If you use fassembler to build opencore, recent versions of fassembler can build i18ndude for you.
If you have fassembler r21452 or earlier, you will have to first update fassembler, like so:

pw@kermit mybuilds $ cd fassembler/
pw@kermit fassembler $ source bin/activate
(fassembler)pw@kermit fassembler $ cd src/fassembler/
(fassembler)pw@kermit fassembler $ svn up
At revision 21488.
(fassembler)pw@kermit fassembler $ python setup.py develop
...
(fassembler)pw@kermit fassembler $ cd ../../..
(fassembler)pw@kermit mybuilds $ deactivate

You only have to do that once, ever.

Once you have a recent enough fassembler, you can just do this:

bin/fassembler i18ndude

Install i18ndude (by hand)

If you don't use fassembler, you'll have to do a little more work:

Important! You cannot install i18ndude into your usual opencore virtualenv, or globally, because it will install newer versions of various zope libraries including zope.tal, and zope will break (see this bug: http://plone.org/products/i18ndude/i18ndudetracker/17)

The solution is to install i18ndude into its own virtualenv:

virtualenv i18ndude
cd i18ndude

source bin/activate easy_install i18ndude

deactivate export PATH=$PATH:$PWD/bin/

cd -

You might want to make that PATH modification permanent for future convenience.

Rebuild everything, the quick way

Once i18ndude is on your PATH, I suggest using the opencore/src/opencore/rebuild_18n script to do everything in one step.
Note that script is new in opencore 0.15; for earlier versions, you can download the script here.

Run it from the base of your opencore source tree.  If it works, you're done.  If you have problems, the below instructions show how to do all the same steps by hand.

Rebuild everything, the long way

  1. Build the .pot file (Extract the newly marked up text)

    Use i18ndude to rebuild the opencore.pot file. It scans all the pt, py, and zcml files throughout the opencore product and extracts all strings marked for translation. So when you're in the src/opencore directory:
    $ i18ndude rebuild-pot --pot opencore/locales/opencore.pot --create opencore .
    
  2. Update the .po files (language catalogs)

    You'll need to sync the opencore.po files with the new opencore.pot. You can do this using i18ndude's "sync" command or by using poEdit. Here's an example using sync:

    $ i18ndude sync --pot opencore/locales/opencore.pot opencore/locales/en/LC_MESSAGES/opencore.po
    opencore/locales/en/LC_MESSAGES/opencore.po: 5 added, 9 removed
    
  3. Update the test .az translation
    See "Integrating with Flunc" below for directions.

Commit your changes

Regardless of how you rebuild the message catalogs, be sure to check in your changes!


How to change a translation

Search for the text string in the language's opencore.po using poEdit. When you're sure you're editing the correct string, see if a value already exists for the string. If no value is set, the default value is used. Create or modify the value, then save the po file.

About .mo files

Zope needs a compiled .mo version of each .po file. These are supposed to be created or updated automatically when Zope starts up. Watch out for WARNING messages from PlacelessTranslationService in the event log or on stdout (when run in the foreground); these typically mean something is wrong with your .po file and needs to be fixed. See the Troubleshooting section below.

Manually compiling the translation files (opencore 0.13 and later)

If there are any problems with the .mo files being auto-generated, you can generate them automatically if you have GNU gettext installed. To compile them, for each subdirectory of opencore/locales, use the msgfmt command, something like this in bash:

cd opencore/locales
for subdir in `find . -name LC_MESSAGES`; do
    echo Updating in $subdir
    msgfmt -o $subdir/opencore.mo $subdir/opencore.po
done

If there are any errors, you must correct them now and run msgfmt again. This can happen if the .po file is malformed due to a bug in either i18ndude or create-az.py. You may be able to hack around the problem by hand-editing the .po file; or it may help to delete opencore/locales/opencore.pot and again use i18ndude to recreate it and re-merge it with the .po files.

A warning about newlines

The msgfmt command is very sensitive about newlines: if a msgstr (translation string) ends with a newline, its msgid must end with a newline; and if the msgstr begins with a newline, the msgid must also begin with a newline. i18ndude is not very good about enforcing this, so msgfmt can complain about the output of i18ndude. I think newlines in msgids look ugly, so we can work around this by manually editing the .po file and adding non-newline whitespace before or after the newline in the msgstr.


How to make it live on the site (opencore 0.12 and earlier)

This is not necessary with opencore 0.13 (plone 3) or later!

SSH to theman and svn up in Products/OpenPlans/i18n directory.

On the ZMI, navigate to Control_Panel/TranslationService/OpenPlans.i18n-opencore-en.po/ and click "Reload this catalog."


Overriding opencore.po

The nycstreets.org file contains sputnik/i18n/sputnik-opencore-en.po which is set up to override values in opencore-en.po. Any values set in sputnik-opencore-en.po will take precedence over values set in opencore-en.po. Blank values in sputnik-opencore-en.po are just ignored. This facilitates managing text that is sputnik specific and text that should apply across all opencore installations.

Integration with Flunc

Flunc is modified to request the "az" translation by default. This can be overridden with the -L parameter. There is a po file at opencore/locales/az/LC_MESSAGES/opencore.po that has all its msgstr values set to the corresponding msgids. The creation and update of this translation can be done by running opencore/locales/create-az.py. You should be sure to use i18ndude to sync the english translation first, because the az translation is created by copying and modifying the en translation.

Using AZ was a pragmatic decision based on the increasingly dubious assumption that nobody who ever uses opencore will have Azerbaijani (az) as their first preferred language in their browser preferences. We should really revisit that decision.

Troubleshooting


  • English text appears in az translation (flunc): Try re-running create-az.py and re-running msgfmt as described above.

  • Default text appears instead of translation: Check the opencore.pot file to see if your message id is there. If not, re-run i18ndude as described above to recreate the .pot file. Then check the .po files.
  • opencore.pot is up to date but I still get default text:
    • Check the opencore.po file to verify that the message id is really there.
    • If it's there, try running msgfmt as described above to make sure the .po file is valid.
    • If it's not there, see hints below.
  • A message id from a page template is not showing up in opencore.pot or opencore.po:
    • Make sure the files are up to date by re-running i18ndude
    • If that doesn't help, make sure you have the i18n:domain="opencore" in the template, as described above.
  • A message id from python code is not showing up in opencore.pot or opencore.po:
    • Make sure you set the domain in the message factory. This is done by default if you import the message factory like: from opencore.i18n import _
    • Make sure you mark the message for translation, like _(u'msg_id', u'Default text here')
    • Make sure the files are up to date by re-running i18ndude.
  • I installed i18ndude and now zope won't start!
    Read the section above about how to safely install i18ndude. To fix your broken zope, remove the zope.* packages installed by i18ndude into $VIRTUALENV/lib/python/site-packages/ 
  • i18n usage in OpenCore isn't complete.
  • We're working on it though!  We've got a Trac ticket "i18n everywhere" at http://trac.openplans.org/openplans/ticket/2729 to track progress on the goal.  Our i18n practices have been spotty in the past (we're a bunch of Americans... ;) and some of the interface remains untranslated, but we understand that complete and responsible i18n practices are a very high priority for many of our users and we're working steadily towards the goal of having a fully internationalized interfac and many available translations.  There is also an ongoing list of i18n tasks to note down the more difficult cases when we see them.