-
Getting Started
last modified April 13 by slinkp
Are you impatient!? You can skip the preliminaries, make sure you have the dependencies, and get right to the quick install.
Contents
- Introduction (what is this, anyway?)
- Reduction (quick-start instructions)
- Preparation (dependencies)
- Instruction (detailed installation instructions)
- Graduation (mildly advanced build configuration options)
- Dysfunction (troubleshooting, support, etc.)
Introduction (wherein we learn the answers to a number of pressing questions)
Where am I?
This page provides instructions for people who are interested in installing their own deployment of the OpenPlans software. OpenPlans is the software which powers openplans.org (<-- warning: recursive link), the very site which you are now visiting.
OpenPlans is free, open source software. It consists of a considerable amount of custom code, built on top of a number of other popular open source technologies. Notable underlying components include Python, Plone, Pylons, Paste, WordPress, and MySQL.
Who is this for?
OpenPlans software, once it's installed, is for anyone who knows how to use a computer. But this page is about installing the OpenPlans software, which requires a bit more savvy. You should at least have some basic familiarity with Linux and/or Unix systems administration. OpenPlans is heavily Python-based, so ideally you'll be somewhat familiar with using and installing Python packages using easy_install.
What is required?
Currently only Unix-like (e.g. Linux, FreeBSD, Mac OS X) environments are supported by the OpenPlans software. While in theory there is nothing that would prevent the software from being made to run in a Windows environment, since the current developers only use Unix environments for development and deployment, all of the existing build tools are Unix-centric. If you are interested in getting OpenPlans to work in a Windows environment, let us know on the dev list.
Additionally, OpenPlans depends on the availability of a number of other tools, including Python (2.4.3 - 2.4.5), Subversion, Apache, PHP, and MySQL. These must be installed and available to the OpenPlans installation tools or the installation will not succeed.
What does the installation process entail?
Assuming all of the prerequisite dependencies are installed and correctly configured, the process of getting the OpenPlans software installed is pretty straightforward. It consists of the following steps:
- Install the OpenplansBoot package.
- Run the 'new_openplans_site' command to bootstrap a directory structure for managing an OpenPlans deployment.
- Run the 'newbuild.sh' script generated by the new_openplans_site command to launch a build of the OpenPlans software.
That's it! Once the build completes, you should have a fully working OpenPlans site, ready to start up and start accepting requests.
Is it really that easy?
Well, yes and no. It may take some twiddling to get the prerequisite dependencies set up correctly, if they're not already installed and in use on your target machine. And the build in step 3 is lengthy; there are a number of things that can go wrong during that process, which may require some troubleshooting. Macs in particular can be a bit finicky. Generally, however, it's not too hard to get things going, and after you've got it working the first time, subsequent builds (whether to create a new site or to upgrade an existing) work with very little trouble.
Great! How do I get started?
Glad you asked! Just keep reading... the rest of this page details the exact steps you'll want to take, outlining some pitfalls along the way. Good luck, and we hope you enjoy playing with OpenPlans!
Reduction (wherein we list the installation steps w/o all of the extraneous mess)
With that all said, here's the bare-minimum set of installation instructions.
$ easy_install-2.4 https://svn.openplans.org/svn/OpenplansBoot/trunk $ new_openplans_site site.domain.com base_port # (<-- enter config svn repo URL) $ cd site.domain.com $ ./newbuild.sh openplans/trunk # (<-- enter admin pw, mysql root pw, possibly 'p' to accept ssl certificate) $ cd builds/YYYYMMDD $ ./bin/supervisord
Or, if you want a Plone 2.5-based installation:
$ easy_install-2.4 https://svn.openplans.org/svn/OpenplansBoot/trunk $ new_openplans_site site.domain.com base_port # (<-- enter config svn repo URL) $ cd site.domain.com $ ./newbuild.sh openplans/branches/plone25 # (<-- enter admin pw, mysql root pw, maybe 'p' to accept ssl certificate) $ cd builds/YYYYMMDD $ ./bin/supervisord
Then point your browser at http://localhost:base_port/. Not so bad, eh?
Note that by default, the server on [base_port] is configured to only listen to requests from localhost; the assumption is that you will run a local proxy (e.g. Apache mod_proxy, with any number of caching layers in between) to expose the application. If you want to connect directly, or if you are running a remote proxy, edit the {path_to_builds/YYYYMMDD}/etc/deliverance/deliverance.ini file and change the second instance of localhost to 0.0.0.0 to tell it to accept all connections.
For more detailed instructions, explanations and troubleshooting, read on...
Preparation (wherein we learn the prerequisite dependencies)
What you will need
We'll start with a list of all of the pieces that need to be in place before you can begin:
-
Python 2.4: You need to have Python 2.4 available on your PATH (or specified using the extra_path variable). It must be 2.4.3 or greater. Sorry, Python 2.5 is not yet supported. If you build Python from source, you must first be sure to have the dev packages for zlib and openssl installed (ubuntu: libssl-dev and zlib1g-dev).
- Virtualenv
- Easy Install: If your Python 2.4 instance does not already support the easy_install command, you'll want to make sure it's available.
- MySQL: You need to have a running MySQL server and the administrative root password for that server. We develop on mysql 5..0. Earlier versions are not supported. On Ubuntu you will need at least mysql-server, mysql-client, and libmysqlclient*-dev (whatever the latest version is).
- Apache w/ PHP: You must have a version of the Apache web server, containing a functional PHP installation. We normally use apache 2 and php 5.2 with the "hash" module installed, and support for MySQL. (On Ubuntu, you should install php5-cli and php5-mysql). Also, both the Apache httpd and PHP php-cgi binaries must be available on your PATH (or specified using the extra_path variable). Don't worry, OpenPlans will not interfere with an existing Apache server; instead it generates a custom httpd.conf and uses the existing Apache binary to launch an instance listening on a different port for its own needs.
- Subversion: Most of the OpenPlans software is retrieved from the internet using Subversion, so 'svn' (with SSL support) must be available on your PATH (or specified using the extra_path variable).
- A config svn repository: In addition to retrieving software, OpenPlans uses Subversion to manage the configuration files associated with its various services. You will need to specify the URL of a working Subversion repository that OpenPlans can use for this purpose. WARNING: This configuration will contain sensitive system information, including passwords to some of the OpenPlans services. The svn repository you use should NOT be available to untrusted users. If you have a full subversion installed on your deployment machine, it is very simple to use "svnadmin create /path/to/config/repo" to create a repository and then "file:///path/to/config/repo" as your repository URL.
- Miscellaneous other unix tools: Our build scripts assume you have wget, bash, tar, patch, make. You probably have most or all of these, but notably, Macs do not have wget installed by default. If you discover any other command that the build requires which isn't installed on your system, let us know on the dev list so we can update this page.
- libxml2 (libxml2-dev on Ubuntu)
- libxslt (libxslt-dev or libxslt1-dev on Ubuntu)
- libjpeg (On Gentoo systems, emerge media-libs/jpeg; on Ubuntu, libjpeg62-dev or something similar)
- sqlite (libsqlite3-dev on Ubuntu)
Instruction (wherein we cover the installation process in no small detail)
If you want less detail, skip ahead to the abbreviated instructions .
Now that you've got the prerequisite pieces in place and functional, you're ready to get to the actual process of installing OpenPlans, via the following steps:
- Install the OpenplansBoot package: This bootstraps the installation tools. Run this command:
$ easy_install https://svn.openplans.org/svn/OpenplansBoot/trunk
We recommend you use the virtualenv package to install OpenplansBoot into a virtual Python environment, rather than into the system Python environment. Regardless of whether you do this, OpenPlans will make heavy use of virtualenv for managing its own services.
As noted above, Python 2.5 or later is not supported at this time. Many systems use 2.5 as the default Python these days, so you may need to explicitly use easy_install-2.4 to ensure that your build uses the right Python version. - Run the 'new_openplans_site' command: Installing OpenplansBoot should put a 'new_openplans_site' command on your executable path. It should be invoked as such:
$ new_openplans_site site.domain.com base_port
Where site.domain.com is the hostname at which you plan on making your OpenPlans site available (you can choose any descriptive name if you're just trying to test it out locally) and base_port is the numeric port at which the site will be made available. WARNING: OpenPlans actually consists of many interacting HTTP services, all exposed as one integrated site available at the base_port. Each of these services needs its own port, so you should choose a base_port that is followed by a space of available ports. At the time of this writing, OpenPlans uses 8 ports, from base_port through base_port+7, although you may want to leave even more available space to support additional services that OpenPlans may add in the future.
When you run the new_openplans_site command, you'll be prompted for the URL of your configuration svn repository. Once this is provided, the command should return almost immediately, having created a directory with the name of your specified hostname. This directory will contain a sparse OpenPlans deployment skeleton directory structure, as well as the 'newbuild.sh' script that you will use in the next step. - Run the 'newbuild.sh' script: Okay, now we get to the actual meat of the process. The newbuild.sh script launches a lengthy build process, which will, after a bit of time, generate a fully working OpenPlans installation, ready to be launched and to start accepting requests. You can go get a snack, but you shouldn't wander too far because you will be prompted for a bit more information during the process. Here's how you invoke the command:
$ cd site.domain.com $ ./newbuild.sh requirements_directory
In the above command, requirements_directory is the name of a subversion directory containing files that specify all of the underlying dependencies of the OpenPlans software. OpenPlans has a number of different build configurations, this parameter allows us to easily choose between them. The simplest choice for getting started is the following:$ ./newbuild.sh openplans/trunk
If you run newbuild.sh with no arguments, you will see a list of the available requirements directories.
NOTE: If you want to try out the older Plone 2.5-based OpenPlans stack, you can use the following invocation.$ ./newbuild.sh openplans/branches/plone25
After executing newbuild.sh, there will be a short delay as initial requirements are downloaded. Eventually you will be prompted for an admin password that will protect your OpenPlans installation. You can enter a password of your choosing (twice), or you can simply hit 'Enter' to autogenerate a random password. In either case, the password will be stored in an admin.txt file that you can refer to later.
The next thing you will be prompted for is the MySQL root password. This is needed because OpenPlans will be creating a number of MySQL databases to support some of its services.
There's one more issue that may arise: some of the software is downloaded from the Plone subversion repository, which is protected by an unsigned SSL certificate. If you have never connected to this repository, then you will be prompted about the certificate, and given a chance to accept or reject it. If this happens, we recommend you hit 'p' to 'p'ermanently accept the certificate . Warning: due to a bug, you might not see this prompt. If the build seems stuck, just hit 'p' and see if that gets things moving again :-)
Or you can avoid this prompt altogether by running the following command before running newbuild.sh:$ svn ls https://svn.plone.org/svn/plone
This will also prompt you to accept the certificate, if you haven't already done so; if you choose 'p' at this point, then it will not come up again while running the build.
That's it! Assuming that all of the prerequisites are configured and available to the build process as needed, then your build will complete and you will have a fully functional OpenPlans installation. Your build will live within the 'builds' directory, in a subdirectory named after the current date. Let's fire it up! - Start the supervisor daemon: OpenPlans uses supervisor to manage the various services of which it is comprised. To start OpenPlans, then, you use the following commands:
$ cd builds/YYYYMMDD $ ./bin/supervisord
This starts the supervisor daemon. You can examine the processes that are being managed using:$ ./bin/supervisorctl status
You would also use 'supervisorctl' to stop, start, and restart the services, either individually (by service name, e.g. 'supervisorctl restart opencore') or as a whole (using 'supervisorctl restart all'). If you want to shut down the entire rig, including the supervisor daemon itself, use:$ ./bin/supervisorctl shutdown
- Use the OpenPlans site: Assuming you haven't stopped or shutdown the services, you should now be able to point a web browser at http://localhost:base_port and immediately start using the OpenPlans site. C'est finis. :-)
Graduation (wherein we learn about configurability and tweakage)
Chances are good that, if you're going to manage a few different OpenPlans installations, you're going to use a single configuration repository for all of your configuration sets. It's a bit annoying to have to interactively enter the svn repository URL each time you run the new_openplans_site command. Luckily (you can probably see where this is going...) you don't have to. OpenplansBoot provides a bit of configurability to let you specify some specific build parameters ahead of time, and to save them so that different installations will share certain attributes. This section gives a brief overview of how you can twiddle some knobs.
Creating the newsite.ini file
When you easy_install the OpenplansBoot package, Python installs an "egg" into your Python environment. This egg is really just a directory, containing the package's Python code, as well as certain metadata associated with the package. It's beyond the scope of this document to go into too much detail, but this egg directory can usually be found in your Python environment's lib/python/site-packages directory. It will have a name similar to "OpenplansBoot-0.1dev_r16550-py2.4.egg" (the version number and revision number may vary, obviously). If we call this directory the egg_root, then you should be able to find a sample configuration file at egg_root/openplansboot/newsite.ini.example. This file will contain some sample configuration settings for new builds. The example file has no effect, however; in order to actually impact your builds, you need to 'cp newsite.ini.example newsite.ini' and then tweak the settings in newsite.ini to your liking. If the newsite.ini file exists in this location, it will be used by the new_openplans_site command when bootstrapping the deployments.
What settings can I change?
You should refer to the newsite.ini.example file for a complete description of what settings are available to you and what effects they have. I'll describe a couple of the more useful ones here to get you started:
- base_dir: At TOPP, we keep all of our OpenPlans deployments in a single location on our servers. If the base_dir variable is set, then new_openplans_site will create the deployment skeleton in the specified location, instead of in the current working directory.
- req_user: Similarly, TOPP uses a single user account to manage a single OpenPlans installation. If req_user is set, then both new_openplans_site and the generated newbuild.sh file will check the current user id and will exit with an error if it does not match the specified req_user.
- db_prefix: Wordpress and some other parts of the system may need to have MySQL databases initialized. The db_prefix will be used as the initial part of the database name. You can override this if you care what your databases are named.
- extra_path: Sometimes certain tools that the OpenPlans build needs access to live in places that are not on a standard user's execution path. Any specified extra_path value will be appended to the current user's execution path for use during the build process. Multiple extra paths should be separated by a colon (":") as is standard for the PATH shell environment variable.
- force_ssl: OpenPlans has support for forcing all logins and password changes to be forced through a secure SSL connection rather than the standard unencrypted HTTP. However, getting this to work depends on having a correctly configured SSL server in front of the OpenPlans installation; OpenPlans itself doesn't handle SSL handshaking. If you know that your deployment will have an SSL front-end and you want to activate this feature, set force_ssl to 'True'. NOTE: It is also possible to change this as a per-build setting later, so don't worry if you want to get started without it, you won't be locked in.
- etc_svn_repo: This is the variable that specifies the configuration repository URL. If this is set, you will not be prompted to enter the URL when you run new_openplans_site.
- etc_svn_prefix: If many developers are sharing a single configuration repository, it's possible that there will be namespace collisions between their installations. If two people use 'test' as the name of their deployment, then the OpenPlans build will get confused and will try to use the same configuration checkout for the two distinct installations. When you set the etc_svn_prefix variable, the OpenPlans build will add the specified string prefix to all of your configuration checkins, greatly reducing the chance of conflicting with someone else.
It should also be noted that ALL of these settings, and any others that are supported by newsite.ini, can also be entered on the command line. For instance, if you want to place one single OpenPlans deployment in a different location than the rest, and to force it to use SSL when usually you do not, then you could use the following invocation:
$ new_openplans_site site.domain.com base_port base_dir=/path/to/special/place force_ssl=True
Any setting that is specified on the command line will override the same setting specified in the newsite.ini file.
What about the MySQL root password?
The only remaining piece of information that you have to enter interactively for a new deployment, but which is unlikely to change very often, is the MySQL root password. Because this is a particularly sensitive piece of information, we did not add support for it in the newsite.ini file, nor do we allow it to be entered on the command line. If you don't want to enter this every time, then you will want to create a file named .mysql-root-pw in the home directory of the user that is running the newbuild.sh command. This file MUST be chmod 600, i.e. not readable by other users on the system. If the file exists but is not chmod 600, then the OpenPlans build will exit with an error.
Disabling Mail Confirmation
One thing you may wish to tweak is whether or not new users are required to confirm via email when joining the site. This is turned on by default, but can be annoying when just testing a new build for development purposes. Edit the file etc/opencore/zope_etc/zope.conf, look for a line that says "email-confirmation True", and change that to read "email-confirmation False". Save that file. Then you can restart opencore, by running eg. "bin/supervisorctl restart opencore".
Dysfunction (wherein we learn what to do if things go horribly wrong)
Despite all of our careful preparation and instruction, it is, alas, an imperfect world, and you may find that things do not work as smoothly as intended. What should you do to recover, and/or to get assistance if you don't understand what's going wrong?
D'oh
We'll start with the easier case. What if you're running a build using newbuild.sh, and it fails for an obvious reason, like (for instance) that you forgot to start the MySQL server before running the build. You start the server up, and aren't sure where to go next. You could re-run the newbuild.sh script... but that would start a new build in builds/YYYYMMDD-1, instead of completing the earlier build. If you want to try to pick up the original build where it left off, you can run the following commands:
$ cd builds/YYYYMMDD$ source fassembler/bin/activate
$ fassembler missing
This should restart the build where it left off. When it's complete, use the following commands to reset your Python environment and start up OpenPlans:
$ deactivate$ ./bin/supervisord
Once you've gotten the whole thing installed, you might want to skim
getting started with the code
for some useful tips about coding in & around OpenCore.
WTF?!
Unfortunately, some problems may not have an immediately obvious solution. In fact, you may not have any idea at all what's going on. In that case, OpenPlans developers may be able to help you past your hurdles. There are two primary mechanisms for getting support from the OpenPlans developers. The first is via the opencore-dev mailing list. This is a very active mailing list that is used by all of the primary OpenPlans developers. It is strongly recommended that anyone who is using the OpenPlans software be subscribed to this list in order to keep up with what's going on in OpenPlans development. It is also a fine place to ask questions. Please make sure you include lots of information about the specific problem you're having, including your underlying platform and any error messages that you might be seeing. You may find some help with how to formulate a helpful question here.
You can also find the OpenPlans developers on IRC, in the #openplans channel of irc.freenode.net. Again, this is a very active channel that is used almost daily by the OpenPlans developers. Most of us are in the U.S. and are on-line during the workday, so please plan accordingly if you're in another part of the world and are trying to reach us.