OpenCore Software

I spent the last two days at the Cisco facility in San Jose, attending the third NetSquared conference. NetSquared is a conference that focuses on the intersection of activism and technology. It brings together people from NGOs, foundations, and technology providers to foster conversation, collaboration, fund-raising, and just to generally get folks connected and excited about the interesting ways that the net is being used to enable social change.

There were a number of talks, panels, and conversations, but the central focus of the event was a contest between 21 featured projects. Early on the first day, each of these projects was given 2 minutes to describe themselves; this was immediately followed by an hour-long session where attendees could walk around and chat w/ project members to learn more. Near the end of the last day, they were all given another 2 minutes to make their closing pitch, after which attendees were able to vote for which projects they felt were most deserving. Each person was given 3 votes, which could be distributed as desired (i.e. all for one project or spread among 2 or 3). the top three vote winners received $25000, $15000, and $10000, respectively; the rest split equally the remaining available grant money, which rumor had it came to about $2000 per.

It was a lot fun and inspiring to see what good work people are doing in the world. The coolest thing I saw was by Ushahidi, the winner of the $25000 prize. They’ve been mapping outbreaks of violence and peace as a reporting and research tool during Kenya’s post-election tumult. They’ve got a great timeline map, with which you can get a great visual representation of the way events developed through time. The timeline component is something out of MIT’s SIMILE project, and thus is free for us to use. This could be very useful for TOPP; it’d be a great way to present the summer’s worth of block parties, for instance. And it’d be fun to add geocoded event support to OpenPlans; users who are producing events (sprints, protests, fundraisers, whatever) would be able to enter the time and location, and their event would show up on a regional timeline map. Other folks in that region would get a great visual overview of what was going on around them.

Beyond just generally learning what was out there, the time spent talking to people and listening to talks left me with a few specific takeaways:

1. Mapping is the new black
   
   Nearly everybody is doing mapping related work. Maybe this was a bit slanted, because “Mash-ups” was one of the themes of the conference and, frankly, I can’t think of a compelling mash-up I’ve seen that wasn’t mapping-related. But I think it’s more than that; mapping is very useful, and people are putting mapping tools to good use in all sorts of ways.
2. TOPP’s Geo group is in a great position
   
   While there are a lot of people doing mapping work out there, TOPP has been doing geo stuff for longer than just about anybody I’ve seen in the open source-y non-profit world. And the depth of the work TOPP has done is impressive; we haven’t just used existing tools in clever ways, we’ve been one of the driving forces behind the very existence of a high-powered open source end-to-end GIS system. I spoke to a couple of folks who had been in consideration for larger geo-related contracts w/ gov’t and NGO organizations where the had really wanted to use the open source offering, but had balked in the end from fear of becoming dependent on a one- or two-person consultant team. With TOPP’s depth of experience and resources (human and otherwise), we’ll be able to make such organizations feel much more comfortable. This has been the plan all along, but it’s really exciting to talk to people who are out there in the middle of it confirming that the opportunities really are there, and that we’re in a great place to be able to take advantage of them.
3. TOPP itself is in a great position
   
   Most of the people who are pouring their lives into using technology for good causes are constantly hustling for funding. Either they’re freelancing and are trying to land their next contract (even as they’re working on their current one), or they’re a non-profit that’s constantly pounding the pavement for grants and other funding. TOPP is very unusual, and very lucky, in that we’re given the opportunity to develop our offerings without a lot of pressure to monetize them; only after we’ve established a reputation and developed demonstrable strengths do we start to focus those strengths towards generating a revenue stream. This seems to be working fabulously for the Geo group (see above); I see no reason why it won’t work similarly for OpenPlans, and other projects that we develop, as long as we continue to produce working, usable software.

I managed to talk to a lot of folks throughout the event, folks were generally very interested in what we’re doing. I had a number of occasions where someone would approach me and introduce themselves, saying “Hi! So-and-so said I should come over and talk to you, that I’d be very interested in what you’re doing. What are you doing?” While it was hard to keep describing our organization and our efforts over and over again without sounding lame and canned, that’s a pretty good problem to have. A couple of the connections I made may end up bearing some fruit down the road. In particular, I’d guess that our Geo folks would be interested in talking to the folks who find out about but do not end up getting those bigger contracts I was mentioning above, so we can decide whether we might be interested in trying to land those jobs. Also, there were a couple of nice mapping front end tools that would probably add value to our greater GIS offerings, if we made sure those tools work with our stack.

All in all I think it was valuable for me to be there. And I think that we should have a slightly bigger presence next year, including giving a presentation of our own. I’m sure they’d be happy to have us.

We’re at PyCon! The lion’s share of the openplans / opencore developers are here:

Ian Bicking, Josh Bronson,  Mel Chua, Jeff Hammel, Anil Makhijani, Robert Marianski, Doug Mayle, Whit Morris, and me (Paul Winkler). Look for us and say hi!  We are friendly and mostly harmless. Oh, and we might have a job or two open.

I’m giving a talk Friday at 3:55, and Ian’s doing one on Sunday at 11 AM

A lot of times at the Snow Sprint I was struck by a feeling that I was in the middle of a group of people who were really doing open source right. So on the plane home I started thinking about what that meant and why I haven’t felt that way at TOPP.

Well, a lot of us have said at various times that our attempts at doing open source development have often been frustrating and rarely seem to be worth the effort. Coming out of the sprint, I think that’s because we are missing a very big piece of the “True Spirit of Open Source”. Doing open source properly is not just a matter of putting the GPL on our code—it is not about making a legal contract but about making a social one. It is about committing to reuse and collaboration both internally and externally and it is well worth the effort.

Writing, finding, and using software all take time and care and with someone paying for our time and care we ought to think of these acts as investments. We should make the most of those investments by seeking code reuse and encouraging collaboration. The more we encourage heavy use of software and heavy reuse of code, the better our ability to improve and solidify it. Whether or not we’ve read Dreaming in Code (though everyone here should—as soon as possible—seriously) we all know that software is hard to do well. It’s full of unexpected behavior, prone to breaking when you touch it, and rarely optimized. We can improve it only by testing and exercising it: give it to people to use, poke it in new ways, put it under stress, see what breaks, fix it and don’t let it break in the same way again.

Laziness is a virtue

Because software is expensive to write, it is most responsible of us to write as little as we can. And because software is hard to write, it’s rarely done once it’s written; most software has tremendous room, and often necessity, for improvement. So our goal should be to maximize the impact of every improvement we make. This primarily means seeking to increase the breadth of our code’s effect: convincing other people to use our software, sharing code across multiple products, and reusing products in as many ways as we can across everything we develop.

To give a specific, if vacuous, example, we have five or more different pieces of code deployed on OpenPlans that generate and send email to users: Listen sends mail, Twirlip sends mail, WordPress sends mail, and Opencore sends mail in at least two different ways. Suppose, instead, that we had a single shared way to send mail to users. Aside from the immediate benefit of less overall knowledge to store, the different mail-sending needs of all these applications would force us to develop a stable and reliable method for sending mail, and when heavy usage of one of those dependent applications inevitably reveals an inefficiency or bug in the mail-sending code, multiple otherwise unconnected parts of our site would instantly benefit from our resulting improvements. Put that same mail-sending code on two or more websites and encourage other developers to deploy it in their applications, and we can tremendously increase both the number and variety of stress points and the magnitude of impact of each strengthening, both the opportunity and the payoff for improvements.

So maximizing adoption of every piece of software by both users and developers is a forward-thinking and cost-effective stance. It gives programmers the most opportunities to strengthen their (buggy, brittle) code and ensures that every (hard, pricey) act of improvement has a large and wide-reaching effect.

We’re not alone

It is not only creating software that needs be treated as an investment. In a true open source environment using others’ software, too, is an investment that we should make the most of. When we choose to depend upon existing free technology—in any way: WordPress, Xinha, Zope, Pylons, Trac—we will benefit ourselves and others by doing all that we can to improve it, participate in its communities of users and developers, and encourage the growth of those communities. The core reasons are the same as with our own code: the more eyes and hands that are on a piece of software, the better it can become.

There is, of course, an additional benefit to encouraging outside developers, namely lessening our internal development and maintenance burden by gaining volunteer contributors. When we create our own software we thus have the additional responsibility of building these communities from nothing. But when we are dealing with existing open source software this goal is all the easier to achieve. A community of developers already exists; our only responsibility is to gain admittance. We should treat our lack of commit rights not as an obstacle, but as a challenge.

Alongside the fuzzy feelings that come with earning the respect of an established group, there is, in keeping with the best of the open source mentality, great potential for everybody to win when we are welcomed into an existing community. We stand to reap the benefits of the efforts of the intelligent, experienced and caring people who develop the software that we have decided is worthy of our use, and we gain the potential to influence the path and priorities of their development with our input and interest. They, meanwhile, gain the feedback and work of a new set of smart (and funded) contributors with a real interest in the success of their project.

Finding those unicorns

I’ve framed this primarily as an economic argument to be concrete, but of course there’s more to it than that. The same stance is also about committing to respectful and positive relationships with outside agents; putting our faith in open and collaborative development environments and in the potential for groups of smart, committed people to produce great things even with differing agendas; taking the time and care to make our work as good as it can be and letting the rest of the world be the judge of it. It’s about being responsible consumers who contribute back to the products we consume, and being responsible producers who proudly stand by what we produce. And then there’s our aspirations at TOPP to do good and produce real positive change: all the time and money we can save by reusing and building on existing solutions can be diverted toward new projects to do even more good in the world, with cascading effects.

So let’s build and get involved with communities, submit and fix bugs in others’ code, give thanks and contribute ideas to the people whose software we use.

Let’s release Flunc, Externalator and Deliverance to the world for real, with welcoming websites and helpful documentation, and encourage people to use them and talk to us about them on mailing lists and IRC channels. Along with everything else it’s a good way to put our software to the test: to put a different spin on the dogfood concept, if no one else wants to use our software, why should we? Let’s keep up with Xinha development and submit behavioral patches to its authors instead of keeping our work isolated on our “vendor branch.” Let’s tell people about the recommendations of our smart interaction designers and submit our implementations of that advice; how many open source software communities have an excess of input from web software designers? Let’s release WordPress and Trac plugins to their wider communities and let’s solicit contributions and feedback to Listen and Opencore.

In short, let’s try to practice better open source. After all, we can only bring the open source ethos and philosophy to the broader world after we get it right in our software.

Okay, so maybe that’s a bit of a bold title, considering a) most of you have been using fassembler for a little while already and b) Ian wrote fassembler, not me.

However, it does feel like just today we’ve put the last little bit of polish on it, so that it’s officially ready for prime time for all of our deployment needs. There are still some parts of the process that can be improved, but it’s definitely workable, and it’s orders of magnitude easier to use than anything we’ve had up until now.

How, exactly, should it be used? Well, if you just want to use it to build a local development rig of the OpenPlans software, you can refer to the fassembler how-to. If you want to actually deploy a persistent site, one that lives on either our dev or our live server, you can refer to the OpenPlans deployment instructions.  The deployment instructions go into quite some detail about how our directories are laid out, and how you should manage the configuration and requirements for separate sites, and separate builds of the same site. Please do not try to do any deployment of any site on either flow or theman without reading this first.

Good luck and happy building!

What’s our definition of success, our institutional purpose? Personally I have a very clear sign of success in my own mind: how strongly do I recommend our service to other people? I’m a a bit shy about suggesting people use “my” stuff, where “my” means something for which I feel responsible for both in its success and failure (which of course includes openplans.org). If I personally felt confident recommending openplans.org to anyone, especially people who I don’t want to give personal technical support, then I would feel part of a successful project. I’d love it if then a lot of people everywhere used it, but my personal feeling of satisfaction is not contingent on that. Or maybe that’s what my notion of success+1 would become.

What about the development analog: would I recommend our software to other developers (not hosted by us)? Frankly I don’t care much about this. Making our software generally usable I think is an important discipline for us, but to me it’s not an end. Why not? I’m not really sure.

In his blog post, cabraham takes a stab at how we define success by saying:

Our primary mission is also being achieved by NYCStreets.org and OpenPlans.org, which are tools that empower people to organize social action projects. The way to measure success here would be to track the overall traffic of the sites as well as the quality of the projects being organized on the site and the degree to which they facilitate real-world change.

My concern here is similar to my previously stated concerns that we are pulling each other in different and incompatible directions. TOPP is a large organization with a variety of projects under its wing, making it difficult to steer projects such that they complement each other rather than conflict. Using the same metrics for success for both NYCstreets.org and OpenPlans.org is an example of this.

A brief history…

While there has never been a clear definition as to what OpenPlans is or who it targets, one shared vision of OpenPlans was as a website geared at empowering citizens to organize and effect change, specifically in regards to issues of urban and community planning. This vision dovetailed nicely with the goals of our advocacy branch—as Streetsblog and Streetfilms developed the propaganda, OpenPlans.org would provide the tools. Yet OpenPlans.org was conceived of much too broadly to ever get very far in developing a community to complement those on Streetsblog and Streetfilms. As a result, NYCstreets.org was launched to combine the tools provided by OpenPlans with the focus of purpose that drive Streetsblog and Streetsfilms.

OpenPlans versus Openplans.org

As a consequence of the launch of NYCstreets.org, however, OpenPlans.org was stripped of a large part of its stated audience: no longer can OpenPlans.org build a community around urban planning issues without directly competing against its sibling, NYCstreets.org. Direct competition between the two sites hurts both and benefits neither.

There has always been confusion around the distinction between OpenPlans, the software, and OpenPlans.org, the website, and many have viewed them as two sides of the same coin. Although narrowly conceived, there was a never a problem with this view and we muddled along just fine. But now that we are running a service—NYCstreets.org—using the OpenPlans software stack we have an opportunity to reposition ourselves along the lines of the GeoServer model: we must reconsider OpenPlans as a product, capable of being easily installed and customized, rather than a tailor-made package designed solely to run our OpenPlans.org service and no more. The definition of success cabraham proposes is reasonable for measuring the success of OpenPlans.org, but to continue thinking of OpenPlans as having a one-to-one relationship with OpenPlans.org is short-sighted and will likely result in failure.

­The Success of OpenPlans ­

The possibilities of success are much richer for OpenPlans if conceived of as a product. We are no longer limited to the constraints of a single website with a single goal but instead have the opportunity to build software that can address many problems—ranging from facilitating Plone sprints as OpenPlans.org currently does, to organizing communities in the form of NYCstreets.org, or even facilitating the project management needs of small local governments. OpenPlans as a product can’t be over-designed and over-engineered to meet overly specific use cases, but should instead be built to be flexible, easy to use, and broadly useful.

Despite having an opportunity to broaden our scope, we must be careful to focus our efforts on improving the usability of our product. NYCstreets.org provides us the opportunity to explore real-world users and situations around which to build our software, and the existing OpenPlans.org website will continue to provide an entirely different set of examples. My suspicion is that we will learn that our model of segregating functionality by type (separating pages from tasks, for example) is not as useful as the BackPack model of allowing multiple content types on each page—after all, users don’t care about what type of data something is as much as the context to which it belongs.

It is my hope that OpenPlans will mature to the point that it can consider sustainability models like those proposed by Cholmes for GeoServer, perhaps even becoming as ubiquitous for project management as WordPress is for blogging.

I’ve been thinking a bit about Nick’s question, “how do we define success here at TOPP?” I’d been thinking about this for some time since coming up with TOPP News.

It seems to me that organizations all have some mission, or purpose, “why they exist.” They also define themselves by a set of shared “values”, things that the organization believes to be worth holding on to. Defining success at TOPP then would be defining how well we are achieving our mission and how well we are holding on to our values. The tough part is usually how to measure these.

So first there’s our mission: to build technology to enhance the role of the citizen in democratic society.

This is being done through StreetFilms and Streetsblog, by bringing people’s awareness to the quality of life on the streets of NYC and by educating people in how to improve street life. Success here would be a measure of how well we are achieving this by measuring, say, the overall traffic of the sites and the quality and quantity of people’s comments.

Our primary mission is also being achieved by NYCStreets.org and OpenPlans.org, which are tools that empower people to organize social action projects. The way to measure success here would be to track the overall traffic of the sites as well as the quality of the projects being organized on the site and the degree to which they facilitate real-world change.

And then GeoServer which is an open, interoperable infrastructure of geographic information. I’m not as familiar with the current uses of GeoServer but I would imagine that the success of GeoServer could be measured by looking at the quantity and quality of real-world projects that use GeoServer to accurately inform their real-world decisions.

Part of accomplishing our mission is to be continually looking for and trying out new ideas and projects that use technology to enhance the role of the citizen (eg. Melkjug).

Next, we need to look at our values. These are things such as:

• innovation - contributing good software and ideas to the world
• positive participation - in the opensource software community
• providing a supportive and nourishing work environment for employees
• neighborliness - being good West Village neighbors
• capability development of our employees
• “green” – operating our office in a way that conserves and recycles
• accountability – taking responsibly for our actions and our work

Having defined both our mission and our values, I would therefore conclude by defining success at TOPP as: “achieving our mission while also acting in alignment with our values.”

So, I’ve been here at TOPP for a bit more than two months now and my contributions as far as fuzzy, non-software-related stuff like blogs have been pretty close to nil. I’m not sure that’s a particularly bad thing, but nonetheless a blog post or two seems unlikely to go awry, so here goes.

The conversation that prompted me to even consider posting here occurred this past Monday at the impromptu Geoserver front- and back-end teams standing meeting that we held after Nick informed us that the OpenCore folks could have a better meeting without us. Seb pointed out that what I’ve been working on for the past few weeks is pretty opaque to him and that it might be worthwhile to blog a bit about it. Given the success of Seb’s blog posts, I imagine he’s right about that, but I’m not going to blog about it now. If you’re really interested, there’s some info about it on the Geoserver wiki. (http://docs.codehaus.org/display/GEOSDOC/RESTful+Configuration+API and http://docs.codehaus.org/display/GEOSDOC/RESTful+User+Management+API)

Since that conversation with Seb, I’ve been thinking a bit about what exactly my role at TOPP is, in part due to conversations with Lily, who has been asking me, and presumably the rest of us code monkeys, what it is that we do and how we refer to ourselves, etc. When she asked me I was pretty firm about being a software developer as opposed to a programmer or software engineer, even though officially my title is Software Engineer. I think what I had in mind was kind of a strict taxonomy of byte wranglers (note that I’m using Capital Letters to indicate these are my own definitions and not anything particularly formal):

* A Programmer sits around, takes some very literal, low-level (and, by my subconscious association, uninteresting) specifications and translates them from English to computer-ese. This is what I think of when I use the term ‘code monkey’ in a somewhat serious context, someone that basically sees the process of creating software as a simple conversion rather than a creative process.

* A Software Engineer takes higher-level requirements in the form of detailed specification documents (or, in the absence of such documentation, creates it) and converts them to the kind of specifications that a Programmer can use to make an actual piece of software. Software Engineers treat the process of creating software as a scientific phenomenon with a sort of physics of its own, measuring things like Effective Lines of Code and Defects Introduced Per Defect Addressed and trying to infer useful information from them.

* A Software Developer takes higher-level, but not necessarily very specific requirements and makes them into software. For the Software Developer, the process of creating software is an incremental one that involves constantly referring to the people that are going to be using the software, or at least a working knowledge of the problems that the software is trying to solve. Software Developers treat the process of creating software as a solution to a problem, and define success therein by measuring how well the problem is solved rather than software metrics.

While writing these descriptions out I noticed that I’ve defined Programmers and Software Engineers by what they do and Software Developers by how they think about things. It doesn’t really seem like the three are contradictory, and I know that what I’ve been doing over the past 2 1/2 months has fit each of these roles (and probably each combination of them) at different times.

So I suppose I’ve not really said much, but then I always feel that way after talking about me. At any rate, feel free to refer to me as a Software Engineer or a Software Developer or a Grand Vice Poohbah of Java-Based Endeavors; it seems like the definitions are equally nebulous for all of them.

PS: As I was writing this it occurred to me to consult the oracle wikipedia, and I found that the page for Software Developer redirects to the one for Software Engineer. Shows what I know, I guess.

Nick brought up some very good questions in his “Year in Review” post—none of which I intend to answer here.

Instead, I want to address a larger question he implicitly poses: how do we define success? At the moment we simply ignore the question altogether and are content to ride the wave of our day-to-day inertia, but it is a question that is crucial to our existence as an organization. No doubt we each have our own conception of what would make us successful: for some it is the quality of our software at the code level, for others the value it brings to our users, and for others still a sense of how ‘cool’ it makes them feel. Yet, without a shared vision of how we define success and the path to get there we will continue to be pull each other in different, often incompatible, directions.

Conventional measures of success like market share and profitability, however convenient and problematic they may be, simply don’t apply to a non-profit like ours. This puts us in the difficult position of having to define our own success. As Cholmes is so fond of saying, we’re not a non-profit but a ‘for-benefit’, so do we define success as the amount of ‘benefit’ we produce? How would you go about defining our success?

Coming up:

• Towards a Definition of Success
• Defining Failure
  

­

So, as 2007 draws to a close (I know it’s only November, but I’ve already bought all my christmas presents so I’m feeling a little ahead of schedule), it seems appropriate that we take a moment to look back on what we’ve accomplished this year, and think about where we’re headed for 2008.

Rather than try to sum up a year in my own words, I thought I’d put it out to you. Here are a few questions to get you started (relating to OpenCore/OpenPlans and TOPP):

• What are you most proud of from 2007?
• If you could change one thing about 2007, what would it be?
• What would you like to accomplish in 2008?
• Any new years resolutions?