I spent most of my second week doing some Xinha UI work with Doug. Our broad directive was to improve the Xinha user experience, so we started by talking with Sonali and Bryan about known interface bugs and quirks, and got a good idea of where some of the major usability shortcomings are. My general thoughts on WYSIWYG editors for HTML-based content are that they’re almost always mediocre and are hard to do well as full-fledged desktop applications, and are even tougher to do well as web-based apps. To date, I’ve never seen any browser-based WYSIWYG editor that didn’t require me to either fight its idiosyncricities or delve into the source to get it to do exactly what I wanted it to. Xinha has come a long way, but there’s definitely still a lot of work to be done.

We decided to start by concentrating on Xinha’s popup dialogs, which are used in plugins all over the place, e.g., for adding a link or inserting an image. Since many people find popups annoying, not to mention the fact that they interfere with people’s popup-blockers, we decided to look into converting the existing popups into lightboxes. Not wanting to reinvent the wheel, we looked at several different options for lightbox libraries. I hadn’t realized just how many lightbox libraries have (no pun intended) popped up over the past year or two. Unfortunately, many of them have significant downsides: some aren’t released under free licenses, and more than a few appear to no longer be under active development. We ultimately opted to give nyroModal a shot, since it has had more than one release, uses JQuery, is relatively full-featured, and is released under an MIT license.

Not long after we started using nyroModal, however, we came across a branch of Xinha called new-dialogs that was started a few months ago by one of the core Xinha developers. As its name suggests, the branch is a major update to Xinha’s dialog system and has had a significant amount of work put into it. Details of its development can be found in the change history for ticket #1176. In short, it does almost exactly what we were hoping to do, and overall, does it quite well.

Interestingly, new-dialogs does not use any of the aforementioned libraries for its lightboxes. Instead, it includes its own lightbox code written from scratch. Its dialogs are resizable, draggable, and even dockable, though I’m not sure how necessary (or even desirable) the first two are for a modal system.

Much of what’s left to do with new-dialogs is to transition all of the plugins that use the old popup dialogs over to the new system. Doug and I have started working on this process by updating the UnFormat plugin.  Doug has since done the BackgroundImage plugin and I’ve done the Abbreviation and FindReplace plugins. There are still quite a few plugins to go, but once the rest are updated there shouldn’t be much holding us back from making the jump to a popup-free Xinha.

Filed June 30th, 2008 under ui, xinha

The Promise

In preparation for a ramp-up in testing the OpenPlans Plone 3 upgrade efforts, I’ve been revisiting OSAF’s Windmill functional testing tool. The story is compelling; if you’re going to be launching a browser and clicking around to test a site anyway, why not turn on a recorder that will auto-generate test suites that can reproduce your actions? Even if it ends up making the original testing take a little while longer, that pays for itself the first time you re-run the tests.

With something concrete to accomplish, then, I sat down to see if Windmill delivers on this promise.

Bootstrapping

Getting started went pretty smoothly. Installing was easy, on Ubuntu Gutsy, anway; just regular setuptools stuff. I’m working from the Windmill trunk, so I created and activated a python 2.5 virtualenv, checked the code out from http://svn.osafoundation.org/windmill/trunk/, and ran ‘python setup.py develop’ to install it into the environment.

Windmill is then launchable with ‘windmill firefox URL’. This will open firefox, albeit without any of the customizations that you might have set up in your profile. It also opens a smaller Windmill IDE browser window, and starts a Windmill “controller service”, which exposes the API for manipulating the browser window.

Capturing Tests

The IDE is pretty simple. It’s implemented in HTML and Javascript, and it provides 4 primary features: a test recorder, a test runner, a DOM explorer, and an “Assert” explorer. The test recorder is what I’m first interested in. I click the record button, and the browser window jumps to the foreground. So I start clicking around, entering text and submitting forms, lo and behold, I can see what I’m doing being captured in the IDE. Good start.

To start, I make an incorrect login attempt, and, upon failure, try to create a new account, getting all the way to the “check your email” confirmation screen. I turn off the recorder, and click on the ’save’ link in the IDE window. Windmill can export the test suites as either python or JSON. I’ve chosen python, so I get another window with the following text:

# Generated by the windmill services transformer
from windmill.authoring import WindmillTestClient

def test():
    client = WindmillTestClient(__name__)

    client.click(link=u'Sign in')
    client.type(text=u'bogus', id=u'__ac_name')
    client.type(text=u'bobobobo', id=u'__ac_password')
    client.click(name=u'login')
    client.click(link=u'Create account')
    client.type(text=u'testuser1', id=u'id')
    client.type(text=u'Test UserOne', id=u'fullname')
    client.type(text=u'test1@example.com', id=u'email')
    client.type(text=u'testy', id=u'password')
    client.type(text=u'testy', id=u'confirm_password')
    client.click(name=u'task|join')

I paste this into an emacs buffer and save it as windmill_tests.py. Then I shut down the windmill process (I have to ‘kill’ it to make sure everything terminates correctly :P) and pass in the test I just created with ‘windmill firefox URL test=windmill_tests.py’. Sure enough, the browser launches, hits the site, and starts following links and entering text! Initial success makes much happy.

Controller API

The ‘client’ variable in the python code above is a handle to a WindmillTestClient object, which exposes the Windmill Controller API. The controller is what is driving the browser. One fun trick is that you can put a pdb in your test code and you’ll get an interactive prompt which you can use to control the live browser window. Now enjoy the power of a python command line interface to teh interwebs, while still being able to access all the rich multimedia that today’s users demand! It’s definitely more fun than it should be to fill out web forms and click on links by typing in ‘client.type’ and ‘client.click’ commands.

Assertions

Astute readers will have noticed that the second run of the user creation test would have a different result than the first. Indeed, when I used Windmill to run the test steps I’d captured, it ended with an error message informing me that the username was already taken. Real tests of course will want to make assertions, to ensure that the behaviour is really what you want. Enter the Assertion Explorer. At any point, you can click on the Assertion Explorer button, and then click somewhere on the page, and the IDE will generate a best-guess assertion for you for the element that you selected.

The generated assertions are sometimes spot-on. Whenever I clicked on a portal status message, for instance, it asserts (using XPath) that the PSM element exists on the page, and that it contains the specified text. When I click on regular page text, however, it only checks for the existence of the clicked node, when I want it to check the text as well.

The asserts are a part of the Controller API, and are pretty easy to understand. Here’s an example of a couple that I generated:

    client.asserts.assertNode(xpath=u'/html/body/div/div/div/div/div')
    client.asserts.assertText(xpath=u'/html/body/div/div/div/div/div', validator=u'Welcome! You have signed in.')

    client.asserts.assertNode(xpath=u’/html/body/div/div/div/div/div’)
    client.asserts.assertText(xpath=u’/html/body/div/div/div/div/div’, validator=u’Your changes have been saved.’)
    client.asserts.assertNode(link=u’something else+’) 

Nothing particularly mysterious there.

The Warts

Despite the early successes, it wasn’t long before I hit a couple bumps. Here’s an overview of the issues that came up for me:

  • Hard to extract info from the page

    The Controller API works pretty well for controlling the page and making assertions, but there don’t seem to be very good hooks for extracting information from the page itself into the python environment. For instance, in order to complete the user registration, the test will need to log in as an admin and hit a special page which will return the user’s confirmation key. This will need to be extracted from the response and then used as part of the URL of a subsequent request. Even a simpler case, such as simply extracting the URL of the page that the browser is currently visiting, doesn’t come clear to me from a thorough comb of the documentation.
  • The ‘waits’ stuff seems to be broken ATM

    The Controller API has a whole selection of ‘waits’ methods, which will tell the test running to not move on until some criteria is met, either a specific amount of time has elapsed, or the page has loaded, or an element shows up on the page, etc. For me, running FF2.0.0.14 on Ubuntu Gutsy, these were completely broken. Any time I try to use one of these calls, the test suite just stops right there. This is a show-stopper, since I very quickly hit false failures due to the tests running more quickly than the browser was responding.
  • Xinha typing didn’t get picked up by the recorder

    Most of the actions I performed in the browser were dutifully recorded by the IDE. The only exception to this is any text I would enter into the Xinha editor. Whenever I would edit a wiki page, the recorder would skip directly from clicking the ‘edit’ link to clicking on the ’save’ button. I think we can work around this by just adding these commands by hand to the generated python. I’m not 100% on this, though.

Python is Nice

As I’ve mentioned, it’s possible to export the generated tests and assertions as either JSON or python code. Having such a straightforward way to drive the tests from pure python is a big plus for those of us who like that sort of thing. We can use try:finally to make sure clean-up code gets hit, and httplib2 to talk to the server to actually perform the clean-up. Control is easy in python. A quick peek at the code indicates that it does pass unrecognized options in to the test suites themselves, which means that we can code up test suites that understand additional control parameters as we need.

The Good News

While I did stumble on a couple of issues, I am very happy to report that the folks in the #windmill channel on freenode are very helpful. Even better than that, they’re very happy to get my bug reports, and are very responsive about fixing them. I spent about a day playing with Windmill a few months back. In that time I uncovered one bug and one usability weirdness. They were both fixed within days. The issues I’ve raised this time are already recorded as issues in their tracker, and I’m told that they should be resolved by the end of the week.

Conclusions

Takeaways? I’d say “cautiously wildly enthusiastic” best describes how I feel. Windmill is very close to delivering on its promise, making it ridiculously easy to generate robust, JavaScript-supporting test suites that can be trivially run on IE, Safari, and FireFox. If the pattern of developer responsiveness continues, and the issues that come up are either easily worked around or are resolved within Windmill itself, then I think this really hits the sweet spot. It didn’t take long for me to hit a couple of pretty big issues, however. If it turns out that there are a lot of similar bombs in there, that the problems stack up faster than the developers can deal with them, then it would probably end up being a headache for us.

I don’t really expect this, however. I know OSAF is using Windmill internally, and the developers that I’ve had contact with seem very enthusiastic about getting my reports and getting the problems resolved. I’m going to continue my exploration, and I hope to generate a thorough set of Windmill tests for the Plone 3-based OpenPlans stack over the next week.

Filed June 25th, 2008 under Uncategorized

This week was my first as a regular TOPP employee, after having not been here since I was an intern last summer. The current plan is for me do a few rounds of rotations, pairing with folks for a week or so to get a taste for what they’re doing and (hopefully) contributing something helpful. After getting settled in a bit, I began my first such pairing with Jeff, working on adding CAPTCHA support to the new user registration page on Trac provided by the AccountManager plugin, per ticket #8.


Broadly speaking, there are two primary ways to tackle this problem. The first is to write a stand-alone plugin that implements the IRequestFilter and ITemplateStreamFilter interfaces. The second  approach comes in two parts, the first of which is to modify the AccountManager plugin to provide a new extension point for making additions to the user registration page. The second part is to then write a plugin that implements this new interface to actually add the CAPTCHA to the form and then verify it is correct upon submission.


As an example of the first approach, I wrote a quick and dirty prototype, CaptchaAuth. To explore the second option, I wrote RegistrationConfirmationPatch to modify AccountManager to include a minor variation of the IRegistrationConfirmation interface suggested in ticket #8. I then wrote SimpleCaptcha to implement this interface and provide very basic CAPTCHA support.


As it stands, the pure plugin approach, CaptchaAuth, is a little hackish. While it does have the advantage of not requiring any modifications to AccountManager, it has some downsides. First, it relies on an XPath to grab on to the portion of the form to add the CAPTCHA to. This means that future changes to AccountManager could inadvertently break it. Second, error messages are poorly integrated, and as of now raise an ugly TracError instead of looking like other form validation errors on the page.


The RegistrationConfirmationPatch/SimpleCaptcha approach, on the other hand, has the downside of requiring modification to AccountManager, but does not have the other problems mentioned above. It also has the advantage that the patch provides a generalized interface that could be used to add any additional elements to the new user registration form. As an example, the same interface could be used for e-mail verification of new users.


I’ve e-mailed the Trac dev listserv to get to some input from the core developers and the community at large. Hopefully we’ll be able to work together to come up with a flexible and clean solution for allowing plugins to add form elements to the new user registration page.

Filed June 20th, 2008 under first-week, trac