Agile Underwear Gnomes

Agile is all about the profit

While considering my teammate Amos’ recent blog post where he debates what Agile really is my mind was drawn to a statement our current client made during one of our early project kickoff meeting.  While discussion why his clients have been successful following his plans he made the statement:

You will never go broke, making a profit

At the time this seemed like a good joke and litte more.  However, the more I thought about it, the more it rang true.  What does this have to do with Agile software development?  While questioning what is or is-not Agile and what his definition of Agile might be missing, he says:

… The one thing that I didn’t mention above. … Delivering a useable project at any point in development. That is it. That is what I’m missing. This has got to be the hardest part of agile. …

I think the key here is the value of a shipped product is ALWAYS greater than that of one sitting on the shelf.  Value is the key.  To me, the agile mindshift is one of always producing value for the customer, for the team, or for yourself.

Produce value for the client:

• by adapting to their ever changing business needs with understanding, speed, and energy
• by aligning all user stories with tangible user value and goals
• by planning and documenting in inverse proportion to the item’s priority and deadline
  • worry about what is in front of you first and put off decisions to the last responsible moment (stolen and probably butchered from a WWBBD: What Would Brian Button Do)
• by working on the highest bang-for-the-buck stories first, top down, and delivering working code as often as possible

Produce value for the team:

• by conducting frequent retrospectives where the team can think about how they are doing and how they can be better
• by realizing that many/most of a team’s problems are really “people” issues masquerading as technical ones
• by actively paying down technical debt
• by reducing knowledge silos and keeping the team only full of valuable members and all members valuable

Produce value for your self:

• by staying motivated and positive
• by keeping the thirst for knowledge alive and always learning from any experience
• by taking greater pride in the success of the team over the success of an individual ego
• by working to be a great communicator: to team members, to the client, to your code (Tests)

I didn’t mean for this post to turn into a list of things to do … but often our practices are the best way we can take abstract principles and make them tangible.  So long as the practices don’t become to focal point, they are a useful way to monitor, enforce and discuss the important concepts.

Profit really is the key.  Our client helped his clients by making sure they were profitable.  Sure, maybe they could have been more profitable, if they had taken bigger risks and bet the farm for instance, but year after year a steady profit is inspiring.

Which brings me to the Underwear Gnomes of South Park.  Their business plan was a simple one.

1. Collect Underpants
2. …
3. Profit

By focussing on providing value (profit) at each point along the way, maybe step 2 wouldn’t be such a mystery.  Granted, gnomes are pretty agile to begin with.  Fast too.

Test-infected Marshmallows and Being a Passionate Professional

Alex Miller presented a talk at his recent Strange Loop conference about Walter Mischell’s marshmallow experiments.  Mischell studies children’s ability to defer gratification when they were put in a room with a marshmallow and told they would receive another one in ~20 minutes if they didn’t eat the first.  The longer term study found that kids who were able to wait, scored much higher in “life metrics” later on (e.g. work success, school studies, etc).

Alex’s talk got me thinking about deferring gratification in software development …. which imediatley made me think about testing.

In college, there were no tests.  Just me, alone, with lots of code.  And, there were lots of headaches and confusion.  Fast.

Now I write tests and consider the behavior of my system before I write a single line of code.  Now every line of code was written with a pair.  After every iteration the team gets together to discuss how we are doing and how we can do it better.  There are some headaches but a lot less confusion.  Now there is pride and quality.

Having the discipline of writing tests first, and developing software in thin vertical slices from the top down is hard.  It is hard because we desire closure, and immediate feedback.  But by focusing our energy on producing a long term quality product, we can achieve a higher level of satisfaction.

Being surrounded with that many passionate professionals makes you want to be better.

A huge thanks to Alex for putting on a great conference here in St. Louis.  I can’t wait for next year.

placeholders and currying

Wherein Javascript borrows a language feature from Arc

Today, I was going to write about Google Maps until I stumbled onto this little gem. My post still involves Google Maps code, but my focus is on Javascript and how virtually anything you can think of that might make the language more of a joy to program in, you can implement yourself.

Our post begins with a plain Google Maps application, virtually copied right from Google’s API docs example. I made a few changes though, highlighted below. I hate mixing event handlers in HTML tags and I have, as of yet, still not installed a decent mixed mode for editing Javascript inside of an HTML source file. So I pulled the embedded script into it’s own file and added listeners for load and unload events on the window.

<html xmlns="http://www.w3.org/1999/xhtml">
  <head>
    <meta http-equiv="content-type" content="text/html; charset=utf-8"/>
    <title>Google Maps JavaScript API Example</title>
    <script type="text/javascript" src="http://maps.google.com/maps?file=api&v=2&sensor=true"></script>
    <script type="text/javascript" src="maps.js"></script>
  </head>
  <body>
    <div id="map_canvas" style="width: 500px; height: 300px"></div>
  </body>
</html>

function initialize() {
    if (GBrowserIsCompatible()) {
        var map = new GMap2(document.getElementById("map_canvas"));
        map.setCenter(new GLatLng(37.4419, -122.1419), 13);
        map.setUIToDefault();
    }
}

function destroy() {
    GUnload();
}

window.addEventListener("load", initialize, false);
window.addEventListener("unload", destroy, false);

The default map is centered on Google’s headquarters in Palo Alto. Of course, Palo Alto isn’t very pleasing to look at for anyone who isn’t a smug self-centered Palo Altoan, but I suspect that most people don’t change this, at least initially, because it’s daunting to try and figure out what your hometown’s latitude and longitude is. Google Maps itself, though actually provides a decent way to turn a string location into a GLatLng, using a GeoCoderThingy. The getLatLng() method takes a callback. I live in St. Louis and my original post had to do with our lovely Forest Park, so the code below demonstrates how to center your map on America’s largest urban park.

function initialize() {
    var geolocator = new GClientGeocoder();
    geolocator.getLatLng("Forest Park, St. Louis, MO", displayMapAt);
}

function displayMapAt(center) {
    if (GBrowserIsCompatible()) {
        var map = new GMap2(document.getElementById("map_canvas"));
        map.setCenter(center, 13);
        map.setUIToDefault();
    }
}

Notice the extract parameter and method rename refactoring here. It’s good and our code is clean, but Forest Park doesn’t appear prominent enough on the map for my liking. It should be bigger, more zoomed in. Pretty easy, just extract parameter around the hardcoded zoom level, and adjust the callback a bit since now displayMapAt will take two parameters.

    geolocator.getLatLng("Forest Park, St. Louis, MO", function(center) {
	    displayMapAt(center, 14);
    });

function displayMapAt(center, zoom) {

This is ugly. Anonymous functions are powerful, but hell if I hate reading them, all curly and indented funny and pretending to be simple variables when there’s a bunch of characters there that don’t ever belong in any variable name ever.

We can use my fn.js (see Lambda Lounge, February 2009) and curry() to rewrite the callback like so:

    geolocator.getLatLng("Forest Park, St. Louis, MO", displayMapAt(14));
...
var displayMapAt = function(zoom, center) {

Much better, but we had to rearrange displayMapAt‘s parameters to get it to work and they were in a good order before, all matching the order they’re in when they’re used and such and not to mention what would happen if displayMapAt wasn’t your code and you couldn’t just mix it all up willy nilly. Haskell, which I borrowed most (all) of the ideas for fn.js from has the notion of “reversing” a function; basically creating a new function that takes the original functions arguments in reverse order. That would work here, but Javascript and Haskell’s syntactic beauty lean in different directions, so I looked to a different place for inspiration to solve this tiny, tiny problem: Arc.

Paul Graham’s hundred year language, Arc, has this placeholder syntax for anonymous functions that is ever so useful.

arc> (map [+ _ 10] ‘(1 2 3))
(11 12 13)
arc> (trues [if (odd _) (+ _ 10)]
‘(1 2 3 4 5))
(11 13 15)

So I broke into fn.js and added an identity check for the underscore variable. Once your function has all of it’s parameters, if one of them is the underscore variable (not equal to, is the reference itself) then we once again return an intermediate curried-sorta function that will on final invocation replace it’s placeholder value with a newly passed one. Right now the code only supports a single placeholder parameter, but there’s no reason you couldn’t alter the check and replace code to handle multiple placeholders.

Updated: see this Hacker News thread and Tom Robinson’s comments below for the whole story, but suffice to say === doesn’t work the way I thought it did, and _ needs a value and a simple == comparison. Also, these placeholder changes have been pushed to fn.js on github.

var _ = {};

function curry(f) {
  ...
	var placeholder;
	for(var i = 0; i < args.length; i++) {
	    if(args[i] == _) {
		placeholder = i;
		return function(replaceWith) {
		    var allargs = args.slice();
		    allargs[placeholder] = replaceWith;
		    return f.apply(null, allargs);
		};
	    }
	}
  ...
}

In the end, our displayMapAt function keeps it’s original parameter ordering, our callback isn’t an ugly inline anonymous function and the underscore very succintly points out where a variable is missing and that a future call will replace it with a real value.

    geolocator.getLatLng("Forest Park, St. Louis, MO", displayMapAt(_, 14));

var displayMapAt = function(center, zoom) {

fin.

Mutual Authentication with CLIENT-CERT, Tomcat 6, and HttpClient

M is for Mutual Authentication:

How “it must be simple” turned into “god that was annoying”.

I spent most of today wrestling with getting a JEE webapp running in Tomcat 6 to enforce mutual authentication by using the rarely used CLIENT-CERT authentication.  If you love spending your days translating poor error messages and frustrations into Google queries as much as I do, read on for some notes I took along the way.  If not, check out this amazing chart over at XKCD.

The mission: lock down a particular action on our web application so that only a single other system could authenticate and invoke it.  Also provide an example client application which can access the webapp.

Back story: I have worked with certificates and various forms of authentication before, but never this combination.  We want a small subset of the application to only be accessed via SSL with both server and client providing trusted certificates for mutual authentication.  The portion of the application will only be accessed by a single other system inside the company intranet.

How it went down:  The following are the basic steps to get everything setup in Tomcat 6 and a fairly standard JEE webapp.

• First we need to know who we are, who they are, and make sure we ONLY trust them and no one else
  • To do this we need a certificate to identify ourselves.  Java’s provided keytool.exe provides an easy way to create a self signed certificate within a keystore.  Keytool can also help us export our public certificate out of our keystore.  This should also be done for the client.
  • Next import the public certificates into a new truststore for the opposite system (A only trusts B and B only trusts A).  Keytool to the rescue again.
  • Note: Java keystore files and certs of in a different format than the PKCS12 files that can be created by OpenSSL
• Next configure a Tomcat to support SSL communication
  • Add a new connector to thees server.xml configuration specifying HTTPS, the SSL protocol, where the webapp’s keystore and truststores are, etc
  • <Connector className=”org.apache.coyote.tomcat4.CoyoteConnector”
    port=”8443″ enableLookups=”true”
    acceptCount=”100″ connectionTimeout=”20000″
    useURIValidationHack=”false” disableUploadTimeout=”true”
    scheme=”https” secure=”true” SSLEnabled=”true”
    keystoreFile=”ourApp.keystore” keystorePass=”changeit”
    truststoreFile=”ourApp.truststore” truststorePass=”changeit”
    clientAuth=”false” sslProtocol=”TLS”
    />
  • Note: clientAuth can have 3 values
    • True – all connections through this connector require client authentication
    • Want – the web app will ask for authentication but not require it
    • False – connections do not require client authentication UNLESS the web app specifies it is required via a security constraint with CLIENT-CERT chosen (this is the one we want)
• Configure a new user with the required security role in the tomcat-users.xml if the in-memory realm is used
  • <role rolename=”secureconn”/>
  • <user username=”CN=TheirApp, OU=Unknown, O=Unknown, L=Unknown, ST=Unknown, C=Unknown” password=”null” roles=”secureconn”/>
• Configure out web.xml to use a CLIENT-CERT authentication constraint for our action
  • <security-constraint>
    <web-resource-collection>
    <web-resource-name>Demo App</web-resource-name>
    <url-pattern>/secure/*</url-pattern>
    </web-resource-collection>
    <auth-constraint>
    <role-name>secureconn</role-name>
    </auth-constraint>
    </security-constraint><login-config>
    <auth-method>CLIENT-CERT</auth-method>
    <realm-name>Demo App</realm-name>
    </login-config>

    <security-role>
    <role-name>secureconn</role-name>
    </security-role>

• Create a client to hit the application
  • Here we used the Apache Commons HttpClient (3.x b/c of client restrictions)
  • Several pages talk about various ways to add security protocols to HttpClient code but the examples are left lacking and most point to the AuthSSLProtocolSocketFactory class which for some reason is not present int he 3.x binaries but IS present in the 3.x SRC bundles.
  • ** I will include some sample code tomorrow ….
• Test things using Firefo
  • You can add a personal certificate to identify yourself through Firefox’s preferences -> Advanced -> Exncryption
  • Firefox uses PKCS12 certificates which can be created easily from scratch through a few OpenSSL commands or from an existing keystore certificate through one hairy keytool command
  • ** I will include the commands and a couple links tomorrow …..

Hopefully these notes will help someone in the future with getting things setup correctly.  When I started this morning I couldn’t find an end to end example to build on.

Edit: Huge thanks to Matt Todd for helping me with this.  You can check out his blog at: http://emergentdevelopment.blogspot.com/ .

Webkit CSS Transitions

The iPhone’s Safari browser has a few tricks up it’s sleeve to make web apps feel more like native ones

In contrast to Javascript-controlled animation frameworks like Scriptaculous, the iPhone’s Webkit browser is capable of animations that utilize the native hardware.

How to do this was a bit confusing to me at first, but there are really only 2 and a half steps:

1. “Observe” the properties whose changes you’d like to animate by listing them in a comma-delimited value of -webkit-transition-property
1.5 (Specify an animation duration via -webkit-transition-duration)

div {
  -webkit-transition-property: opacity, background-color;
  -webkit-transition-duration: 1.5s;
}

and

2. Change the property (maybe in Javascript in response to an event)

function handleEvent(event) {
    event.currentTarget.style.opacity = 0.0;
    event.currentTarget.style.backgroundColor = 'blue';
}

and instead of instantly re-rendering the element with the new style, Webkit will transition the “observed” CSS properties from their original values to their newly set ones over the duration you specified.

I don’t usually even bother writing the observation code in my stylesheets, either, as jQuery makes it dead simple for whatever DOM object I’ve currently got a reference to:

$(anelement).css("-webkit-transition-property", "opacity, background-color")
            .css("-webkit-transition-duration", "1.5s");

What makes this cool and powerful is that the iPhone’s browser defines an extra CSS property, -webkit-transform, which describes full 3D transformations that can be applied to page elements. Animated transitions can observe this property just like any other CSS property and very nifty, fully 3D, and natively optimized animations can be created with a fraction of the code a native app utilizing OpenGL might have to write.

Apple published this guide with more detailed information on all of the visual effects that Safari supports.

Cucumber: Making UATs the healthy choice

My current project team has been eating its vegetables.  Has yours?

Selenium is a great tool for testing webapps from a true browser driven user’s perspective.  By driving most mainstream browsers via a core Javascript library, a Selenium script can easily test how your app behaves (or misbehaves) just like a real user would see.

“Webrat lets you quickly write robust and thorough acceptance tests for a web application.”  Written in Ruby, Webrat allows acceptance tests to exercise an arbitrary web application through its Selenium integration, or Rails apps without starting a server.

Cucumber allows code to be written in a simple domain specific language (DSL) that anyone can read.  Layer Cucumber on top of Webrat and Selenium, and you have User Acceptance Tests (UATs) that the client can read (and ideally write), that the entire development team understands, and which can be run as BDD or regression tests.

Cucumber example taken from cuckes.info

Without going into the details of how each of these libraries work, below are some of my notes, tips, and impressions of using Cucumber to write UATs for a greenfield Java EE application over the past ~10 weeks.

• Ideally, each test would run in its own transaction so that they wouldn’t interfere with each other and could be run in parallel.  Using Cucumber (Ruby) to test a totally separate web application running on Tomcat (Java) presents a problem.  The team debated several possible solutions using JRuby and having the Cucumber build spawn an embedded Tomcat instance but decided that the cost outweighed the benefit to the current project.
• Since we were forced to use pure JDBC by our client, we decided to use ActiveRecord migrations to build our tables, and generate SQL which we could re-leverage from our Cucumber tests.  Custom creation Webrat steps were written to push setup test data for each model.
• DB2 support with the Ruby ActiveRecord drivers is lacking proper install instructions on Linux (damn thing requries a full DB2 install to run). By leveraging JRuby, we were able to use the DB2 JDBC drivers instead which worked much better (without installing DB2).  Getting Cucumber, Webrat, and Selenium running in JRuby took a bit of finesse and shebang wrangling but eventually worked.
• Outputting Cucumber results to a JUnit style XML output made our integration on our Hudson CI server simple and easy to read. (The Chuck Norris plugin helps too)
• We tag each test with a story number (e.g. @10045) so that we can quickly run all tests for a given feature easily: cucumber –tags @10045.
• Our QA lead works to define and write-up our Cucumber tests during pre-iteration planning and before each story enters our work queue (the team uses a modified Kanban board approach to pull work through each iteration, Cucumber is our first queue).  When she has a test written, she will tag it with @in-process to let us know that it is ready to be worked on, but not yet implemented.
• We have custom Rake tasks to run all of our “finished” tests as well as just our “in-process” ones.  The in-process task will fail if any tests pass (they should be marked as finished).  Unfortunately this rake task doesn’t work with the Hudson/JUnit build.
• Ongoing issues we still struggle with are:
  • making sure that small variations in DSL verbiage don’t muddy up our tests (it is easy for the team to accidentally end up with two different commands which do the same or similar actions without a clear distinction)
  • cross browser and separate environment testing happens on different dedicated servers on dependent builds so that they don’t end up stepping on each other
  • as it is with any testing library, care needs to be taken to keep only common tests grouped together and setup/background work common to all tests that NEED it.  Pushing too much work into common actions results in slow tests
  • Tests are supremely faster and more valuable than manual tests, but are not as fast as JUnit tests and will never replace manual exploratory testing.

Overall the team has been very pleased with our ability to drive our development from the tests and verify that new changes don’t break existing functionality from the users’ perspectives.

* Another Asynchrony team is working on leveraging Cuke4Duke to do similar testing of a Java thick client.  Should be interesting.

Edit: Thanks to Amos King for some proof reading help.  You can find his blog over at Dirty Information.