Stay on Target

Improve Your iOS Frank Cucumber Acceptance Tests

Here are some tips I've learned to build more readable, flexible acceptance tests for iOS devices using Frank and Cucumber.  See the full article on the Cyrus Cylinder:

The Cyrus Cylinder: Improve Your iOS Frank Cucumber Acceptance Tests

Introducing Bane, a Test Harness for Server Connections

Bane is a tool that makes it easy to test your application in scenarios where any third-party server dependencies behave in unexpected ways.  This post explains why you might want to use Bane, how to get started, and the design motivations behind Bane. 

Problem: The World is Unsafe (For an Application)

If you are building an application, you may depend on third-party servers or other external sources for your data.  You may use any of the publicly available web services from companies like Google, Yahoo!, or Twitter.  Or perhaps you use web services in an internal business application to obtain data from authoritative sources.  Most of the time these services are reliable and work as you expect.

Yet however reliable these services claim to be, at some point they’ll behave in an unusual manner.  The servers may be busy and accept your connection but never respond, they might reply in an unexpected response format, or there might be network problems that require endless retransmits of the data - and that's just a few of the ways that things can go wrong.  It’s up to you to program defensively so that if the remote service fails, your application doesn’t fail as well.  Michael Nygard’s excellent book “Release It!” is full of patterns and anti-patterns for building applications that can survive in these situations; I can’t recommend it highly enough.

To make sure your application isn’t susceptible to external service failures, you should test your application under various hostile conditions.  However, getting your external services to behave in the desired fashion can be a real challenge.  Instead, Nygard recommends building a tool that can simulate various bad behaviors, as described in the chapter “Test Harness.”  

Introducing Bane

Bane is an open source implementation of the test harness described in “Release It!”  Bane is written in Ruby and is available as a Ruby gem.  To install it, you’ll need a working installation of Ruby.  Install Bane with “gem install bane” which makes an executable “bane” available from the command line.  To make sure Bane is installed correctly, type “bane” with no arguments from the command line.  You should see a usage message like this:

$ bane

Usage: bane port_number <servers>

All behaviors:

 - CloseAfterPause

 - CloseImmediately

 - DelugeResponse

 - DelugeResponseForEachLine

 - FixedResponse

 - FixedResponseForEachLine

 - HttpRefuseAllCredentials

 - NeverRespond

 - NewlineResponse

 - NewlineResponseForEachLine

 - RandomResponse

 - RandomResponseForEachLine

 - SlowResponse

 - SlowResponseForEachLine

Bane requires you specify a port number on which to start listening and a behavior to use to respond on that port.  A behavior is one of the several nefarious ways in which Bane can respond, such as never responding, sending random data, closing the connection immediately, etc.  Behaviors are named in CamelCase (e.g. NeverRespond, NewlineResponseForEachLine) because they correspond to Ruby classes.

A Usage Example

Imagine you’ve built an application to track financial investment portfolios.  All stock quotes are obtained via a third-party web service.  You’d like to see how your application behaves when the web service accepts your request but never sends a byte of data in response.  Let’s pretend that your stock quote service listens on port 8080.

Start up the NeverRespond behavior on port 8080, which accepts a connection but then never sends a response.

$ bane 8080 NeverRespond

[Tue Sep 28 10:49:25 2010] NeverRespond 127.0.0.1:8080 start

Point your application at port 8080 and see how your application behaves.   Notice that Bane logs the incoming connections:

[Tue Sep 28 10:50:05 2010] NeverRespond 127.0.0.1:8080 client:61459 127.0.0.1<127.0.0.1> connect

You may need to add timeouts or other failure strategies to keep your application functioning under this scenario.  Consult “Release It!” for more information and other alternatives.

Bane comes with several bad behaviors you can use to test out your application in the manner described above.  See the README and wiki pages for more information.

The Design Motivations Behind Bane

Before I created Bane, I used to write little one-off scripts to create these badly behaving servers.  They aren’t much work; Ruby’s standard library includes a GServer class that makes it easy to quickly create a multithreaded TCP/IP server.  In fact, if you look at the source code, you’ll see that Bane uses GServer internally.  So why not just create these little servers each time by hand?

It turns out that each time I had to create one, I’d take about five minutes to bring up the RDoc and get all the parameters right.  So I created Bane to have all these common bad behaviors available in just a few seconds.  Right now Bane only supports two response protocols: send a response immediately upon connection and respond after each newline sent by the client.  I haven’t needed other response protocols on my projects yet, but it should be simple to create variations that respond after every n bytes, etc.  

However, if it turns out there needs to be some advanced protocol (e.g. respond after certain keywords, preserve state, etc.) then you may be better off spending the five to ten minutes to write your own server in Ruby using GServer or Sinatra, or some other language of your choice.  My goal is to keep Bane easy to use for common cases, to get a bad behavior up and running in a few seconds rather than five minutes.

Manual or Automated?

I use Bane primarily for manual exploratory tests, and write unit tests around my applications to create fast, repeatable automated tests.   Typically I use Bane in my consulting work to test out an existing application and see how it behaves, find the failure modes, write automated unit tests to expose the behavior, and then fix the problem by applying the patterns described in “Release It!”  Many of these unit tests use test doubles to throw exceptions that would be created by the I/O library in use.

That said, Bane provides an API through the Bane::Launcher class for automation in Ruby.  Take a look at Bane’s integration tests and the examples folder in the source distribution for an example of this API in use.

How to Contribute

Bane is open source and I welcome your feedback or contributions.  Here are a few ways you can contribute:

Provide Feedback: Please take Bane for a spin on your project and let me know if it helps, where it didn’t, and any suggestions for enhancements.

Write new behaviors: Fork Bane on GitHub and contribute useful bad behaviors or any other items described in “Release It!” which are not yet implemented (see the README for a list).  In particular, Bane does not yet support any of the low-level protocol errors in “Release It!”, such as the remote end endlessly sending TCP/IP resend packets.  To my knowledge, this low-level behavior is not possible with Ruby’s I/O library and requires a C extension.

Code Reviews: If you know Ruby and have some ideas about how to improve the code, please fork Bane and suggest some changes.

For more information

• Bane at GitHub
• Bane’s FAQ
• E-mail me at dan at danielwellman dot com

Flexible Argument Matching with Mocha

If your Ruby unit tests are failing because you change the signature of a method and all your mock expectations fail, you might have a case of Overspecified Software.  To fix this problem, make your test require only the parameters that are significant.  To quote Gerard Meszaros, "If a value is not important to a test, it's important that it not be in the test."

For example, imagine you had a test which took an options Hash as a parameter.  You might have an expectation like this:

processor.expects(:order).with(:payment => :credit_card, :delivery => :express, :group => :separate)

Unless your test is verifying that all of these parameters are set correctly, it might be better to change your tests so that each test verifies only the parameters that are important.  In this post, I'll describe how to do this with Mocha, but Flex Mock also provides similar features.

Mocha provides a flexible syntax to match parameters of expectations.  For example, if you wanted to check that a given key was present, and ignore the rest of the arguments, you could say:

processor.expects(:order).with(has_key(:payment))

If you are interested in both the key and value, you could use has_entry, or the alternate has_entries:

processor.expects(:order).with(has_entry(:payment => :credit_card))

If you just wanted to make sure something was NOT present, you can use the Not operation to invert the condition:

processor.expects(:order).with(Not(has_key(:emergency_order)))

Or specify any number of conditions using the any_of condition.  

processor.expects(:order).with(any_of(includes(:delivery), has_key(:shipping)))

Mocha provides many more variations, see the Mocha::ParameterMatchers RDoc for more possibilities.

By combining these parameter matchers, you can make tests that are more expressive about the arguments they care about and avoid unrelated test failures when you make a harmless change to your code.

Automated Testing Tips from the Google Wave Team (on Google Web Toolkit)

Adam Schuck of the Google Wave team spoke at Google I/O 2009 about building the Wave client using Google Web Toolkit (GWT).  You can watch a video of the presentation online: Google Wave: Powered by GWT. The Google Wave client is a great demonstration of what's possible with GWT -- and what's coming in GWT 2.0.

One of Adam's key points was that automated testing is crucial to the project's success.  You can watch this section of the presentation starting at around 46:30.

One of Adam's slides lists the Wave team's testing strategies:

1. Use the Model View Presenter pattern to build testable client code.  This pattern decouples widget code from user interface logic. 
2. Prefer JUnit test cases over GWTTestCases, since JUnit test cases run a few orders of magnitude faster than GWTTestCases.  This means developers will be more likely to run the tests frequently.
3. Use an end-to-end browser testing tool like WebDriver to verify the whole system.  Adam emphasized making these tests as clear and readable as possible so that other developers can understand their intent.

If you'd like to learn more about testing GWT applications, here are some articles I've written on the subject:

• Google Web Toolkit: Writing Ajax Applications Test First demonstrates using the Model View Presenter pattern and discusses JUnit tests versus GWTTestCases.  This article was adapted and included in the GWT online documentation.
• Mocking GWT widgets with GWTMockUtilities describes how to write plain ol' JUnit tests to test code which manipulates GUI components.
• Testing JSON parsing using JavaScript Overlay Types in GWT 1.5 shows how to test JavaScript Overlay Types, the Wave team's preferred way of mapping JSON messages to objects.
• Strategies for Testing GWT Widgets and View Components provides some suggestions for testing custom UI components.

Paul Infield-Harm and I will be presenting "Agile AJAX: The Google Web Toolkit Experience" at the Agile 2009 conference in Chicago, IL, where we'll demonstrate how to use GWT to build and test applications in an Agile process.

Blogging at StickyMinds.com

I've started a blog on the StickyMinds website; I'll try to post a new entry every week on topics which include software development, process, and testing. 

This brings up the question: "How do I decide which site to use for any given blog post?"  My current strategy is to post the more code and tool-specific items here (such as GWT, Scala, etc.) and post the rest at the StickyMinds blog site.  I'm sure this will change, so watch this space.

Thanks to Joey McAllister and the folks at StickyMinds for this great opportunity.

Testing Tips Article in May/June 2009 Better Software Magazine

Is your team starting to write automated tests but you don't know where or how to start?  Have you test-driven a stack but you're not sure how to apply the TDD techniques to your project?  I've written about my experiences helping teams kick-start their test-writing in the article "What to Expect When You're Automating Testing: Test-last Tips from an Agile Expert".  It's now in print in the May/June 2009 issue of Better Software Magazine.  You can download the article in PDF format below.

Download PDF

Thanks to Pedro Yang for his feedback.  Continued thanks to the team at Better Software including Lee Copeland, Joey McAllister, Jean Middleton, and Heather Shanholtzer for their superb editing help.

Update: Thanks to Boris for catching the year typo.

Test-Driven Google Web Toolkit article in the GWT documentation

Sumit Chandel at Google has posted "Testing Methodologies Using Google Web Toolkit" in the official GWT documentation, an adaptation of my take on test-driven GWT.  You can find it in the articles section of the GWT project page. 

The example test code has been rewritten in EasyMock, which means you can now find an example in both JMock (in my original article) and EasyMock.

Thanks to Sumit for his feedback on the original article, and now for sharing it with the GWT community.

Mocking GWT widgets with GWTMockUtilities

When Google released GWT 1.5, they slipped in a testing utility class called GWTMockUtilities.  It's not very well known, in fact, it's not even in the documentation.  The top hit (as of the time of writing this blog post) in a Google search for GWTMockUtilities is simply a direct link to the source code.  So what is this class useful for?

Quite simply, GWTMockUtilities allows you to mock out GWT library classes using tools like JMock, Mockito, or EasyMock in your plain ol' JUnit test cases.  Here's the class JavaDoc description:

Defangs GWT.create(Class) to allow unit tests to mock out Widgets and other UIObjects

OK, so it has something to do with mocking widgets in unit tests, but it's still a little cryptic.  Let's look at the JavaDoc for the disarm() method:

Replace the normal GWT.create() behavior with a method that returns null instead of throwing a runtime exception. This is to allow JUnit tests to mock classes that make GWT.create() calls in their static initializers. This is not for use with GWTTestCase, and is not for use testing widgets themselves. Rather, it is to allow pure java unit tests of classes that need to manipulate widgets.

To understand what's going on here, let's try writing a test that doesn't use GWTMockUtilities.  Here's a simple test that uses JMock and tries to mock the GWT Button class.  Note the setup which enables CGLIB to mock concrete classes.

@RunWith(JMock.class)
public class MockingWithoutGWTMockUtilitiesTest {

    Mockery context = new JUnit4Mockery() {{
        // Enable CGLIB mocking of concrete classes
        setImposteriser(ClassImposteriser.INSTANCE);
    }};

    @Test
    public void mock_a_button() {
        // This throws an ExceptionInInitializer due to
        // a static initializer when loading Button
        Button button = context.mock(Button.class);
    }
}

Running this test yields the following exception:

java.lang.ExceptionInInitializerError
    at sun.reflect.GeneratedSerializationConstructorAccessor1.newInstance(Unknown Source)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:494)
    at org.objenesis.instantiator.sun.SunReflectionFactoryInstantiator.newInstance(SunReflectionFactoryInstantiator.java:40)
    at org.objenesis.ObjenesisBase.newInstance(ObjenesisBase.java:59)
    at org.jmock.lib.legacy.ClassImposteriser.createProxy(ClassImposteriser.java:133)
    at org.jmock.lib.legacy.ClassImposteriser.imposterise(ClassImposteriser.java:66)
    at org.jmock.Mockery.mock(Mockery.java:139)
    at org.jmock.Mockery.mock(Mockery.java:120)
    at com.danielwellman.gwtmocks.client.MockingWithoutGWTMockUtilitiesTest.mock_a_button(MockingWithoutGWTMockUtilitiesTest.java:26)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    ...
Caused by: java.lang.UnsupportedOperationException: ERROR: GWT.create() is only usable in client code!  It cannot be called, for example, from server code.  If you are running a unit test, check that your test case extends GWTTestCase and that GWT.create() is not called from within an initializer or constructor.
    at com.google.gwt.core.client.GWT.create(GWT.java:91)
    at com.google.gwt.user.client.ui.UIObject.<clinit>(UIObject.java:139)
    ... 33 more

The method GWT.create() is getting called in a static field initializer of UIObject, which happens when the JVM loads the Button class (which descends from UIObject).  The exception message recommends that your test should extend GWTTestCase, the integration test case which runs JavaScript in the hosted-mode browser.  But in our test we aren't using a real Button, we're just trying to make a fake one.  So why should we pay the test time penalty of running in the slower GWTTestCase?

GWTMockUtilities provides you another way to solve this problem.  By wrapping your test methods with GWTMockUtilities.disarm() and restore(), you can temporarily disable the harmful side effects of calling GWT.create() in a static initializer.  (It turns out that all widgets cause this behavior.)

Here's a sample:

@RunWith(JMock.class)
public class MockingWithGWTMockUtilitiesTest {

    Mockery context = new JUnit4Mockery() {{
        // Enable CGLIB mocking of concrete classes
        setImposteriser(ClassImposteriser.INSTANCE);
    }};

    @Before
    public void disableWidgets() {
        GWTMockUtilities.disarm();
    }

    @After
    public void reEnableWidgets() {
        GWTMockUtilities.restore();
    }

    @Test
    public void mock_a_button() {
        Button button = context.mock(Button.class);
    }
}

A (slightly) more realistic example

For example, imagine the scenario of creating a dialog which prompts the user for some text.  The Submit button should only be enabled when the user has actually entered something.

Here's a sample test for the Controller object, which will be given the full contents of the text field every time the user presses a key via the textChangedTo(String) method.

@RunWith(JMock.class)
public class ControllerTest {
    Mockery context = new JUnit4Mockery() {{
        // Enable CGLIB mocking of concrete classes
        setImposteriser(ClassImposteriser.INSTANCE);
    }};

    @Before
    public void disableWidgets() {
        GWTMockUtilities.disarm();
    }

    @After
    public void reEnableWidgets() {
        GWTMockUtilities.restore();
    }

    @Test
    public void enables_submit_button_if_text_entered() {
        final Button button = context.mock(Button.class);

        Controller sut = new Controller(button);

        context.checking(new Expectations() {{
            oneOf(button).setEnabled(true);
        }});
        sut.textChangedTo("Hello");
    }
    
    @Test
    public void disables_submit_button_if_text_cleared() {
        final Button button = context.mock(Button.class);

        Controller sut = new Controller(button);

        context.checking(new Expectations() {{
            oneOf(button).setEnabled(false);
        }});
        sut.textChangedTo("");
    }
}

Notice that we mock out our Button widget, notify the Controller that the text changed, and expect our button to be enabled or disabled appropriately.

Here's the code for the Controller which passes these tests:

public class Controller {
    private Button button;

    public Controller(Button button) {
        this.button = button;
    }

    /**
     * Callback from a form's text field onChange.
     *
     * @param newText The text entered
     */
    public void textChangedTo(String newText) {
        if (isBlank(newText))
            button.setEnabled(false);
        else
            button.setEnabled(true);
    }

    private boolean isBlank(String newText) {
        return "".equals(newText.trim());
    }
}

Design Implications

So when would you want to actually use GWTMockUtilities?  In general, I prefer to wrap the view in an interface and mock it out in my controller tests as I wrote about in this article.  However, if the controller only needs to communicate with one or two widgets as in the above example, you may want to skip creating the view interface directly and just pass in the widget directly.  For me, I'd consider this an exceptional case rather than the norm, but you should do what feels natural for your applications.

Intro to GWT article in October 2008 Better Software Magazine

My article "Google Web Toolkit: Your Shortcut to Ajax Web Applications" (download article) is now in print in the October 2008 issue of Better Software Magazine.  This is an introduction to GWT and will be followed in an upcoming issue by an article that demonstrates using GWT to build Ajax applications test-first.

Thank you to my reviewers for all your feedback: Steve Berczuk, Mushfeq Khan, Joe Leo, and Ariel Valentin.  Many thanks to Sumit Chandel of the GWT team and Brad Neuberg at Google for their technical feedback and support.  Thank you to Lee Copeland, Chuck Allison, Heather Shanholtzer and the team at Better Software Magazine for the opportunity and excellent technical editing.

Updated Oct 13, 2008: Added link to download a PDF copy of the article.

Testing JSON parsing using JavaScript Overlay Types in GWT 1.5

Bruce Johnson has posted a new entry on the Google Web Toolkit blog entitled Getting to really know GWT, Part 2: JavaScript Overlay Types. In the article, Bruce shows us how to use the new GWT 1.5 JavaScript overlay types to easily parse JSON into Java objects.

I previously wrote about Testing JSON Parsing with GWTTestCase using the original GWT JSON parsing libraries, so I'm posting a new example of testing JSON parsing using the new JavaScript Overlay Types.

Bruce's example parses some customer data into a Java Customer object. Here's the source from the article, with an additional "parse" factory method that parses a Customer object from a String containing JSON.

class Customer extends JavaScriptObject {

  // Overlay types always have protected, zero-arg ctors
  protected Customer() { }

  // Typically, methods on overlay types are JSNI
  public final native String getFirstName() /*-{ return this.FirstName; }-*/;
  public final native String getLastName() /*-{ return this.LastName; }-*/;

  // Note, though, that methods aren't required to be JSNI
  public final String getFullName() {
    return getFirstName() + " " + getLastName();
  }

  // Parse from JSON, based upon code from
  // http://sites.google.com/site/io/gwt-and-client-server-communication
  public static final native Customer fromJson(String input) /*-{
    return eval('(' + input + ')')
  }-*/;
}

Here's a simple test case demonstrating using an existing JavaScript object as well as parsing from JSON:

public class CustomerTest extends GWTTestCase {
  // Required by all GWTTestCases
  public String getModuleName() {
    return "JsniOverlays";
  }
 
  public void testUsesUnderlyingJavaScriptObject() {
  Customer customer = testCustomer();
    assertEquals("Jimmy", customer.getFirstName());
    assertEquals("Webber", customer.getLastName());
    assertEquals("Jimmy Webber", customer.getFullName());
  }

  // Sample test fixture
  private native Customer testCustomer() /*-{
    return { "FirstName" : "Jimmy", "LastName" : "Webber" }
  }-*/;
 
  public void testParsesFromJson() {
    Customer customer = Customer.fromJson("{ FirstName: \"Murphy\", LastName: \"Plankton\" }");
    assertEquals("Murphy", customer.getFirstName());
    assertEquals("Plankton", customer.getLastName());
  }
}

Notice that testUsesUnderlyingJavaScriptObject refers to a test fixture creation method written in JavaScript using JSNI. This test is similar to Bruce's example which coerces a JavaScript object into a Java object.

testParsesFromJson passes in a String containing JSON, which is what a remote service might do after receiving a response from a server. Notice how I'm not testing getFullName again in this test; it's already been verified in another test.

Testing Considerations

One challenge with using overlay types is that your objects contain JSNI -- which means that they fail to run in a plain JUnit TestCase. Instead, they must be run in a GWTTestCase, and that means a much slower test.

To solve this problem, I'm tempted to create an interface for the methods I need on my Customer object, then create a Test Double to use in my tests. This would encourage referring to the object by it's interface, which is a good thing.

Still, it seems a little annoying that I need to jump through some hoops so that my core domain object (Customer) doesn't accidentally call JavaScript in my tests. I'm not sure if this would be a real problem or not; experience will tell.

Congratulations to the GWT team on their hard work in making some great improvements to GWT 1.5!