Stay on Target

Improve Your iOS Frank Cucumber Acceptance Tests

Here are some tips I've learned to build more readable, flexible acceptance tests for iOS devices using Frank and Cucumber.  See the full article on the Cyrus Cylinder:

The Cyrus Cylinder: Improve Your iOS Frank Cucumber Acceptance Tests

Stories of Adopting Alternative JVM Languages in Java Projects in Better Software Magazine

Groovy, JRuby, Scala, and Clojure are some of the most widely used alternative Java Virtual Machine (JVM) languages today.  I spoke with several developers who adopted one of these languages in a Java-based project.  This article ("Alternative JVM Languages for Java Projects") tells their stories - which language they picked, why they picked it, and how they integrated it with their existing Java code base or infrastructure.

This article includes stories from the following people:

• Moss Collum: Gradually migrating code from Java to Groovy
• Kris Nuttycombe: Starting to use Scala to write tests for a Java project
• Najati Imam: Seamlessly deploying JRuby on Rails in an existing Java infrastructure
• Bob McWhirter: Combining some of the best parts of Java with Ruby language to build something new: TorqueBox
• Dan Chamberlain: Using Clojure to more naturally and succinctly model the business problem domain

 

Download PDF

 

This article appears in the September/October 2011 issue of Better Software Magazine.

Thanks to Heather Shanholtzer, Lee Copeland, Jean Middleton, and Cheryl Burke at Better Software Magazine for many things - the opportunity to write this article, top-notch editing, and valuable feedback.  Thanks to my friends Alex Baranosky, Moss Collum, Paul Infield-Harm, Joe Leo, Jason Reid and colleagues at Cyrus Innovation for their formative suggestions for the article.

Introducing Bane, a Test Harness for Server Connections

Bane is a tool that makes it easy to test your application in scenarios where any third-party server dependencies behave in unexpected ways.  This post explains why you might want to use Bane, how to get started, and the design motivations behind Bane. 

Problem: The World is Unsafe (For an Application)

If you are building an application, you may depend on third-party servers or other external sources for your data.  You may use any of the publicly available web services from companies like Google, Yahoo!, or Twitter.  Or perhaps you use web services in an internal business application to obtain data from authoritative sources.  Most of the time these services are reliable and work as you expect.

Yet however reliable these services claim to be, at some point they’ll behave in an unusual manner.  The servers may be busy and accept your connection but never respond, they might reply in an unexpected response format, or there might be network problems that require endless retransmits of the data - and that's just a few of the ways that things can go wrong.  It’s up to you to program defensively so that if the remote service fails, your application doesn’t fail as well.  Michael Nygard’s excellent book “Release It!” is full of patterns and anti-patterns for building applications that can survive in these situations; I can’t recommend it highly enough.

To make sure your application isn’t susceptible to external service failures, you should test your application under various hostile conditions.  However, getting your external services to behave in the desired fashion can be a real challenge.  Instead, Nygard recommends building a tool that can simulate various bad behaviors, as described in the chapter “Test Harness.”  

Introducing Bane

Bane is an open source implementation of the test harness described in “Release It!”  Bane is written in Ruby and is available as a Ruby gem.  To install it, you’ll need a working installation of Ruby.  Install Bane with “gem install bane” which makes an executable “bane” available from the command line.  To make sure Bane is installed correctly, type “bane” with no arguments from the command line.  You should see a usage message like this:

$ bane

Usage: bane port_number <servers>

All behaviors:

 - CloseAfterPause

 - CloseImmediately

 - DelugeResponse

 - DelugeResponseForEachLine

 - FixedResponse

 - FixedResponseForEachLine

 - HttpRefuseAllCredentials

 - NeverRespond

 - NewlineResponse

 - NewlineResponseForEachLine

 - RandomResponse

 - RandomResponseForEachLine

 - SlowResponse

 - SlowResponseForEachLine

Bane requires you specify a port number on which to start listening and a behavior to use to respond on that port.  A behavior is one of the several nefarious ways in which Bane can respond, such as never responding, sending random data, closing the connection immediately, etc.  Behaviors are named in CamelCase (e.g. NeverRespond, NewlineResponseForEachLine) because they correspond to Ruby classes.

A Usage Example

Imagine you’ve built an application to track financial investment portfolios.  All stock quotes are obtained via a third-party web service.  You’d like to see how your application behaves when the web service accepts your request but never sends a byte of data in response.  Let’s pretend that your stock quote service listens on port 8080.

Start up the NeverRespond behavior on port 8080, which accepts a connection but then never sends a response.

$ bane 8080 NeverRespond

[Tue Sep 28 10:49:25 2010] NeverRespond 127.0.0.1:8080 start

Point your application at port 8080 and see how your application behaves.   Notice that Bane logs the incoming connections:

[Tue Sep 28 10:50:05 2010] NeverRespond 127.0.0.1:8080 client:61459 127.0.0.1<127.0.0.1> connect

You may need to add timeouts or other failure strategies to keep your application functioning under this scenario.  Consult “Release It!” for more information and other alternatives.

Bane comes with several bad behaviors you can use to test out your application in the manner described above.  See the README and wiki pages for more information.

The Design Motivations Behind Bane

Before I created Bane, I used to write little one-off scripts to create these badly behaving servers.  They aren’t much work; Ruby’s standard library includes a GServer class that makes it easy to quickly create a multithreaded TCP/IP server.  In fact, if you look at the source code, you’ll see that Bane uses GServer internally.  So why not just create these little servers each time by hand?

It turns out that each time I had to create one, I’d take about five minutes to bring up the RDoc and get all the parameters right.  So I created Bane to have all these common bad behaviors available in just a few seconds.  Right now Bane only supports two response protocols: send a response immediately upon connection and respond after each newline sent by the client.  I haven’t needed other response protocols on my projects yet, but it should be simple to create variations that respond after every n bytes, etc.  

However, if it turns out there needs to be some advanced protocol (e.g. respond after certain keywords, preserve state, etc.) then you may be better off spending the five to ten minutes to write your own server in Ruby using GServer or Sinatra, or some other language of your choice.  My goal is to keep Bane easy to use for common cases, to get a bad behavior up and running in a few seconds rather than five minutes.

Manual or Automated?

I use Bane primarily for manual exploratory tests, and write unit tests around my applications to create fast, repeatable automated tests.   Typically I use Bane in my consulting work to test out an existing application and see how it behaves, find the failure modes, write automated unit tests to expose the behavior, and then fix the problem by applying the patterns described in “Release It!”  Many of these unit tests use test doubles to throw exceptions that would be created by the I/O library in use.

That said, Bane provides an API through the Bane::Launcher class for automation in Ruby.  Take a look at Bane’s integration tests and the examples folder in the source distribution for an example of this API in use.

How to Contribute

Bane is open source and I welcome your feedback or contributions.  Here are a few ways you can contribute:

Provide Feedback: Please take Bane for a spin on your project and let me know if it helps, where it didn’t, and any suggestions for enhancements.

Write new behaviors: Fork Bane on GitHub and contribute useful bad behaviors or any other items described in “Release It!” which are not yet implemented (see the README for a list).  In particular, Bane does not yet support any of the low-level protocol errors in “Release It!”, such as the remote end endlessly sending TCP/IP resend packets.  To my knowledge, this low-level behavior is not possible with Ruby’s I/O library and requires a C extension.

Code Reviews: If you know Ruby and have some ideas about how to improve the code, please fork Bane and suggest some changes.

For more information

• Bane at GitHub
• Bane’s FAQ
• E-mail me at dan at danielwellman dot com

Flexible Argument Matching with Mocha

If your Ruby unit tests are failing because you change the signature of a method and all your mock expectations fail, you might have a case of Overspecified Software.  To fix this problem, make your test require only the parameters that are significant.  To quote Gerard Meszaros, "If a value is not important to a test, it's important that it not be in the test."

For example, imagine you had a test which took an options Hash as a parameter.  You might have an expectation like this:

processor.expects(:order).with(:payment => :credit_card, :delivery => :express, :group => :separate)

Unless your test is verifying that all of these parameters are set correctly, it might be better to change your tests so that each test verifies only the parameters that are important.  In this post, I'll describe how to do this with Mocha, but Flex Mock also provides similar features.

Mocha provides a flexible syntax to match parameters of expectations.  For example, if you wanted to check that a given key was present, and ignore the rest of the arguments, you could say:

processor.expects(:order).with(has_key(:payment))

If you are interested in both the key and value, you could use has_entry, or the alternate has_entries:

processor.expects(:order).with(has_entry(:payment => :credit_card))

If you just wanted to make sure something was NOT present, you can use the Not operation to invert the condition:

processor.expects(:order).with(Not(has_key(:emergency_order)))

Or specify any number of conditions using the any_of condition.  

processor.expects(:order).with(any_of(includes(:delivery), has_key(:shipping)))

Mocha provides many more variations, see the Mocha::ParameterMatchers RDoc for more possibilities.

By combining these parameter matchers, you can make tests that are more expressive about the arguments they care about and avoid unrelated test failures when you make a harmless change to your code.

Text File Processing with Ruby from the Command Line

My colleague David showed me some useful ways to use Ruby for simple text processing from the command line.  I'm blogging about it here to remind myself how it works.  (Thanks for the tip, David!)

The Ruby interpreter can be run in a mode which executes a one-line script against a text file.  You can choose to print the results of the script to standard out or to replace the file.  In this way, Ruby becomes a text processing tool like awk, sed, and grep.

For example, if you've got a file with a list of recipe names (in "recipe_index.txt") and you'd like it to contain only those recipes that have 'banana' in the title, you'd do something like this:

ruby -ni.bak -e 'print if /banana/i' recipe_index.txt

-n feeds every line of input_file into your script, simulating sed's text file processing

-i applies changes to recipe_index.txt in-place, otherwise prints the results to standard out.  If you specify any text after -i, such as "-i.bak", Ruby will first make a copy of your input file and append the extension.  In this example, you'd end up with recipe_index.txt.bak

-e specifies a one-line Ruby script to execute, which prints the line only if it matches the case-insensitive regular express 'banana'

You can also use -p if you want to print out every line of the input file as well as your script's result.  For example, if you wanted to prepend the number of characters in each line to the beginning of the line, you'd say:

ruby -pe 'print $_.size' input_file"

Note the magic variable $_ (dollar sign underscore) which holds the contents of the last line matched by gets().

Ariel's new blog series on applying Ruby techniques in Java

My colleague Ariel at Cyrus Innovation has started a new series of blog posts entitled "How Ruby made me into a better Java developer".  In this series, he's going to be talking about he and his team applied many of the techniques from Ruby and Rails to a Java web application.

I've seen some of the changes the team has made and I was impressed; I especially liked the way they built action tests that could check the view (just like Rails functional tests).

I'm looking forward to reading the whole series!

Writing Rails integration tests with Webrat

I frequently use Selenium to write end-to-end integration tests for web applications, but these are usually the most expensive tests to build and maintain.  On the positive side,

• they drive a real web browser so it's how a real user will interact with the site
• JavaScript is enabled so you can test AJAX sites
• they provide an automated way to verify functionality across several browsers

On the other hand:

• they take a lot of time and effort to write and maintain
• they can be brittle and require a lot of thinking about timing, especially for AJAX sites
• they are slow to run, and can be painful to watch over and over when diagnosing failures

Rails provides functional tests which are used to test a single controller, but don't work well for simulating multiple controller tests.  Since functional tests are usually for a single request, you end up explicitly shoehorning session data into your requests.  This is fine for a unit test, but can lead to errors when the tests are inserting the wrong session data. 

Rails Integration Tests allow you to make requests across multiple controllers, and even let you create multiple independent requests to simulate multiple users accessing your application.  To write integration tests, you write in terms of HTTP requests like so:

put_via_redirect "/machines/update" ,
  :id => 45,
  :machine => { :name => "Lawn Mower" , :category_id => 2 }

The difficulty with these tests is that they again rely on the programmer to write tests that pass the correct list of parameters.  The response from a request can be examined by the tests, but there's nothing stopping you from posting a request with input parameters that aren't possible from a real form.  If you're writing tests to ensure your site works when accessed by a web browser, then you need to put in extra work just to make sure the tests behave like a browser. 

In some cases, this is a good thing, as you may want to test that your site is protected against attacks.  Your site might have non-browser clients and expose its functionality as web services, in which case the tests more closely resemble a real client's usage.

But is there some way to get the real browser behavior of Selenium but the speed of acceptance tests?

Acceptance tests with webrat

Webrat, by Bryan Helmkamp, is a plugin for Rails which lets you write integration tests using a domain-specific language for describing a user's interaction with a web browser.  Using webrat, an integration test might look something like this:

class CreatingMachinesTest < ActionController::IntegrationTest
 fixtures :users, :repairs, :outages, :machines, :inspections, :images, :categories

 def test_create_a_new_machine
  @user = users(:frank)

  # Navigate to the start page 
  visits "sessions/new"
  assert_response :success 

  # Log in to the site by filling out user credentials
  fills_in "login", :with => @user.login
  fills_in "password", :with => "test"
  clicks_button "Log in"

  # This is still an integration test, so we have access to the
  # session state if we need it
  assert_equal @user.id, session[:user], "Should have been logged in"

  clicks_link "New machine"
  assert_response :success 

  fills_in "Manufacturer", :with => "SuperSaws"
  fills_in "Model number", :with => "X-1000"
  fills_in "Serial number", :with => "10-20039" 
  selects "Chainsaws", :from => "Category"

  clicks_button "Create"
  assert_response :success 

  # Check the that the machine was actually created
  assert_not_nil Machine.find_by_manufacturer_and_model_number("SuperSaws", "X-1000") 
 end
end

Under the hood, webrat uses Hpricot to parse HTML and build requests using the real web forms.  You write tests by simulating filling in form elements and clicking the submit button.  Webrat knows the type of request (GET, POST, etc.) and the target action URL since it's processing the real form HTML.   To find the form items, webrat uses some sensible defaults like the name attribute or corresponding label for a text field, and the link text for clicking links.  These methods like "fills_in", "clicks" and "selects" are also verifying that the form elements actually exist in the response page.  This allows you to test what a user would see without using a real web browser, and that means faster integration tests.

Rails only (for now)

webrat is a plugin used to help test Rails applications.  It's not a standalone Ruby object library that controls a text-only browser (remember lynx?).  If you look at the source code for the webrat plugin, you'll see it's clearly part of the Rails integration test infrastructure:

module ActionController
 module Integration
 
  class Session
   def visits(path)
    request_page(:get, path)
   end
   ...

Notice how it's tied in to ActionController::Integration::Session, which is part of the integration testing facilities for Rails.  Don't try to use webrat to test your Java based Struts application; it only tests in-process Rails applications.  There are experimental changes in the webrat git repository which support testing Merb web applications, another Ruby framework.

Not really a web browser

Since webrat is just a helper for building Rails integration tests, it suffers the same problems.  For example, a real web browser will pick the first item in select list/combo box by default, which means that submitting a form will always submit some value for these fields.  webrat will not do this.  If you forget to set your select list form values, their values will not be posted to the server, which is almost always an error.  I hit this very same problem in my first webrat tests.

Diagnosing failures in webrat can also be a bit difficult, since you have no visual feedback of what your test is doing.  This is the same problem that integration and functional tests have; it's not webrat's fault.  You may want to put more assertions in your tests to ensure you're actually looking at the page you think you are (which you'd do in a Selenium test anyways).

No JavaScript

webrat is not a real web browser, so it doesn't run JavaScript.  If your site is AJAX based, you're probably better off sticking with something that uses a real browser like Selenium or Watir.

The Year of the (web)rat

I've only just started playing with webrat, but I'd consider using it for a project that didn't use a lot of JavaScript.   Webrat allows you to test your applications using the HTML forms instead of hand-coding HTTP requests, which can prevent a lot of annoying bugs in tests. There is a lot of material on the web about using webrat with RSpec for customer story tests, but for my purposes I didn't need the extra layer of RSpec.

webrat is used on big projects and under active development, so I'm excited to track its progress.