I frequently use Selenium to write end-to-end integration tests for web applications, but these are usually the most expensive tests to build and maintain. On the positive side,
- they drive a real web browser so it's how a real user will interact with the site
- JavaScript is enabled so you can test AJAX sites
- they provide an automated way to verify functionality across several browsers
On the other hand:
- they take a lot of time and effort to write and maintain
- they can be brittle and require a lot of thinking about timing, especially for AJAX sites
- they are slow to run, and can be painful to watch over and over when diagnosing failures
Rails provides functional tests which are used to test a single controller, but don't work well for simulating multiple controller tests. Since functional tests are usually for a single request, you end up explicitly shoehorning session data into your requests. This is fine for a unit test, but can lead to errors when the tests are inserting the wrong session data.
Rails Integration Tests allow you to make requests across multiple controllers, and even let you create multiple independent requests to simulate multiple users accessing your application. To write integration tests, you write in terms of HTTP requests like so:
put_via_redirect "/machines/update" ,
:id => 45,
:machine => { :name => "Lawn Mower" , :category_id => 2 }
The difficulty with these tests is that they again rely on the programmer to write tests that pass the correct list of parameters. The response from a request can be examined by the tests, but there's nothing stopping you from posting a request with input parameters that aren't possible from a real form. If you're writing tests to ensure your site works when accessed by a
web browser, then you need to put in extra work just to make sure the tests behave like a browser.
In some cases, this is a good thing, as you may want to test that your site is protected against attacks. Your site might have non-browser clients and expose its functionality as web services, in which case the tests more closely resemble a real client's usage.
But is there some way to get the real browser behavior of Selenium but the speed of acceptance tests?
Acceptance tests with webrat
Webrat, by Bryan Helmkamp, is a plugin for Rails which lets you write integration tests using a domain-specific language for describing a user's interaction with a web browser. Using webrat, an integration test might look something like this:
class CreatingMachinesTest < ActionController::IntegrationTest
fixtures :users, :repairs, :outages, :machines, :inspections, :images, :categories
def test_create_a_new_machine
@user = users(:frank)
# Navigate to the start page
visits "sessions/new"
assert_response :success
# Log in to the site by filling out user credentials
fills_in "login", :with => @user.login
fills_in "password", :with => "test"
clicks_button "Log in"
# This is still an integration test, so we have access to the
# session state if we need it
assert_equal @user.id, session[:user], "Should have been logged in"
clicks_link "New machine"
assert_response :success
fills_in "Manufacturer", :with => "SuperSaws"
fills_in "Model number", :with => "X-1000"
fills_in "Serial number", :with => "10-20039"
selects "Chainsaws", :from => "Category"
clicks_button "Create"
assert_response :success
# Check the that the machine was actually created
assert_not_nil Machine.find_by_manufacturer_and_model_number("SuperSaws", "X-1000")
end
end
Under the hood, webrat uses Hpricot to parse HTML and build requests using the real web forms. You write tests by simulating filling in form elements and clicking the submit button. Webrat knows the type of request (GET, POST, etc.) and the target action URL since it's processing the real form HTML. To find the form items, webrat uses some sensible defaults like the name attribute or corresponding label for a text field, and the
link text for clicking links. These methods like "fills_in", "clicks" and "selects" are also verifying that the form elements actually exist in the response page. This allows you to test what a user would see without using a real web browser, and that means faster integration tests.
Rails only (for now)
webrat is a plugin used to help test Rails applications. It's not a standalone Ruby object library that controls a text-only browser (remember lynx?). If you look at the source code for the webrat plugin, you'll see it's clearly part of the Rails integration test infrastructure:
module ActionController
module Integration
class Session
def visits(path)
request_page(:get, path)
end
...
Notice how it's tied in to ActionController::Integration::Session, which is part of the integration testing facilities for Rails. Don't try to use webrat to test your Java based Struts application; it only tests in-process Rails applications. There are experimental changes in the webrat git repository which support testing Merb web applications, another Ruby framework.
Not really a web browser
Since webrat is just a helper for building Rails integration tests, it suffers the same problems. For example, a real web browser will pick the first item in select list/combo box by default, which means that submitting a form will always submit some value for these fields. webrat will not do this. If you forget to set your select list form values, their values will not be posted to the server, which is almost always an error. I hit this very same problem in my first webrat tests.
Diagnosing failures in webrat can also be a bit difficult, since you have no visual feedback of what your test is doing. This is the same problem that integration and functional tests have; it's not webrat's fault. You may want to put more assertions in your tests to ensure you're actually looking at the page you think you are (which you'd do in a Selenium test anyways).
No JavaScript
webrat is not a real web browser, so it doesn't run JavaScript. If your site is AJAX based, you're probably better off sticking with something that uses a real browser like Selenium or Watir.
The Year of the (web)rat
I've only just started playing with webrat, but I'd consider using it for a project that didn't use a lot of JavaScript. Webrat allows you to test your applications using the HTML forms instead of hand-coding HTTP requests, which can prevent a lot of annoying bugs in tests. There is a lot of material on the web about using webrat with RSpec for customer story tests, but for my purposes I didn't need the extra layer of RSpec.
webrat is used on big projects and under active development, so I'm excited to track its progress.