<?xml version="1.0" encoding="UTF-8"?>
<org.eclipse.epf.uma:ContentDescription xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:org.eclipse.epf.uma="http://www.eclipse.org/epf/uma/1.0.6/uma.ecore" xmlns:epf="http://www.eclipse.org/epf" epf:version="1.5.1" xmlns:rmc="http://www.ibm.com/rmc" rmc:version="7.5.1" xmi:id="_s60KoMM3EdmSIPI87WLu3g" name="test_suite,_0aDz0MlgEdmt3adZL5Dmdw" guid="_s60KoMM3EdmSIPI87WLu3g" changeDate="2006-09-29T09:48:57.000-0700" version="1.0.0">
  <mainDescription>&lt;h3>&#xD;
    Introduction&#xD;
&lt;/h3>&#xD;
&lt;p>&#xD;
    The test suite provides a means of managing the complexity of the test implementation. Many system test efforts fail&#xD;
    because the team gets lost in the minutia of all of the detailed tests, and subsequently loses control of the test&#xD;
    effort. Similar to UML packages, test suites provide a hierarchy of encapsulating containers to help manage the test&#xD;
    implementation. They provide a means of managing the strategic aspects of the test effort by collecting tests together&#xD;
    in related groups that can be planned, managed, and assessed in a meaningful way.&#xD;
&lt;/p>&#xD;
&lt;p>&#xD;
    The test suite represents a container for organizing arbitrary collections of related test scripts. This may be&#xD;
    realized (implemented) as one or more automated regression test suites, but the test suite can also be a work plan for&#xD;
    the implementation of a group of related manual test scripts. Note also that test suites can be nested hierarchically,&#xD;
    therefore one test suite may be enclosed within another.&#xD;
&lt;/p>&#xD;
&lt;p>&#xD;
    Sometimes these groups of tests will relate directly to a subsystem or other system design element, but other times&#xD;
    they'll relate directly to things such as quality dimensions, core &quot;mission critical&quot; functions, requirements&#xD;
    compliance, standards adherence, and many others concerns that are organized based on requirements and therefore cut&#xD;
    across the internal system elements.&#xD;
&lt;/p>&#xD;
&lt;p>&#xD;
    Consider creating test suites that arrange the available test scripts, in addition to other test suites, in many&#xD;
    different combinations: the more variations you have, the more you'll increase coverage and the potential for finding&#xD;
    errors. Give thought to a variety of test suites that will cover the breadth and depth of the target test items.&#xD;
    Remember the corresponding implication that a single test script (or test suite) may appear in many different test&#xD;
    suites.&#xD;
&lt;/p>&#xD;
&lt;p>&#xD;
    Some test automation tools provide the ability to automatically generate or assemble test suites. There are also&#xD;
    implementation techniques that enable automated test suites to dynamically select all or part of their component test&#xD;
    scripts for each test cycle run.&#xD;
&lt;/p>&#xD;
&lt;p>&#xD;
    Test suites can help you maintain your test assets and impose a level of organization that facilitates the entire&#xD;
    testing effort.&amp;nbsp; Like physical objects, tests can break. It's not that they wear down, it's that something changed&#xD;
    in their environment. Perhaps they've been ported to a new operating system. Or, more likely, the code they exercise&#xD;
    has changed in a way that correctly causes the test to fail. Suppose you're working on version 2.0 of an e-banking&#xD;
    application. In version 1.0, this method was used to log in:&#xD;
&lt;/p>&#xD;
&lt;p class=&quot;codeSample&quot;>&#xD;
    public boolean login (String username);&#xD;
&lt;/p>&#xD;
&lt;p>&#xD;
    In version 2.0, the marketing department has realized that password protection might be a good idea. So the method is&#xD;
    changed to this:&#xD;
&lt;/p>&#xD;
&lt;p class=&quot;codeSample&quot;>&#xD;
    public boolean login (String username, String password);&#xD;
&lt;/p>&#xD;
&lt;p>&#xD;
    Any test that uses the first form of the login will fail. It won't even compile. In this example you could find that,&#xD;
    not many useful tests can be written that don't use login. You might be faced with hundreds or thousands of failing&#xD;
    tests.&#xD;
&lt;/p>&#xD;
&lt;p>&#xD;
    These tests can be fixed by using a global search-and-replace tool that finds every instance of login(something) and&#xD;
    replaces it with login(something, &quot;dummy password&quot;). Then arrange for all the testing accounts to use that password,&#xD;
    and you're on your way.&#xD;
&lt;/p>&#xD;
&lt;p>&#xD;
    Then, when marketing decides that passwords should not be allowed to contain spaces, you get to do it all over again.&#xD;
&lt;/p>&#xD;
&lt;p>&#xD;
    This kind of thing is a wasteful burden, especially when, as is often the case, the test changes aren't so easily made.&#xD;
    There is a better way.&#xD;
&lt;/p>&#xD;
&lt;p>&#xD;
    Suppose that the test scripts originally did not call the product's login method. Rather, they called a library method&#xD;
    that does whatever it takes to get the test logged in and ready to proceed. Initially, that method might look like&#xD;
    this:&#xD;
&lt;/p>&#xD;
&lt;p class=&quot;codeSample&quot;>&#xD;
    public boolean testLogin (String username) {&lt;br />&#xD;
    &amp;nbsp; return product.login(username);&lt;br />&#xD;
    }&#xD;
&lt;/p>&#xD;
&lt;p>&#xD;
    When the version 2.0 change happens, the utility library is changed to match:&#xD;
&lt;/p>&#xD;
&lt;p class=&quot;codeSample&quot;>&#xD;
    public Boolean testLogin (String username) {&lt;br />&#xD;
    &amp;nbsp; return product.login(username, &quot;dummy password&quot;);&lt;br />&#xD;
    }&#xD;
&lt;/p>&#xD;
&lt;p>&#xD;
    Instead of a changing a thousand tests, you change one method.&#xD;
&lt;/p>&#xD;
&lt;p>&#xD;
    Ideally, all the needed library methods would be available at the beginning of the testing effort. In practice, they&#xD;
    can't all be anticipated-you might not realize you need a testLogin utility method until the first time the product&#xD;
    login changes. So test utility methods are often &quot;factored out&quot; of existing tests as needed. It is very important that&#xD;
    you perform this ongoing test repair, even under schedule pressure. If you do not, you will waste much time dealing&#xD;
    with an ugly and un-maintainable test suite. You might well find yourself throwing it away, or being unable to write&#xD;
    the needed numbers of new tests because all your available testing time is spent maintaining old ones.&#xD;
&lt;/p>&#xD;
&lt;p>&#xD;
    Note: the tests of the product's login method will still call it directly. If its behavior changes, some or all of&#xD;
    those tests will need to be updated. (If none of the login tests fail when its behavior changes, they're probably not&#xD;
    very good at detecting defects.)&#xD;
&lt;/p>&#xD;
&lt;h3>&#xD;
    Abstraction helps manage complexity&#xD;
&lt;/h3>&#xD;
&lt;p>&#xD;
    The previous example showed how tests can abstract away from the concrete application. Most likely you can do&#xD;
    considerably more abstraction. You might find that a number of tests begin with a common sequence of method calls: they&#xD;
    log in, set up some state, and navigate to the part of the application you're testing. Only then does each test do&#xD;
    something different. All this setup could-and should-be contained in a single method with an evocative name such as&#xD;
    readyAccountForWireTransfer. By doing that, you're saving considerable time when new tests of a particular type are&#xD;
    written, and you're also making the intent of each test much more understandable.&#xD;
&lt;/p>&#xD;
&lt;p>&#xD;
    Understandable tests are important. A common problem with old test suites is that no one knows what the tests are doing&#xD;
    or why. When they break, the tendency is to fix them in the simplest possible way. That often results in tests that are&#xD;
    weaker at finding defects. They no longer test what they were originally intended to test.&#xD;
&lt;/p>&#xD;
&lt;h3>&#xD;
    Throwing away tests&#xD;
&lt;/h3>&#xD;
&lt;p>&#xD;
    Even with utility libraries, a test might periodically be broken by behavior changes that have nothing to do with what&#xD;
    it checks. Fixing the test doesn't stand much of a chance of finding a defect due to the change; it's something you do&#xD;
    to preserve the chance of the test finding some other defect someday. But the cost of such a series of fixes might&#xD;
    exceed the value of the tests hypothetically finding a defect. It might be better to simply throw the test away and&#xD;
    devote the effort to creating new tests with greater value.&#xD;
&lt;/p>&#xD;
&lt;p>&#xD;
    Most people resist the notion of throwing away a test, at least until they're so overwhelmed by the maintenance burden&#xD;
    that they throw all the tests away. It is better to make the decision carefully and continuously, test by test, asking:&#xD;
&lt;/p>&#xD;
&lt;ul>&#xD;
    &lt;li>&#xD;
        How much work will it be to fix this test well, perhaps adding to the utility library?&#xD;
    &lt;/li>&#xD;
    &lt;li>&#xD;
        How else might the time be used?&#xD;
    &lt;/li>&#xD;
    &lt;li>&#xD;
        How likely is it that the test will find serious defects in the future? What's been the track record of it and&#xD;
        related tests?&#xD;
    &lt;/li>&#xD;
    &lt;li>&#xD;
        How long will it be before the test breaks again?&#xD;
    &lt;/li>&#xD;
&lt;/ul>&#xD;
&lt;p>&#xD;
    The answers to these questions will be rough estimates or even guesses. But asking them will yield better results than&#xD;
    simply having a policy of fixing all tests.&#xD;
&lt;/p>&#xD;
&lt;p>&#xD;
    Another reason to throw away tests is that they are now redundant. For example, early in development, there might be a&#xD;
    multitude of simple tests of basic parse-tree construction methods (the LessOp constructor and the like). Later, during&#xD;
    the writing of the parser, there will be a number of parser tests. Since the parser uses the construction methods, the&#xD;
    parser tests will also indirectly test them. As code changes break the construction tests, it's reasonable to discard&#xD;
    some of them as being redundant. Of course, any new or changed construction behavior will need new tests. They might be&#xD;
    implemented directly (if they're hard to test thoroughly through the parser) or indirectly (if tests through the parser&#xD;
    are adequate and more maintainable).&#xD;
&lt;/p></mainDescription>
</org.eclipse.epf.uma:ContentDescription>
