Testing Toolbox

Andrew MacNeill, September 2004

 

You always test every aspect of their application--- right ?

 

Every application needs to get tested either it gets done before the product is released, which is ideal or your users will do it for you, which usually turns out to be a big mistake. Even if you don't formally recognize testing in your project development cycle, you are still, in fact, testing from day one. Here are some tools and techniques that can help ease the pain of testing your application and adding it into your development process. Along the way, I'll try to answer some immediate objections to formalized testing.

 

The ideal software project plan incorporates testing right from the start, involving users or non-programmers, or at the very least, programmers not involved with the project, to provide an outside view of the application. Involving testers from the very beginning allows a project to move forward smoother, if only for the reason that they can understand exactly what is being tested and why it's important. Look at even a simplistic project timeline (see figure 1) and the opportunity for where testing begins is almost immediate.

 

When Do I Begin Testing? If you consider the initial proposal as the project start, testing starts when the user reads (and accepts) your understanding of the project.

 

Table 1 shows common types of tests and where they may typically start in the project process. One type of testing may stop at the end of a particular phase. For example, one would hope that requirements testing would be completed before the beta tests. If not, it's unlikely the acceptance test will ever be met.

 

Phase

Type of Test

Description

 

Ad-Hoc / Exploratory

Informal testing that is done at various times.

1. Design

Requirements

Tests that ensure a designed functionality is known, expected output is known and provide a way to verify it.

1.2 Design/Development

Usability

Tests that verify the user-friendliness of an application or component.

2. Design/Development

Unit

The process of testing a specific piece of code.

3. Development

Coverage

Testing how much of the code in an application is actually being executed.

3. Development

Functional / System

Tests that verify the functionality of an application or component based on the documented functionality.

3. Development

Integration

Testing combined parts of an application to determine if they function together correctly.

3.4 Development

Alpha

Done when development is nearing its end, core functionality is defined and requirements

4. Development

Smoke

Quick tests to determine if the software is working enough to let other people test it.

4. Testing

Beta

Testing when development is essentially complete and problems need to be found before final release.

4. Testing

Compatibility

Tests how the application works in a particular environment (hardware, software and network).

4. Testing

Install/Uninstall

Tests how an application installs and uninstalls.

4. Testing

Load/Performance/Stress

Tests that put the application through a maximum use process, by repeating frequent tasks or using the application with large amounts of data.

4. Testing

Recovery

Tests how well the application recovers in the case of a crash.

4. Testing

Regression

Re-testing of an application after modifications have been made. This ensures that nothing was broken by making changes.

5. Implementation

Acceptance

Final testing typically done by the user to confirm overall functionality of the application.

What kind of testing do you do? Many tests overlap into each other, both by definition and by phase.

 

If the person who will be doing the majority of your testing is the end-user (or one of the end users), it's important for them to realize that they are part of the project team, not simply the consumer at the end of the road. By involving them at the beginning of the project, when requirements and features are first identified, they are testing the assumptions that will guide the application to its final destination.

 

Objection: But My Project Doesn't Allow For All These Tests

 

Many programmers have welcomed the mantra of Extreme Programming, a series of practices that may seem, to some, to fly in the face of standard project processes. How can you do alpha testing when functionality is constantly being refactored and features being added? But in fact, Extreme Programming combines this type of testing into virtually every aspect of its process. If you are talking with the customer, then they are actively being involved in ensuring that requirements are being met (the Requirements test). If you are reviewing code with your partner, usability issues as well as coverage tests are likely being performed even without formally recognizing it.

 

The great thing about tests is that there are only two possible outcomes: pass or fail; and every test, from requirements to acceptance, must have some mechanism for achieving one or the other. Performance and load testing is a great example of where sometimes murky ideas are brought into play. Is it adequate for an application to take 2 hours to run a report? That is a requirement that must be answered before you can even consider coming up with a performance test.

 

The tests noted above could be broken into two types: ones that involve user input and ones that don't. Certainly all tests require someone write them but do they need to have someone who is there during the testing? The Requirements and Acceptance Tests require that an actual person sign off on it. Usability and code reviews require that individuals react to a piece of code or interface component. However, other tests such as regression or performance are more automated in nature, only requiring someone at the end to review the results. This is typically where many project teams start turning away from formalized testing: the amount of work required to do these automated tests. Thankfully, Visual FoxPro provides a number of tools to help us with many of these aspects.

 

FoxUnit, as discussed last month, covers the Unit Testing aspect and how it can be automated. Coverage testing can be done using the Coverage Profiler. This tool is often used more for seeing how long it takes each line of code to execute in an application. But by running an entire application with Coverage turned on, reveals what pieces of your code are actively being used. With the concept of "everything written must be tested", figure 2 looks at the Coverage tool analysis.

 

 

Everything must be tested. The coverage tool shows that while I thought I had tested my entire program, I only really hit about 70% of it, leaving 30% untested.

 

The Coverage Profiler is very easy to use. Turn it on in the Debugger or just set SET COVERAGE TO xxxx.LOG in the Command window and run your application. Run through as much of it as possible. While it's running, the coverage log file is being created. Be warned though: the Log files will be huge. For the application in figure 2 that ran less than a minute, the size of the log file was just over 3 megabtypes (for only 587 lines)

 

Realistically, to cover an entire application, you need a way to automatically run the functions found in your application. You can do it yourself or by using automated testing tools. Automated testing tools can also be used for other types of tests as well, such as regression and load testing.

 

Automated Testing

 

Automated testing tools typically record your actions and then allow you to replay them.

There are a number of tools available for automated testing, including Vermont's HighTest Plus (http://www.vtsoft.com) and IBM's TeamTest, formerly RationalTeamTest. The biggest problem with most commercial tools is that they don't natively support Visual FoxPro. This means that you can't integrate FoxPro commands into the scripts and will likely need to learn their scripting language. Yikes! Why should you have to learn a new scripting language (regardless how easy) just to test?

 

Starting with VFP 7, the Fox team included a test harness tool which was built using Active Accessibility technology. The Test harness tool is found in the Tools\Test folder in your VFP root directory. You can run it at any time by typing:

 

DO HOME()+"TOOLS\TEST\AATEST"

 

The toolbar includes a VCR button interface where the Red button starts recording a script (see figure 3). By default, these scripts are stored in the AASCRIPTS table. Once recorded, a script may be played back either by itself or with other scripts. You can also modify the script and add your own directives.

 

Record and Play. Like a VCR, the VFP Test Harness tool lets you record, pause and reply test scripts.

 

The entire test harness comes with source code so you can easily play around with it to see how it was built. The Test Log tab shows all of the playbacks with a Pass/Fail result. A test is said to have failed if it is unable to complete the recorded events. The Test Log also shows the duration of the test and how much memory was used from when the test was first started.

 

The Test harness script files can take some getting used to reading. A sample script is shown below

 

KeyPress 00000100 1 57 32 0 customer.vcr1.cmdnext 1.57200F

KeyPress 00000100 1 57 32 0 customer.vcr1.cmdnext 0.59100F

KeyPress 00000100 1 57 32 0 customer.vcr1.cmdnext 0.56100F

 

The Event or action that is taken is listed with the object that was hit as well as the position.. Hit the Edit script button and you can add your own directives into the script. Enter *<PAUSE> to pause the playback. When a script is paused, you can insert new recordings or an existing script. Table 2 shows other script directives. The Harness comes with some default tests for testing the wizards but these can be cleared out by zapping the AASCRIPTS table.

 

Directive

Usage

*<PAUSE>

Pauses execution of a script.

*COMMAND <command>

Executes a single VFP command

*DO <program>

Executes a FoxPro program

*=<function>

Runs a function

*IF <expression>
*ELSE

*ENDIF

Executes code after the IF provided the <expression> is true.

Table 2 Scripting Tests. The VFP Test Harness lets you call FoxPro commands from within the script.

 

As with many recording tools, the Test Harness doesn't always respond well to every type of application event, including drag and drop, or actions that require very specific mouse movements (such as resizing). But it does work for basic playback, often a huge time-saver when testing applications. It also works primarily with its own script table. If you have multiple applications to test, you might find it easier to backup and restore AASCRIPTS files as you work. A huge benefit is that the Harness is written in Visual FoxPro so you don't have to learn a new language to use it.

 

If you have used the Test Harness and find it lacking, there is another choice. FoxRunner from CAL (Computer Aided Logistics, http://www.cal.de) is a tool written in FoxPro that tracks activity in Visual FoxPro much like the Test Harness. Perhaps one of the greatest benefits of using FoxRunner is that it provides a place for you to document your test cases, the Test Case Manager (see figure 4). FoxRunner lets you group your tests into suites so you can document other aspects about it, all within your development environment.

 

Figure 4 Keeping Track of My Tests. FoxRunner's Test Manager is a great way of grouping your projects and tests, all within FoxPro.

 

FoxRunner also has the ability to create and restore data snapshots (even within scripts), generate test data and print reports based on your test results. The scripting language is far more enhanced than VFP's Test Harness, using more VFP-like commands and including features for automatic repeating (useful for simulating data entry) and running scripts from within your own application (such as to demonstrate features to users).

 

Objection: I Don't Have Time To Test Everything

 

Hopefully, if your project has incorporated some concepts of test-driven development and verified user requirements, this won't be a major issue. But realistically, there's the way things should be and there's the way things really are. Sometimes, although no one wants to admit it, there simply isn't enough time to test everything. Even when there is, the testing process can sometimes involve going through the steps so meticulously that the real way a user will work with an application isn't being tested at all.

 

But here's an easy way to test as many different pieces as possible: let FoxPro do it for you. This isn't a "guided test plan" approach but rather simply a way to ensure you haven't left any pieces just hanging. It's a "what would happen if the user hit everything possible on the screen". The other way of doing this would be to find the nearest available toddler and have them play with your application for a few hours

 

The goTester class uses a timer to check if there is a new active form in the application. If there is, it then goes through every object on the form and runs the events that might be triggered by a user. Each class can be customized to run specific events with a CASE statement. For those times when the application appears to stall, hit Ctrl+F12 and the testing object will start on the current form. The code for this solution is included in the Professional Resource CD.

 

Test results are stored in a table named TESTLOG that logs the name of the object being tested, the timestamp and any errors that were reported. The class itself is broken into four separate methods: TestForm, TestContainer, TestObject and then finally TestMethod.

 

The TestForm method checks to see if it has already tested this form by checking its Hwnd property. This is stored in an aObjects array so that the same form isn't tested multiple times unnecessarily. However, by storing its HWnd property, if the same form is created from multiple points (such as two different calling forms), it is still tested twice since that may create different results.

 

The TestContainer method loops through every control on the object that was passed to it.

 

LPARAMETERS toObject

FOR EACH LoControl IN toObject.Controls

lcClass = LOWER(locontrol.baseclass)

lcForm = toObject.Parent.Name+"."+toObject.Name

THIS.TestObject(loControl)

DO CASE

CASE lcClass='container'

THIS.TestContainer(loControl)

CASE lcClass='commandgroup'

THIS.TestContainer(loControl)

CASE lcClass='pageframe'

LOCAL lni

FOR lni = 1 TO loControl.PageCount

loControl.Activepage = lni

THIS.TestContainer(loControl.Pages(lni))

ENDFOR

OTHERWISE

THIS.TestObject(loControl,lcForm)

ENDCASE

ENDFOR

 

It first tests the control itself and then checks the baseclass. If the object is another container, all of the objects within that container are also tested the same way. The TestObject method calls TestMethod for each of the appropriate methods.

 

THIS.TestMethod(toObject,"Click")

THIS.TestMethod(toObject,"Valid")

 

The TestMethod procedure verifies that the method is valid and then executes it, catching any exceptions into the log file.

 

LPARAMETERS toObject,tcMethod

IF PEMSTATUS(toObject,tcMethod,5)

lcMsg = ""

TRY

toObject.&tcMethod()

CATCH TO loErr

lcMsg = loErr.Message + CHR(13) + ;

"on line "+TRANSFORM(loerr.Lineno)+CHR(13)+;

"Content: "+loErr.LineContents

ENDTRY

INSERT INTO (THIS.cLogFile) VALUES ;

(THIS.cObjName,toObject.Name+"."+tcMethod,DATETIME(),lcMsg)

ENDIF

 

 

One thing the class doesn't native handle are user-defined parameters. If a method expects specific parameters, they need to be hard-coded.

 

Why Can't I Just BINDEVENT?

 

Now, readers who have been following Drew Speedie's BINDEVENTs articles might be jumping up and down going "Use BINDEVENT Use BINDEVENT!" But sadly, BINDEVENT isn't going to help us here without some special code.

 

It would be nice to be able to say:

 

BINDEVENT(_SCREEN,"ActiveForm",goTester,"TestForm")

 

However, BINDEVENT does not work with every property and method available in _SCREEN and ActiveForm is one of them (neither does FormCount). So there's no immediate way to tell goTester to test a form.

 

VFP 9 lets you BINDEVENT to the HWnd and use Windows messages, however as of this writing, BINDEVENT does not react to the Windows API calls for CreateForm or CreateFormAs.

 

I mentioned above that this test class isn't the best way to test an application but it does offer a way of ensuring that no code that might be run by a user is left untested. I ran it on an older application before it was about to be put into production. After ten minutes, I opened the TESTLOG table and looked for any records with any exceptions noted. I was able to find a number of small errors that would have been caught with more thorough testing. In the same way that VFP continues to catch smaller project problems during the rebuild process, the Auto Tester finds the little nasty bugs that slip between the cracks. Consider it simply another tool in your testing toolbox.

 

The perfect software project is one where the specifications have been locked down, each individual line of code has been reviewed, debugged and tested, the application has been load-tested on every variety and possible combination of operating system and driver and the entire application is being run by someone who knew and agreed with all of the assumptions that were made during the design process. Unfortunately, as developers, all we can do is use as many tools as possible to ensure that what we turn out is as clean as it possibly can be. There are tools at our disposal now we just have to use them.