Example of Black Box Testing in Extreme Programming
Question :
Suppose we have an application that includes an interface (method) whose purpose is to find a "matching" record in a collection, if one exists. If none exists, the method is to return null.
The collection is large. Some users of this method have partial knowledge of the collection's order, so that they know that the record they want, if it is in there at all, occurs at or after some integer index in the collection.
So the method accepts a value, let's say a string /find/, to match the record on, and an integer /hint/, to be used as a hint to start the search. The first record in the table is numbered zero. The largest meaningful /hint/ value is therefore N-1, where N is the number of records in the table.
We want the search to always find a record if one exists, so that if /hint/ is wrong, but /find/ is in some record, we must still return a matching record, not null.
Now then. Assuming a black box, what questions do we want to ask, what tests do we want to write, against our method
Answer :
1. Generate various data sets to run the 'search' method against.
1a. Vary the number of items in the collection: create collections with 0, 1, 10, 100, 1000, 10000, 100000, 1 million items for starters; it may be the case that we hit an operating system limit at some point, for example if the items are files in the same directory (ever done an ls only to get back a message like "too many arguments"?)
1b. For each collection in 1a., generate several orderings: increasing order, decreasing order, random, maybe some other statistical distributions.
1c. Vary the length of the names of the items in the collection: create collections with 0, 1, 10, 100, 1000 items, where the names of the items are generated randomly with lengths between 1 and 1000 (arbitrary limit, which may change as we progress testing).
1d. Generate item names with 'weird' characters (especially /, \, :, ; -- since they tend to be used as separators by the OS).
1e. Generate item names that are Unicode strings.
2. Run (and time) the 'search' method against the various collections generated in 1. Make sure you cover cases such as:
2a. The item we search for is not in the collection: verify that the search method returns Null.
2b. The item we search for is in position p, where p can be 0, N/2, N-1, N.
2c. For each case in 2b, specify a hint of 0, p-1, p, p+1, N-1: verify that in all combinations of 2b and 2c, the search method returns the item in position p.
2d. Investigate the effect of item naming on the search. Does the search method work correctly when item names keep getting longer? When the item names contain 'weird' or Unicode characters?
2e. Graph the running time of the search method against collection size, when the item is or is not in the collection (so you generate 2 graphs). See if there is any anomaly.
2f. Run the tests in 2a-2d in a loop, to see if the search method produces a memory leak.
2g. Monitor various OS parameters (via top, vmstat, Windows PerfMon) to see how well-behaved the search functionality is in regards to the resources on that machine.
2h. See how the search method behaves when other resource-intensive processes are running on that machine (CPU-, disk-, memory-, network- intensive).
source : www.agiletesting.blogspot.com
Suppose we have an application that includes an interface (method) whose purpose is to find a "matching" record in a collection, if one exists. If none exists, the method is to return null.
The collection is large. Some users of this method have partial knowledge of the collection's order, so that they know that the record they want, if it is in there at all, occurs at or after some integer index in the collection.
So the method accepts a value, let's say a string /find/, to match the record on, and an integer /hint/, to be used as a hint to start the search. The first record in the table is numbered zero. The largest meaningful /hint/ value is therefore N-1, where N is the number of records in the table.
We want the search to always find a record if one exists, so that if /hint/ is wrong, but /find/ is in some record, we must still return a matching record, not null.
Now then. Assuming a black box, what questions do we want to ask, what tests do we want to write, against our method
Answer :
1. Generate various data sets to run the 'search' method against.
1a. Vary the number of items in the collection: create collections with 0, 1, 10, 100, 1000, 10000, 100000, 1 million items for starters; it may be the case that we hit an operating system limit at some point, for example if the items are files in the same directory (ever done an ls only to get back a message like "too many arguments"?)
1b. For each collection in 1a., generate several orderings: increasing order, decreasing order, random, maybe some other statistical distributions.
1c. Vary the length of the names of the items in the collection: create collections with 0, 1, 10, 100, 1000 items, where the names of the items are generated randomly with lengths between 1 and 1000 (arbitrary limit, which may change as we progress testing).
1d. Generate item names with 'weird' characters (especially /, \, :, ; -- since they tend to be used as separators by the OS).
1e. Generate item names that are Unicode strings.
2. Run (and time) the 'search' method against the various collections generated in 1. Make sure you cover cases such as:
2a. The item we search for is not in the collection: verify that the search method returns Null.
2b. The item we search for is in position p, where p can be 0, N/2, N-1, N.
2c. For each case in 2b, specify a hint of 0, p-1, p, p+1, N-1: verify that in all combinations of 2b and 2c, the search method returns the item in position p.
2d. Investigate the effect of item naming on the search. Does the search method work correctly when item names keep getting longer? When the item names contain 'weird' or Unicode characters?
2e. Graph the running time of the search method against collection size, when the item is or is not in the collection (so you generate 2 graphs). See if there is any anomaly.
2f. Run the tests in 2a-2d in a loop, to see if the search method produces a memory leak.
2g. Monitor various OS parameters (via top, vmstat, Windows PerfMon) to see how well-behaved the search functionality is in regards to the resources on that machine.
2h. See how the search method behaves when other resource-intensive processes are running on that machine (CPU-, disk-, memory-, network- intensive).
source : www.agiletesting.blogspot.com
