Chapter 25 of 32

Software Testing Fundamentals

So far we have gathered requirements (Requirements Engineering), planned the project (Software Project Management) and designed the system (Software Design Principles, UML Diagrams, Data Flow Diagrams). Now we ask the uncomfortable question: does it actually work?

Testing is how we find out. It is also one of the largest activities in any real project and a very common interview topic, for developers as well as testers. This chapter lays the foundation: the vocabulary, the principles and the process. The following chapters build on it: Black-Box Testing and White-Box Testing & Cyclomatic Complexity cover how to design test cases, and Levels of Testing covers unit, integration, system and acceptance testing.

Software testing is the process of evaluating a software product to find defects and to check whether it meets its specified requirements and its users' needs, by executing it (dynamic testing) or examining it without execution (static testing).


Why Do We Test?

A common answer is "to show the program works". That is the wrong mindset. Glenford Myers, in The Art of Software Testing (1979), defined testing as "the process of executing a program with the intent of finding errors". If you set out to show that it works, you will unconsciously choose easy inputs. If you set out to break it, you will find the problems before users do.

The objectives of testing are:

  • Find defects before the software reaches users.

  • Verify that requirements are implemented as specified.

  • Validate that the product meets the users' real needs.

  • Build confidence in the level of quality, and give decision-makers information (for example, whether to release).

  • Prevent defects: reviewing requirements and designs early stops many defects from being written in the first place.

  • Reduce risk of failures in production, including legal, financial and safety risk.

Real failures show what is at stake. The Ariane 5 rocket (1996) self-destructed about 40 seconds after launch because a 64-bit floating-point value was converted to a 16-bit signed integer and overflowed, in code reused from Ariane 4 without testing against the new rocket's flight profile. The Therac-25 radiation machine (1985 to 1987) delivered massive overdoses because of race conditions in its control software.


Error, Defect, Failure

These three words are used loosely in daily speech, but in testing they mean different things. The terms below follow the ISTQB glossary and IEEE usage.

TermAlso calledDefinitionWho / where
ErrorMistakeA human action that produces an incorrect resultIn the developer's head
DefectFault, bugAn imperfection in a work product (code, design, requirement) caused by an errorIn the code or document
Failure—The system deviates from its expected behaviour during executionObserved at run time
   Error (mistake)  ──causes──▶  Defect (fault/bug)  ──may cause──▶  Failure
   by a person                    in the work product                 when executed

Let us see this chain in real code. A developer writes a leap-year function and forgets the 400-year rule (the error):

def is_leap(year):
    # Developer's error: forgot the 400-year rule
    return year % 4 == 0 and year % 100 != 0

for y in [2024, 2023, 1900, 2000]:
    print(y, is_leap(y))

Output:

2024 True
2023 False
1900 False
2000 False

The defect is the missing or year % 400 == 0 in the code. It sits there silently for 2024, 2023 and 1900, which all give correct answers. Only for 2000 (a leap year, since it is divisible by 400) does the failure become visible: the program says False.

Two important lessons follow:

  • A defect does not always cause a failure. It needs the right input (here, a year divisible by 400) to be triggered. This is why choosing test inputs matters so much.

  • Not every failure comes from a defect. Environmental conditions like radiation flipping a memory bit, or a misconfigured server, can also cause failures.

The underlying reason the error happened (rushed schedule, unclear requirement, missing domain knowledge) is called the root cause. Fixing root causes prevents whole classes of future defects.


Verification vs Validation

Barry Boehm's famous one-liners make this easy to remember:

  • Verification: "Are we building the product right?" Does the product conform to its specification?

  • Validation: "Are we building the right product?" Does the product meet the user's actual needs?

AspectVerificationValidation
QuestionBuilt right (per spec)?Right thing built (for users)?
Checked againstSpecifications, design, standardsUser needs and intended use
Typical activitiesReviews, walkthroughs, inspections, static analysis, unit and integration testsSystem and acceptance testing, user trials, beta testing
Code executionOften not needed (static)Needs the running product (dynamic)
WhenThroughout development, on each work productMostly on the working (or prototype) system
ExampleDoes the fee module apply the 18% GST rule written in the SRS?Can students actually pay fees easily on a phone?

A product can pass verification and fail validation: it perfectly matches a specification that captured the wrong requirement. The V-Model chapter showed how each development phase is paired with a test activity to support both. Together they are often abbreviated V&V.


Static vs Dynamic Testing

AspectStatic testingDynamic testing
Executes code?NoYes
Applied toRequirements, designs, code, test plans, any documentRunning software
TechniquesInformal review, walkthrough, technical review, inspection, static analysis tools (linters, type checkers)Black-box, white-box and experience-based test techniques
FindsDefects directly (ambiguous requirement, unreachable code, missing null check)Failures, which are then traced back to defects
WhenCan start as soon as a document existsNeeds executable code

Static and dynamic testing complement each other. A review can spot that the leap-year rule is incomplete without running anything; a test with 2000 shows the failure.

Testing vs debugging

TestingDebugging
Shows that a failure exists (or finds defects via review)Finds the cause (the defect), fixes it, and checks the fix
Done by testers and developersDone by developers
Can be planned and largely automatedInvestigative; hard to schedule
Starts with known expected resultsStarts with an unexpected result

The Seven Principles of Testing

The ISTQB Foundation Level syllabus (version 4.0) lists seven principles that hold for almost any project. They are a favourite exam question.

1. Testing shows the presence, not the absence, of defects

Testing can prove that defects exist. It can never prove that none remain. Passing 1,000 tests reduces the probability of undiscovered defects; it does not make it zero. (Dijkstra put it as: "Program testing can be used to show the presence of bugs, but never to show their absence.")

2. Exhaustive testing is impossible

Testing every possible input is not feasible except in trivial cases. Consider a function taking two 32-bit integers:

Input combinations = 2^32 × 2^32 = 2^64 = 18,446,744,073,709,551,616

At one billion (10^9) tests per second:
time = 2^64 / 10^9 seconds ≈ 1.84 × 10^10 seconds ≈ 584.5 years

And that is one small function with no state. So we use test techniques (equivalence partitioning, boundary values, coverage), risk analysis and priorities to pick a small set of tests that find the most defects.

3. Early testing saves time and money

A defect found during requirements review costs a conversation; the same defect found after release may need code changes, retesting, redeployment and customer support. Start testing activities (reviews, test design) as early as possible. This is also called shift-left.

4. Defects cluster together

A small number of modules usually contain most of the defects (an application of the Pareto principle). Complex, frequently changed or newly written modules are typical hot spots. Use defect history to focus testing.

5. Tests wear out

If the same tests are run again and again, they eventually stop finding new defects, just as insects become resistant to a pesticide (older syllabi called this the pesticide paradox). Review and update tests, and add new ones. Note that for automated regression testing, repeating tests is exactly the point: they confirm that old behaviour still works.

6. Testing is context dependent

An aircraft control system, a banking app and a college fest website need very different testing: different techniques, rigour, documentation and tools.

7. Absence-of-defects fallacy

Finding and fixing many defects does not guarantee success. A system that is thoroughly tested but hard to use, or that does not meet users' needs, still fails. This is why validation matters as much as verification.


Anatomy of a Test Case

A test case is a set of preconditions, inputs, actions, expected results and postconditions, developed to check a particular test condition.

The expected result must be decided before running the test, from the specification. The source that tells you the correct expected result is called the test oracle (the SRS, a domain expert, a trusted older system, or a mathematical rule).

IDTest conditionInputExpectedActualStatus
TC01Divisible by 4, not by 1002024TrueTruePass
TC02Not divisible by 42023FalseFalsePass
TC03Divisible by 100, not by 4001900FalseFalsePass
TC04Divisible by 4002000TrueFalseFail

Notice that each test case checks one distinct rule of the leap-year specification. That is the seed of equivalence partitioning, covered properly in Black-Box Testing.

Automating the same test cases

Test cases like these are usually automated so they can be re-run on every change. Here is the same suite with Python's built-in unittest:

import unittest

def is_leap(year):
    return year % 4 == 0 and year % 100 != 0     # fault: 400 rule missing

class TestIsLeap(unittest.TestCase):
    def test_divisible_by_4(self):
        self.assertTrue(is_leap(2024))
    def test_not_divisible_by_4(self):
        self.assertFalse(is_leap(2023))
    def test_century_not_leap(self):
        self.assertFalse(is_leap(1900))
    def test_divisible_by_400(self):
        self.assertTrue(is_leap(2000))

if __name__ == "__main__":
    unittest.main()

Output (Python 3.14; older versions format the traceback slightly differently):

..F.
======================================================================
FAIL: test_divisible_by_400 (__main__.TestIsLeap.test_divisible_by_400)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_leap.py", line 14, in test_divisible_by_400
    self.assertTrue(is_leap(2000))
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
AssertionError: False is not true

----------------------------------------------------------------------
Ran 4 tests in 0.001s

FAILED (failures=1)

Each dot is a passed test and F a failed one. (unittest runs test methods in alphabetical order of their names, which is why the failure is third.) Changing the return line to return year % 400 == 0 or (year % 4 == 0 and year % 100 != 0) makes all four pass, and the suite now protects against this defect returning in future: a regression test.


The Test Process (STLC)

Testing is not a single activity at the end. Many Indian syllabi describe the Software Testing Life Cycle (STLC) in six phases:

PhaseMain workKey output
1. Requirement analysisStudy requirements from a testing viewpoint; identify what is testable; clarify ambiguitiesList of testable requirements, questions for clients
2. Test planningDecide scope, approach, resources, schedule, tools, entry/exit criteria, risksTest plan, effort estimate
3. Test case developmentDesign test cases and test data; prepare automation scriptsTest cases, test data, traceability matrix
4. Test environment setupPrepare hardware, software, network and data needed to run testsReady environment, smoke test result
5. Test executionRun tests, compare actual vs expected, log defects, retest fixesExecution results, defect reports
6. Test cycle closureEvaluate exit criteria, report, record lessons learnedTest summary/closure report

The ISTQB syllabus groups the same work slightly differently: test planning, monitoring and control, analysis, design, implementation, execution and completion. The ideas are the same.

Entry and exit criteria

  • Entry criteria: conditions to start a test activity, e.g. "build deployed to test server, smoke test passed, test data ready".

  • Exit criteria (definition of done for testing): conditions to stop, e.g. "all planned tests executed, 100% of high-priority tests passed, no open critical defects, statement coverage ≥ 80%".

Since exhaustive testing is impossible, "when do we stop?" is a risk decision: stop when the exit criteria are met, or when the remaining risk is acceptable for the deadline and budget.

Requirements traceability matrix (RTM)

An RTM maps each requirement to the test cases that cover it, so no requirement is left untested and every test has a reason to exist.

Requirement                        Test cases
REQ-07 Leap-year validation   →    TC01, TC02, TC03, TC04
REQ-08 Date-of-birth not future →  TC05, TC06

Defects: Life Cycle, Severity and Priority

Once a failure is found it is reported as a defect and moves through a defect (bug) life cycle. Names vary by tool (Jira, Bugzilla), but a typical flow is:

New ──▶ Assigned ──▶ Open ──▶ Fixed ──▶ Retest ──▶ Verified ──▶ Closed
 │                                         │
 ├──▶ Rejected (not a bug)                 └──fails──▶ Reopened ──▶ Assigned
 ├──▶ Duplicate
 └──▶ Deferred (fix in a later release)

A good defect report contains: a short title, steps to reproduce, expected result, actual result, environment (build, browser, OS), severity, priority and evidence (screenshots, logs).

AspectSeverityPriority
MeaningHow badly the defect affects the systemHow urgently it must be fixed
Decided byUsually the tester (technical impact)Usually the product owner/manager (business need)
Typical scaleCritical, major, minor, trivialP1 (immediate), P2, P3, P4

The two are independent, as these combinations show:

CombinationExample
High severity, low priorityThe app crashes when a rarely used legacy report is exported on an old browser version that few users have.
Low severity, high priorityThe company's logo or name is misspelt on the home page just before launch.
High severity, high priorityPayments fail for every user.
Low severity, low priorityA minor alignment issue on a rarely visited help page.

An Overview of Test Types

A test type is a group of tests focused on one quality aspect. You will meet these names constantly:

Test typeFocusExample
Functional testingWhat the system does (features vs requirements)Does "Pay Fees" generate a correct receipt?
Non-functional testingHow well it does it: performance, load, security, usability, reliability, compatibilityCan 5,000 students pay fees in the last hour before the deadline?
Confirmation testing (retesting)Check that a specific fixed defect is really fixedRe-run TC04 after fixing the leap-year rule
Regression testingCheck that a change has not broken something that used to workRe-run the whole suite after every commit in CI
Smoke testingQuick check that a new build's critical functions work at all, so deeper testing is worthwhileApp starts, login works, main page loads
Sanity testingNarrow check of a specific area after a small changeAfter fixing the fee rounding, check fee pages only

Test types are independent of how test cases are designed. The three families of design techniques are:

  • Black-box (specification-based): designed from requirements, without looking at code. See Black-Box Testing.

  • White-box (structure-based): designed from the code's internal structure to achieve coverage. See White-Box Testing & Cyclomatic Complexity.

  • Experience-based: error guessing, exploratory testing and checklists, drawing on the tester's knowledge. (Grey-box testing, a mix of black- and white-box knowledge, is also commonly mentioned.)

And they are independent of where in the life cycle testing happens (unit, integration, system, acceptance), which is the topic of Levels of Testing. The DevOps practice of running automated tests on every commit was covered in DevOps & CI/CD.

Manual vs automated testing

ManualAutomated
Human executes steps and judges resultsScripts/tools execute and compare results
Good for exploratory, usability and one-off testsGood for regression, repeated, data-heavy and performance tests
Low setup cost, high cost per runHigh setup and maintenance cost, very low cost per run
Can notice unexpected thingsChecks only what it was told to check

Common Mistakes

  • Using error, defect and failure interchangeably. Error = human mistake, defect = flaw in the product, failure = wrong behaviour at run time.

  • Saying "the software has no bugs because all tests passed". Testing shows presence, not absence, of defects.

  • Writing tests without expected results, or deciding the expected result after seeing the output. That is not testing; it is just running the program.

  • Mixing up verification and validation. Verification checks against the spec; validation checks against user needs.

  • Mixing up severity and priority. Severity is technical impact; priority is business urgency.

  • Treating testing as the last phase only. Reviews of requirements and design are testing too, and they are the cheapest place to find defects.


Interview and Exam Questions

Q1. Differentiate error, fault and failure with an example.

An error is a human mistake (developer forgets the 400-year leap-year rule). It produces a fault/defect in the code (the missing condition). When the faulty code runs with a triggering input (year 2000), the system shows a failure (reports 2000 as not a leap year).

Q2. Can a program contain a defect and never fail?

Yes. If no input that triggers the defect is ever executed (for instance, a code path for an unused configuration), the defect remains latent and no failure is observed.

Q3. Why is exhaustive testing impossible? Give a quick calculation.

The input space is enormous. Two 32-bit integer inputs give 264 ≈ 1.8 × 1019 combinations; even at 109 tests per second that takes about 585 years, before considering state, timing and environments.

Q4. Verification or validation: (a) a code inspection against the design document; (b) a user acceptance test by the college accounts office?

(a) Verification: checking a work product against its specification. (b) Validation: checking that the system meets the real users' needs.

Q5. What is the "pesticide paradox" (tests wear out), and how do you counter it?

Running the same tests repeatedly eventually finds no new defects. Counter it by regularly reviewing and updating test cases, adding new tests and test data, and using different techniques (including exploratory testing).

Q6. Give an example of a high-priority, low-severity defect.

A spelling mistake in the company's name on the home page before a launch. It barely affects functionality (low severity) but must be fixed immediately for business reasons (high priority).

Q7. What is a test oracle?

The source used to determine the expected result of a test, such as the requirements specification, a domain expert, a previous trusted version of the system, or a known formula.


Summary

  • Testing evaluates software to find defects and build confidence that it meets requirements and user needs. Its mindset is to find problems, not to prove correctness.

  • Error (human mistake) → defect/fault/bug (flaw in the product) → failure (wrong run-time behaviour). Defects need the right input to show up as failures.

  • Verification: built right (vs spec). Validation: right product (vs user needs). Static testing examines without running; dynamic testing runs the code.

  • Seven principles: presence not absence; exhaustive testing impossible; early testing; defect clustering; tests wear out; context dependent; absence-of-defects fallacy.

  • A test case has preconditions, inputs, steps and an expected result decided in advance from a test oracle. STLC: requirement analysis → planning → test design → environment → execution → closure.

  • Defects follow a life cycle and are rated by severity (impact) and priority (urgency).

  • Next, Black-Box Testing shows how to choose a small set of powerful test cases from the specification alone.