So far we have gathered requirements (Requirements Engineering), planned the project (Software Project Management) and designed the system (Software Design Principles, UML Diagrams, Data Flow Diagrams). Now we ask the uncomfortable question: does it actually work?
Testing is how we find out. It is also one of the largest activities in any real project and a very common interview topic, for developers as well as testers. This chapter lays the foundation: the vocabulary, the principles and the process. The following chapters build on it: Black-Box Testing and White-Box Testing & Cyclomatic Complexity cover how to design test cases, and Levels of Testing covers unit, integration, system and acceptance testing.
Software testing is the process of evaluating a software product to find defects and to check whether it meets its specified requirements and its users' needs, by executing it (dynamic testing) or examining it without execution (static testing).
Why Do We Test?
A common answer is "to show the program works". That is the wrong mindset. Glenford Myers, in The Art of Software Testing (1979), defined testing as "the process of executing a program with the intent of finding errors". If you set out to show that it works, you will unconsciously choose easy inputs. If you set out to break it, you will find the problems before users do.
The objectives of testing are:
Find defects before the software reaches users.
Verify that requirements are implemented as specified.
Validate that the product meets the users' real needs.
Build confidence in the level of quality, and give decision-makers information (for example, whether to release).
Prevent defects: reviewing requirements and designs early stops many defects from being written in the first place.
Reduce risk of failures in production, including legal, financial and safety risk.
Real failures show what is at stake. The Ariane 5 rocket (1996) self-destructed about 40 seconds after launch because a 64-bit floating-point value was converted to a 16-bit signed integer and overflowed, in code reused from Ariane 4 without testing against the new rocket's flight profile. The Therac-25 radiation machine (1985 to 1987) delivered massive overdoses because of race conditions in its control software.
Error, Defect, Failure
These three words are used loosely in daily speech, but in testing they mean different things. The terms below follow the ISTQB glossary and IEEE usage.
| Term | Also called | Definition | Who / where |
|---|---|---|---|
| Error | Mistake | A human action that produces an incorrect result | In the developer's head |
| Defect | Fault, bug | An imperfection in a work product (code, design, requirement) caused by an error | In the code or document |
| Failure | — | The system deviates from its expected behaviour during execution | Observed at run time |
Error (mistake) ──causes──▶ Defect (fault/bug) ──may cause──▶ Failure
by a person in the work product when executedLet us see this chain in real code. A developer writes a leap-year function and forgets the 400-year rule (the error):
def is_leap(year):
# Developer's error: forgot the 400-year rule
return year % 4 == 0 and year % 100 != 0
for y in [2024, 2023, 1900, 2000]:
print(y, is_leap(y))Output:
2024 True
2023 False
1900 False
2000 FalseThe defect is the missing or year % 400 == 0 in the code. It sits there silently for 2024, 2023 and 1900, which all give correct answers. Only for 2000 (a leap year, since it is divisible by 400) does the failure become visible: the program says False.
Two important lessons follow:
A defect does not always cause a failure. It needs the right input (here, a year divisible by 400) to be triggered. This is why choosing test inputs matters so much.
Not every failure comes from a defect. Environmental conditions like radiation flipping a memory bit, or a misconfigured server, can also cause failures.
The underlying reason the error happened (rushed schedule, unclear requirement, missing domain knowledge) is called the root cause. Fixing root causes prevents whole classes of future defects.
Verification vs Validation
Barry Boehm's famous one-liners make this easy to remember:
Verification: "Are we building the product right?" Does the product conform to its specification?
Validation: "Are we building the right product?" Does the product meet the user's actual needs?
| Aspect | Verification | Validation |
|---|---|---|
| Question | Built right (per spec)? | Right thing built (for users)? |
| Checked against | Specifications, design, standards | User needs and intended use |
| Typical activities | Reviews, walkthroughs, inspections, static analysis, unit and integration tests | System and acceptance testing, user trials, beta testing |
| Code execution | Often not needed (static) | Needs the running product (dynamic) |
| When | Throughout development, on each work product | Mostly on the working (or prototype) system |
| Example | Does the fee module apply the 18% GST rule written in the SRS? | Can students actually pay fees easily on a phone? |
A product can pass verification and fail validation: it perfectly matches a specification that captured the wrong requirement. The V-Model chapter showed how each development phase is paired with a test activity to support both. Together they are often abbreviated V&V.
Static vs Dynamic Testing
| Aspect | Static testing | Dynamic testing |
|---|---|---|
| Executes code? | No | Yes |
| Applied to | Requirements, designs, code, test plans, any document | Running software |
| Techniques | Informal review, walkthrough, technical review, inspection, static analysis tools (linters, type checkers) | Black-box, white-box and experience-based test techniques |
| Finds | Defects directly (ambiguous requirement, unreachable code, missing null check) | Failures, which are then traced back to defects |
| When | Can start as soon as a document exists | Needs executable code |
Static and dynamic testing complement each other. A review can spot that the leap-year rule is incomplete without running anything; a test with 2000 shows the failure.
Testing vs debugging
| Testing | Debugging |
|---|---|
| Shows that a failure exists (or finds defects via review) | Finds the cause (the defect), fixes it, and checks the fix |
| Done by testers and developers | Done by developers |
| Can be planned and largely automated | Investigative; hard to schedule |
| Starts with known expected results | Starts with an unexpected result |
The Seven Principles of Testing
The ISTQB Foundation Level syllabus (version 4.0) lists seven principles that hold for almost any project. They are a favourite exam question.
1. Testing shows the presence, not the absence, of defects
Testing can prove that defects exist. It can never prove that none remain. Passing 1,000 tests reduces the probability of undiscovered defects; it does not make it zero. (Dijkstra put it as: "Program testing can be used to show the presence of bugs, but never to show their absence.")
2. Exhaustive testing is impossible
Testing every possible input is not feasible except in trivial cases. Consider a function taking two 32-bit integers:
Input combinations = 2^32 × 2^32 = 2^64 = 18,446,744,073,709,551,616
At one billion (10^9) tests per second:
time = 2^64 / 10^9 seconds ≈ 1.84 × 10^10 seconds ≈ 584.5 yearsAnd that is one small function with no state. So we use test techniques (equivalence partitioning, boundary values, coverage), risk analysis and priorities to pick a small set of tests that find the most defects.
3. Early testing saves time and money
A defect found during requirements review costs a conversation; the same defect found after release may need code changes, retesting, redeployment and customer support. Start testing activities (reviews, test design) as early as possible. This is also called shift-left.
4. Defects cluster together
A small number of modules usually contain most of the defects (an application of the Pareto principle). Complex, frequently changed or newly written modules are typical hot spots. Use defect history to focus testing.
5. Tests wear out
If the same tests are run again and again, they eventually stop finding new defects, just as insects become resistant to a pesticide (older syllabi called this the pesticide paradox). Review and update tests, and add new ones. Note that for automated regression testing, repeating tests is exactly the point: they confirm that old behaviour still works.
6. Testing is context dependent
An aircraft control system, a banking app and a college fest website need very different testing: different techniques, rigour, documentation and tools.
7. Absence-of-defects fallacy
Finding and fixing many defects does not guarantee success. A system that is thoroughly tested but hard to use, or that does not meet users' needs, still fails. This is why validation matters as much as verification.
Anatomy of a Test Case
A test case is a set of preconditions, inputs, actions, expected results and postconditions, developed to check a particular test condition.
The expected result must be decided before running the test, from the specification. The source that tells you the correct expected result is called the test oracle (the SRS, a domain expert, a trusted older system, or a mathematical rule).
| ID | Test condition | Input | Expected | Actual | Status |
|---|---|---|---|---|---|
| TC01 | Divisible by 4, not by 100 | 2024 | True | True | Pass |
| TC02 | Not divisible by 4 | 2023 | False | False | Pass |
| TC03 | Divisible by 100, not by 400 | 1900 | False | False | Pass |
| TC04 | Divisible by 400 | 2000 | True | False | Fail |
Notice that each test case checks one distinct rule of the leap-year specification. That is the seed of equivalence partitioning, covered properly in Black-Box Testing.
Automating the same test cases
Test cases like these are usually automated so they can be re-run on every change. Here is the same suite with Python's built-in unittest:
import unittest
def is_leap(year):
return year % 4 == 0 and year % 100 != 0 # fault: 400 rule missing
class TestIsLeap(unittest.TestCase):
def test_divisible_by_4(self):
self.assertTrue(is_leap(2024))
def test_not_divisible_by_4(self):
self.assertFalse(is_leap(2023))
def test_century_not_leap(self):
self.assertFalse(is_leap(1900))
def test_divisible_by_400(self):
self.assertTrue(is_leap(2000))
if __name__ == "__main__":
unittest.main()Output (Python 3.14; older versions format the traceback slightly differently):
..F.
======================================================================
FAIL: test_divisible_by_400 (__main__.TestIsLeap.test_divisible_by_400)
----------------------------------------------------------------------
Traceback (most recent call last):
File "test_leap.py", line 14, in test_divisible_by_400
self.assertTrue(is_leap(2000))
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
AssertionError: False is not true
----------------------------------------------------------------------
Ran 4 tests in 0.001s
FAILED (failures=1)Each dot is a passed test and F a failed one. (unittest runs test methods in alphabetical order of their names, which is why the failure is third.) Changing the return line to return year % 400 == 0 or (year % 4 == 0 and year % 100 != 0) makes all four pass, and the suite now protects against this defect returning in future: a regression test.
The Test Process (STLC)
Testing is not a single activity at the end. Many Indian syllabi describe the Software Testing Life Cycle (STLC) in six phases:
| Phase | Main work | Key output |
|---|---|---|
| 1. Requirement analysis | Study requirements from a testing viewpoint; identify what is testable; clarify ambiguities | List of testable requirements, questions for clients |
| 2. Test planning | Decide scope, approach, resources, schedule, tools, entry/exit criteria, risks | Test plan, effort estimate |
| 3. Test case development | Design test cases and test data; prepare automation scripts | Test cases, test data, traceability matrix |
| 4. Test environment setup | Prepare hardware, software, network and data needed to run tests | Ready environment, smoke test result |
| 5. Test execution | Run tests, compare actual vs expected, log defects, retest fixes | Execution results, defect reports |
| 6. Test cycle closure | Evaluate exit criteria, report, record lessons learned | Test summary/closure report |
The ISTQB syllabus groups the same work slightly differently: test planning, monitoring and control, analysis, design, implementation, execution and completion. The ideas are the same.
Entry and exit criteria
Entry criteria: conditions to start a test activity, e.g. "build deployed to test server, smoke test passed, test data ready".
Exit criteria (definition of done for testing): conditions to stop, e.g. "all planned tests executed, 100% of high-priority tests passed, no open critical defects, statement coverage ≥ 80%".
Since exhaustive testing is impossible, "when do we stop?" is a risk decision: stop when the exit criteria are met, or when the remaining risk is acceptable for the deadline and budget.
Requirements traceability matrix (RTM)
An RTM maps each requirement to the test cases that cover it, so no requirement is left untested and every test has a reason to exist.
Requirement Test cases
REQ-07 Leap-year validation → TC01, TC02, TC03, TC04
REQ-08 Date-of-birth not future → TC05, TC06Defects: Life Cycle, Severity and Priority
Once a failure is found it is reported as a defect and moves through a defect (bug) life cycle. Names vary by tool (Jira, Bugzilla), but a typical flow is:
New ──▶ Assigned ──▶ Open ──▶ Fixed ──▶ Retest ──▶ Verified ──▶ Closed
│ │
├──▶ Rejected (not a bug) └──fails──▶ Reopened ──▶ Assigned
├──▶ Duplicate
└──▶ Deferred (fix in a later release)A good defect report contains: a short title, steps to reproduce, expected result, actual result, environment (build, browser, OS), severity, priority and evidence (screenshots, logs).
| Aspect | Severity | Priority |
|---|---|---|
| Meaning | How badly the defect affects the system | How urgently it must be fixed |
| Decided by | Usually the tester (technical impact) | Usually the product owner/manager (business need) |
| Typical scale | Critical, major, minor, trivial | P1 (immediate), P2, P3, P4 |
The two are independent, as these combinations show:
| Combination | Example |
|---|---|
| High severity, low priority | The app crashes when a rarely used legacy report is exported on an old browser version that few users have. |
| Low severity, high priority | The company's logo or name is misspelt on the home page just before launch. |
| High severity, high priority | Payments fail for every user. |
| Low severity, low priority | A minor alignment issue on a rarely visited help page. |
An Overview of Test Types
A test type is a group of tests focused on one quality aspect. You will meet these names constantly:
| Test type | Focus | Example |
|---|---|---|
| Functional testing | What the system does (features vs requirements) | Does "Pay Fees" generate a correct receipt? |
| Non-functional testing | How well it does it: performance, load, security, usability, reliability, compatibility | Can 5,000 students pay fees in the last hour before the deadline? |
| Confirmation testing (retesting) | Check that a specific fixed defect is really fixed | Re-run TC04 after fixing the leap-year rule |
| Regression testing | Check that a change has not broken something that used to work | Re-run the whole suite after every commit in CI |
| Smoke testing | Quick check that a new build's critical functions work at all, so deeper testing is worthwhile | App starts, login works, main page loads |
| Sanity testing | Narrow check of a specific area after a small change | After fixing the fee rounding, check fee pages only |
Test types are independent of how test cases are designed. The three families of design techniques are:
Black-box (specification-based): designed from requirements, without looking at code. See Black-Box Testing.
White-box (structure-based): designed from the code's internal structure to achieve coverage. See White-Box Testing & Cyclomatic Complexity.
Experience-based: error guessing, exploratory testing and checklists, drawing on the tester's knowledge. (Grey-box testing, a mix of black- and white-box knowledge, is also commonly mentioned.)
And they are independent of where in the life cycle testing happens (unit, integration, system, acceptance), which is the topic of Levels of Testing. The DevOps practice of running automated tests on every commit was covered in DevOps & CI/CD.
Manual vs automated testing
| Manual | Automated |
|---|---|
| Human executes steps and judges results | Scripts/tools execute and compare results |
| Good for exploratory, usability and one-off tests | Good for regression, repeated, data-heavy and performance tests |
| Low setup cost, high cost per run | High setup and maintenance cost, very low cost per run |
| Can notice unexpected things | Checks only what it was told to check |
Common Mistakes
Using error, defect and failure interchangeably. Error = human mistake, defect = flaw in the product, failure = wrong behaviour at run time.
Saying "the software has no bugs because all tests passed". Testing shows presence, not absence, of defects.
Writing tests without expected results, or deciding the expected result after seeing the output. That is not testing; it is just running the program.
Mixing up verification and validation. Verification checks against the spec; validation checks against user needs.
Mixing up severity and priority. Severity is technical impact; priority is business urgency.
Treating testing as the last phase only. Reviews of requirements and design are testing too, and they are the cheapest place to find defects.
Interview and Exam Questions
Q1. Differentiate error, fault and failure with an example.
An error is a human mistake (developer forgets the 400-year leap-year rule). It produces a fault/defect in the code (the missing condition). When the faulty code runs with a triggering input (year 2000), the system shows a failure (reports 2000 as not a leap year).
Q2. Can a program contain a defect and never fail?
Yes. If no input that triggers the defect is ever executed (for instance, a code path for an unused configuration), the defect remains latent and no failure is observed.
Q3. Why is exhaustive testing impossible? Give a quick calculation.
The input space is enormous. Two 32-bit integer inputs give 264 ≈ 1.8 × 1019 combinations; even at 109 tests per second that takes about 585 years, before considering state, timing and environments.
Q4. Verification or validation: (a) a code inspection against the design document; (b) a user acceptance test by the college accounts office?
(a) Verification: checking a work product against its specification. (b) Validation: checking that the system meets the real users' needs.
Q5. What is the "pesticide paradox" (tests wear out), and how do you counter it?
Running the same tests repeatedly eventually finds no new defects. Counter it by regularly reviewing and updating test cases, adding new tests and test data, and using different techniques (including exploratory testing).
Q6. Give an example of a high-priority, low-severity defect.
A spelling mistake in the company's name on the home page before a launch. It barely affects functionality (low severity) but must be fixed immediately for business reasons (high priority).
Q7. What is a test oracle?
The source used to determine the expected result of a test, such as the requirements specification, a domain expert, a previous trusted version of the system, or a known formula.
Summary
Testing evaluates software to find defects and build confidence that it meets requirements and user needs. Its mindset is to find problems, not to prove correctness.
Error (human mistake) → defect/fault/bug (flaw in the product) → failure (wrong run-time behaviour). Defects need the right input to show up as failures.
Verification: built right (vs spec). Validation: right product (vs user needs). Static testing examines without running; dynamic testing runs the code.
Seven principles: presence not absence; exhaustive testing impossible; early testing; defect clustering; tests wear out; context dependent; absence-of-defects fallacy.
A test case has preconditions, inputs, steps and an expected result decided in advance from a test oracle. STLC: requirement analysis → planning → test design → environment → execution → closure.
Defects follow a life cycle and are rated by severity (impact) and priority (urgency).
Next, Black-Box Testing shows how to choose a small set of powerful test cases from the specification alone.