Week 7. Software Quality Assurance

Software quality means the product meets its specification. This can be difficult for large systems as:

• There are many stakeholders with potentially conflicting requirements.
• Some quality requirements are difficult to specify unambiguously.
• Software specifications are usually incomplete, evolving and often inconsistent.

Assessing Fitness for Purpose:

• Has the software been properly tested?
• Is the software sufficiently dependable to be put into use?
• Is the performance of the software acceptable for normal use?
• Is the software usable?
• Is the software well-structured and understandable?
• Have programming and documentation standards been followed in the development process?

There are different types of software quality attributes:

The quality of process also affects quality of product. Subjective quality of software systems are largely based on non-functional properties which reflect practical user experience.

• If software functionality is incorrect, users will often find workarounds.
• If software is unreliable or too slow, no user-level fix is available, users lose trust and stop using the software system.

Some non-functional characteristics:

• Safety
• Understand ability
• Portability
• Security
• Test-ability
• Usability
• Reliability
• Adaptability
• Re-usability
• Resilience
• Modularity
• Efficiency
• Robustness
• Complexity
• Learnability

QA Management

We need to think about software quality management at two different levels:

• Organisational level: establish a framework of organisational processes and standards leading to high-quality software.
• Project level: establish a quality plan, goals for the project, processes to be used. Apply specific quality processes, check planned processes have been followed through with.

Also provide an independent check of software development, check project deliverables to ensure they are consistent with the standards.

A quality plan sets out desired product qualities, how these are assessed, defines most significant quality attributes and overall priorities, defines quality assessment process, and sets out organisational standards to apply.

QA Techniques

We use validation to check that we are building the right system.

• Is this was the user requires? We use verification to check if we are building the system right.
• Does it conform to spec?

Ultimately we are after “fitness for purpose” so we want to achieve sufficient confidence of system validity and correctness.

It depends on:

• software purpose: how critical is it to stakeholders
• user expectations: users may have low expectations of certain kinds of software
• business environment: product may need to get to market early

Verification

We may do formal verification if the project is high stakes, high risk, high safety, and / or highly critical. Usually only practical for small and essential subsystems.

Pros:

• can prove there are no erorrs
• enforces very careful thinking about all issues concerning the software

Cons:

• a lot of time and effort
• residual risk remains: can only prove properties of the model
• requires very high levels of formal expertise

Inspection

A group examines part or all of a process or system and its documentation to find potential problems, software or documents may be signed off (signifying progress to next development stage).

Types of review:

• Inspections for defect removal
• Reviews for progress assessment
• Quality reviews

Phases in review process:

• Pre-review activities: plan and prep for review
• Review meeting: author of document / program walks through the document with the review team
• Post-review activities: address problems and issues raised during the review

A team could use distributed reviews, remote reviewing can be done through documents that can be annotated, e.g. GitHub PR reviews.

Inspection Checklist Example

Testing

Program testing tends to show a program does what it is intended to do and to discover program defects before a program is put into use.

Testing means executing a program using artificial data, checking test results for errors, anomalies, or information about non-functional attributes.

Validation Testing It may be used to demonstrate to the developer and customer that software meets its requirements:

• Custom software: at least one test for every requirement in document
• Generic software products: tests for all system features + combination

Test cases reflect the system’s expected use.

Defect Testing Discover situation where the software behaviour is incorrect or undesirable, we root out undesirable system behaviour such as system crashes, unwanted interactions with other systems, incorrect computations and data corruption. Test cases are used to expose defects.

Testing vs. Inspection Testing is to exercise and observe product behaviour, executing test data. Inspection is to use static system representations to discover problems.

Within testing, errors can mask other errors, we need specialised testing environments or mocking, we cannot test static system properties. Within inspections, interactions between errors are not a concern, incomplete versions of a system can be inspected.

Development Testing

The system is tested during development to discover bugs and defects.

• Unit Testing
  • Tests individual program units or classes
  • Focus on testing functionality of objects and methods
  • Interaction types:
    • Parameter interfaces: Data passed from one method or procedure to another
    • Shared memory interfaces: Block of memory is shared between procedures or functions
    • Procedural interfaces: Sub-system encapsulates a set of procedures to be called by other sub-systems
    • Message passing interfaces: Sub-systems request services from other sub-systems
• Component Testing
  • Integrates several individual units to create composite components.
  • Focus on testing component interfaces.
• System Testing
  • Some or all components in a system are integrated and system tested a whole. We focus on testing component interactions.

How to select test cases?

• Black box testing: design test cases without considering details of the unit’s implementation. Aim to cover unit requirements exhaustively. Might overlook some paths through program logic that are only executed in very special circumstances.
  • How to choose unit test cases?
    • Partition testing: identify groups of inputs with common characteristics. Choose test cases from each group. All cases should be processed in the same way.
    • Guideline-based testing: use testing guidelines to choose test cases. Guidelines reflect previous experience of typical programmer errors.
• White box testing: design test cases explicitly considering the unit’s implementation. Aim to cover all branches / paths in the implementation.

General guidelines:

• Choose inputs that force the system to generate all error messages
• Design inputs that cause input buffers to overflow
• Repeat the same input or series of inputs numerous times
• Force invalid outputs to be generated
• Force computation results to be too large or too small

Guidelines for sequence / collection-based inputs:

• Test software with sequences which have only a single value
• Use sequences of different sizes in different tests
• Derive tests so that the first, middle and last elements of the sequence are accessed
• Test with sequences of zero length

(Check the rest of the slides, in 4.02 Development testing.pptx)

Release Testing

A separate testing team test a complete version of the system before it is released to users.

Primary goal: convince supplier that system is good enough for use

Show system delivers its specified functionality, performance and dependability, and does not fail during normal use. Usually a black-box testing process where tests are only derived from the system specification.

User Testing

Users or potential users of a system test the system in their own environment.

Essential, even when comprehensive system and release testing have been carried out

• Influences from the user’s working environment have a major effect on the reliability, performance, usability and robustness of a system
• Cannot be replicated in a testing environment

Types of user testing:

• Alpha testing: Users of the software work with the development team to test the software at the developer’s site
• Beta testing: A release of the software is made available to users to allow them to experiment and to raise problems that they discover with the system developers
• Acceptance testing: Customers test a system to decide whether or not it is ready to be accepted from the system developers and deployed in the customer environment. Primarily for custom systems.