Requirements Driven Risk

1 INTRODUCTION

Requirements-Driven Risk (RDR) is a problem, shared across all government, industry and commercial sectors, that is probably the single biggest cause of failure to meet time, cost and performance targets for major enterprises - if you do not know if you are acquiring, assessing or supporting the right thing, what greater risk could you have? In order to minimize the impact of RDR, we must:

· Understand the nature of the risk;

· Minimize it by:

- Using proven procedures;

- Measuring it to allow its management.

This paper presents an overview of the nature of Requirements-Driven Risk and summarises some of the more important aspects of CSA’s approach based on more than 25 elapsed years of in-depth experience developing requirements specifications for clients across all market sectors.

2 THE NATURE OF REQUIREMENTS-DRIVEN RISK

2.1 Causes of Requirements-Driven Risk

a Misunderstanding

We all experience difficulty when expressing requirements, without exception and whether or not we realize it. Natural language does not lend itself directly to expressing requirements in a complete, coherent, consistent and correct (C4) form[1] - and formal languages have not yet evolved sufficiently for this purpose. In fact there is significant evidence to support the view that they never will. If the information that a Statement of Requirements (SOR) contains is not well understood, it will be impossible to realize to time and cost as a system which works as intended by the specifier – even in the best and most well run organizations.

b Subjectivity

The International Standards Organization (ISO) insists that a requirement is not correctly specified unless it contains an objective test by which it can be demonstrated to have been satisfied. Failure to do so at the outset results in misinterpretation and problems for both specifier and deliverer in order to truly understand what is being asked for (even if they think they do at the time), with concommitant problems at acceptance.

c Validity

SORs for complex systems are necessarily large and complicated. Such SORs invariably contain conflicting requirements that are hard to spot until it’s too late (usually in the design, manufacture or even the in-service phase). Put another way: we do not know if it will work as expected, even if it is built to specification!

d Incompleteness

SORs for complex systems contain statements at various levels of detail, from several viewpoints and covering several layers of concern. Requirements at intermediate levels or layers are often missed by the specifier and have to be guessed at during system realization – often with disastrous implications for the system in question and other systems with which it is to interact.

e Complexity

Every system exists as part of, and will interact with, other elements in a wider system. The consequences of changes in the SORs for related systems are rarely identified and taken into account, resulting in conflicts and gaps in capability between such systems.

2.2 Consequences of Requirements-Driven Risk

Failure to get the requirement right can cause a variety of problems with timeliness, cost and performance. Examples are:

· Failure to gain project approvals (weak business case, based on poor requirement)

· Contractual repercussions (poor understanding of requirement by one, or both parties)

· Cost overruns (re-work; “hidden” requirements)

· Time overruns (re-work; adding in missing detail)

· Performance failures (failure to understand real operational requirement)

· Nugatory work or cancelled projects (failure to filter out spurious requirements)

· Poor initial support arrangements (poorly understood support viewpoint )

· Poor in-use decisions (poorly understood operational baseline)

2.3 The Cost of Requirements-Driven Risk

It is now becoming widely accepted, from research in the USA and in Europe, that if a fault that would have cost £1000 to correct during requirement specification is left until the design phase has been initiated and it is left to the designers to detect and correct, then the cost of correction will increase by a factor of 10, to £10,000. Similarly, if the fault is not detected during subsequent design (which is often the case when parts of complex systems are designed separately) and the fault is left until the system is being manufactured before it is detected and corrected, the cost of doing so will increase by a factor of 100 over that during specification, to £100,000.

If the detection and correction of the fault is not carried out until after the system has gone into service, the cost of correction will increase by a factor of 1000 over that during specification - to £1M! The cost of “working around” such a fault in service can be many times higher again for systems with long life-cycles, such as warships, railway networks and international financial systems. The only certainty is that the fault will be revealed eventually – the longer it takes, the more painful and costly the consequence.

The above costs are not exaggerations. Hardly a week goes by without a system catastrophe being reported in the technical (and, increasingly, national) press as being due to inadequate attention having been devoted to the specification phase of the project. The cost, in Europe alone, of inadequate attention to specifications, is estimated to be running at between £27 billion and £30 billion per year and rising. Such estimates do not include the additional costs due to the inadequacy of the resulting systems and all their knock-on consequences. A report produced by the Standish Group (see Annex) suggests that the overall knock-on costs of poorly produced specifications could well be $Trillions!

3 MINIMIZING REQUIREMENTS-DRIVEN RISK

3.1 The Problem

The fundamental problem is the need to identify the specific requirement for the enterprise and express it in a way that maximizes the probability that it will be satisfied as cheaply and quickly as possible without compromising the performance of the delivered system(s).

3.2 The Solution – Requirements Engineering

There is now a proven, systematic approach known as Requirements Engineering, which will achieve this crucial goal if applied methodically. Before describing this approach we need to make distinctions between Requirements Capture (RC) - “getting it in”, Requirements Engineering (RE) - “getting it right” and Requirements Management (RM) - “getting it done”[2]. All three are equally important but sadly, whilst RC and RM are usually addressed, RE is at best given lip service and, most often, is completely ignored. Failure to get the requirement right means that all other efforts on behalf of the enterprise must be called into question. To quote a well known truism:

“Rubbish in = rubbish out, no matter how well the rubbish is managed”.

3.3 Key Features of Requirements Engineering

To be worthy of the “Engineering” label, RE must have an associated and formally defined methodology. Any such methodology must feature a logically derived procedure to develop the requirement, plus comprehensive and objective metrics that are able to demonstrate improvements in the stated requirement as it is developed:

a. The need for procedural guidance in developing requirements is inescapable if we are to have any confidence that we are going to produce output of consistently high quality across all systems, learning from previous mistakes and inventing wheels only once.

b. The value of objective and meaningful requirements metrics to senior and project management cannot be overstated, since the degree of requirements-driven risk inherent in a particular SOR, at a particular time, must be made explicit in order to allow informed decision-making at all levels, including reassurance of stakeholders.

A good RE methodology must be able to handle complexity - not only within a specific SOR, but also across many related SORs (“system of systems”). This is variously referred to as “Capability Management” or “Knowledge Management” and it demands robust structural guidance from the methodology.

3.4 The Cost of RE

The cost of not getting the requirement right is always too high – usually catastrophically so (see Annex A and, in particular, the CHAOS report on page A-1. The cost of ensuring the quality of the requirement by means of a genuine RE approach is disproportionately low, compared to the benefits it confers. Typically, it is less than 1% of the project whole life cost[3] – and this is able to be reduced even further (by at least an order of magnitude) if RE is employed to create generic requirements for use across system domains (eg Warships, Aircraft, Information Systems, etc) and if it is employed sufficiently early in the life of the project.

3.5 Applying RE to Existing Projects

The earlier RE is applied, the lower will be the overall cost of getting the requirement right and the greater will be the cumulative risk reduction throughout the project lifecycle. That said, RE could be introduced at any stage into an existing project in order to identify and remove the un-quantified requirements-driven risk that are invariably present in all “non-engineered” project specifications. Any resources consumed in doing this will be justified, by several orders of magnitude, as a result of the consequential performance improvements and time & cost savings.

3.6 Getting Started

A minimum risk starting point for anyone contemplating the use of Requirements Engineering, but who are nevertheless nervous about adopting what might be regarded as a new approach, would be to have a formal Subjectivity Audit. There is always delivered value from such an audit and there is invariably a revelation in appreciation of the actual quality of the subject specification compared with the previously perceived quality.

From a client organisation’s point of view a formal audit of a preferred supplier’s tender can deliver more value than going out to competitive tender to a number of companies. For one thing the process is considerably less expensive and, for another, the problem of trying to compare disparate approaches does not arise. Formal subjectivity audits also benefit the manufacturer as well as the client. For example when a manufacturer receives a specification, from a client for a proposed system, its arrival invariably triggers a rash of clarification questions. A formal audit produces a complete set of such questions, in addition to its other benefits, and prepares the way for further development in a manner that is not possible by any means based on visual inspection of documents.

3.7 How RE Reduces Risk Throughout the Project Life Cycle

Table 1 shows the principal benefits to be gained at various stages of the project life cycle. Although the table follows the MoD’s “CADMID” cycle, the benefits can be read across to other cycles.

4 SUMMARY

The main points of this paper can be summarized as:

· The requirement provides the greatest source of project risk.

· The consequences of requirements-driven risk are unacceptable.

· Procedural guidance, metrics and complexity management are necessary to minimize the risk.

· A good Requirements Engineering methodology will address all three of the above issues.

· The cost of minimizing requirements-driven risk by means of a genuine RE methodology is disproportionately low, compared to the benefits it confers.

· It is never too late to start.

5 CONCLUSION

If major projects are ever to be delivered to time, cost and performance, a RE methodology, with the attributes described in Section 3.3 above, must be applied to all projects, regardless of their current stage, to obviate requirements-driven risk.

6 RECOMMENDATION

For those who have little or no experience of producing high quality specifications it needs to be recognized that, just as for any other engineering discipline, the development of a sound RE methodology is a non-trivial exercise. It demands considerable and dedicated effort. Such development cannot be expected to be undertaken as a part-time activity – even by the experienced. It should not be expected that an effective methodology can be produced on the back of any single project. In order to ensure that it will provide adequate guidance a RE methodology needs to have been developed by widely experienced, professional, requirements engineers and scientists - and it needs to have been well tried and tested before it is permitted to be used in anger.

It is strongly recommended that a common and proven RE methodology, with the attributes described in Section 3.3 above[4], together with appropriate consultancy and software support, is applied to all major enterprises, and that the metrics so derived be used as standard performance indicators in risk reduction.

-------------||-------------

Table: PRINCIPAL BENEFITS OF APPLYING RE

Project Phase

Benefits

A good RE methodology assists the user to:

Concept

ü Reduction in nugatory research & development.

ü Early identification of capability gaps.

ü Progressive reduction in cost, time & effort necessary to develop successive user requirements documents in application areas.

ü Flexibility to adapt user requirements to new operational scenarios or technologies across linked systems.

ü Early metrics for risk management across capability areas.

· manage complexity by structuring and linking information at all levels and across all systems;

· acquire and manage knowledge;

· identify generic, re-usable requirements within application domains;

· measure requirements quality.

Assessment

ü Identification and reduction of requirements-driven risk in project approval submissions.

ü Objective, defensible requirements for business cases.

ü Progressive elimination of requirements-driven risk from the Through-Life Management Plan.

ü Objective assurance for all stakeholders of the degree to which their concerns have been addressed.

ü Assistance with the management of technology risk by avoidance of early solution lock-in.

· demonstrate objectively the correctness, completeness, coherence and consistency of the user requirement at any time & from any viewpoint;

· identify and manage constraints imposed by existing systems and infrastructure, the environment, legislation etc;

· derive and link system requirements from the user requirement to provide a solution-independent framework for the assessment of technical proposals.

Demonstration

ü Optimization of performance, time and cost requirements.

ü Reduced contracting risk for customer and supplier.

ü System will work if built to specification.

· monitor and demonstrate the impact across all linked requirements of proposed changes;

· identify and remove ambiguity, incompleteness and inconsistency from the SOR;

· dynamically model the complete requirement against required scenarios before design commences.

Manufacture

ü Right first time

ü System architecture confidence

ü Product data management & enterprise integration

ü Minimize acceptance risks

· design the system against a SOR which has been demonstrated to be correct, complete, coherent & consistent;

· model the impact of proposed and existing equipment on the system before design commitment;

· use the solution-independent, structured requirement as a central reference for distributed design development activities;

· ensure the existence of objective acceptance tests against every requirement, regardless of level (also essential for removing ambiguity);

In-Service

ü Optimized availability within resource constraints.

ü Reduced contracting risk for support.

ü Identification and management of in-service capability shortfalls.

· identify the impact of support decisions on other aspects of the requirement such as availability, and vice-versa;

· maintain a current operational baseline against which in-service decisions and contracts should be based;

· model the impact of changes to the operational base-line requirement (eg change of use, new scenarios/threats) or system parameters (eg capability updates, changes to associated systems)

Disposal

ü Identification and management of disposal risks

· Ensure that stakeholder viewpoints are engineered into the overall SOR in such a way that the impact on each viewpoint of changes to user or system requirements and constraints will be identified

-------------||-------------

CONTENTS

1 INTRODUCTION

2 THE NATURE OF REQUIREMENTS-DRIVEN RISK

3 MINIMISING REQUIREMENTS-DRIVEN RISK

4 SUMMARY

5 CONCLUSION

6 RECOMMENDATION

Requirements Catastrophies

1 INTRODUCTION

2 THE NATURE OF REQUIREMENTS-DRIVEN RISK

4 SUMMARY

5 CONCLUSION

6 RECOMMENDATION

ANNEX

REQUIREMENTS CATASTROPHES

2. IEEE Study of 9 major US Government Software Contracts (Quoted in “Scientific Data Management”, Mar 98)