2.1 Causes of
Requirements-Driven Risk
2.2 Consequences of Requirements-Driven Risk
2.3
The Cost of Requirements Driven Risk
3.1 The
Problem
3.2
The Solution – Requirements Engineering
3.3
Key Features of Requirements Engineering
3.4
The Cost of Requirements Engineering
3.5
Applying RE to Existing projects
3.6 Getting Started
3.7
How RE Reduces Risk Throughout the Project Life Cycle
-------------||-------------
Requirements-Driven
Risk (RDR) is a problem, shared across all government, industry and commercial
sectors, that is probably the single biggest cause of failure to meet time,
cost and performance targets for major enterprises - if you do not know if
you are acquiring, assessing or supporting the right thing, what greater risk
could you have? In order to
minimize the impact of RDR, we must:
·
Understand the nature of the risk;
·
Minimize it by:
-
Using proven procedures;
-
Measuring it to allow its management.
This paper presents an overview of the nature of Requirements-Driven
Risk and summarises some of the more important aspects of CSA’s approach based
on more than 25 elapsed years of in-depth experience developing requirements
specifications for clients across all market sectors.
2.1 Causes of
Requirements-Driven Risk
a
Misunderstanding
We all experience
difficulty when expressing requirements, without exception and whether or not
we realize it. Natural language
does not lend itself directly to expressing requirements in a complete,
coherent, consistent and correct (C4) form[1] - and formal languages have not yet evolved sufficiently for
this purpose. In fact there is significant
evidence to support the view that they never will. If the information that a Statement of Requirements (SOR)
contains is not well understood, it will be impossible to realize to time and
cost as a system which works as intended by the specifier – even in the best
and most well run organizations.
b
Subjectivity
The International Standards Organization (ISO)
insists that a requirement is not correctly specified unless it contains an
objective test by which it can be demonstrated to have been satisfied. Failure to do so at the outset results
in misinterpretation and problems for both specifier and deliverer in order to
truly understand what is being asked for (even if they think they do at the time),
with concommitant problems at acceptance.
c
Validity
SORs for complex systems are
necessarily large and complicated.
Such SORs invariably contain conflicting requirements that are hard to
spot until it’s too late (usually in the design, manufacture or even the in-service
phase). Put another way: we do not
know if it will work as expected, even if it is built to specification!
d
Incompleteness
SORs for complex systems contain statements at
various levels of detail, from several viewpoints and covering several layers
of concern. Requirements at
intermediate levels or layers are often missed by the specifier and have to be
guessed at during system realization – often with disastrous implications for
the system in question and other systems with which it is to interact.
e
Complexity
Every system exists as part of, and will
interact with, other elements in a wider system. The consequences of changes in the SORs for related systems
are rarely identified and taken into account, resulting in conflicts and gaps
in capability between such systems.
2.2 Consequences
of Requirements-Driven Risk
Failure to get the requirement right can cause
a variety of problems with timeliness, cost and performance. Examples are:
·
Failure to gain project approvals (weak business case, based on poor
requirement)
·
Contractual repercussions
(poor understanding of requirement by one, or both parties)
·
Cost overruns (re-work;
“hidden” requirements)
·
Time overruns (re-work; adding in missing detail)
·
Performance failures (failure to understand real operational requirement)
·
Nugatory work or cancelled projects (failure to filter out spurious
requirements)
·
Poor initial support arrangements (poorly understood support viewpoint )
·
Poor in-use decisions (poorly understood operational baseline)
2.3 The
Cost of Requirements-Driven Risk
It is now becoming widely accepted, from
research in the USA and in Europe, that if a fault that would have cost £1000
to correct during requirement specification is left until the design phase has
been initiated and it is left to the designers to detect and correct, then the
cost of correction will increase by a factor of 10, to £10,000. Similarly, if the fault is not detected
during subsequent design (which is often the case when parts of complex systems
are designed separately) and the fault is left until the system is being
manufactured before it is detected and corrected, the cost of doing so will
increase by a factor of 100 over that during specification, to £100,000.
If the detection
and correction of the fault is not carried out until after the system has gone
into service, the cost of correction will increase by a factor of 1000 over
that during specification - to £1M!
The cost of “working around” such a fault in service can be many times
higher again for systems with long life-cycles, such as warships, railway
networks and international financial systems. The only certainty is that the fault will be revealed eventually
– the longer it takes, the more painful and costly the consequence.
The above costs are
not exaggerations. Hardly a week
goes by without a system catastrophe being reported in the technical (and, increasingly,
national) press as being due to inadequate attention having been devoted to the
specification phase of the project.
The cost, in Europe alone, of inadequate attention to specifications, is
estimated to be running at between £27 billion and £30 billion per year and
rising. Such estimates do not
include the additional costs due to the inadequacy of the resulting systems and
all their knock-on consequences. A
report produced by the Standish Group (see Annex to this note) suggests that
the overall knock-on costs of poorly produced specifications could well be
$Trillions!
3 MINIMIZING
REQUIREMENTS-DRIVEN RISK
The fundamental problem is the need to identify
the specific requirement for the enterprise and express it in a way that
maximizes the probability that it will be satisfied as cheaply and quickly as
possible without compromising the performance of the delivered system(s).
3.2 The
Solution – Requirements Engineering
There is now a
proven, systematic approach known as Requirements Engineering, which will
achieve this crucial goal if applied methodically. Before describing this approach we need to make distinctions
between Requirements Capture (RC) - “getting it in”, Requirements Engineering
(RE) - “getting it right” and Requirements Management (RM) - “getting it done”[2]. All
three are equally important but sadly, whilst RC and RM are usually addressed, RE
is at best given lip service and, most often, is completely ignored. Failure to get the requirement right
means that all other efforts on behalf of the enterprise must be called
into question. To quote a well
known truism:
“Rubbish in =
rubbish out, no matter how well the rubbish is managed”.
3.3 Key
Features of Requirements Engineering
To be worthy of the “Engineering” label, RE
must have an associated and formally defined methodology. Any such methodology must feature
a logically derived procedure to develop the requirement, plus comprehensive
and objective metrics that are able to demonstrate improvements in the stated
requirement as it is developed:
a.
The need for procedural guidance in developing requirements is
inescapable if we are to have any confidence that we are going to produce
output of consistently high quality across all systems, learning from previous
mistakes and inventing wheels only once.
b.
The value of objective and meaningful requirements metrics to senior and
project management cannot be overstated, since the degree of
requirements-driven risk inherent in a particular SOR, at a particular time,
must be made explicit in order to allow informed decision-making at all levels,
including reassurance of stakeholders.
A good RE methodology must be able to handle
complexity - not only within a specific SOR, but also across many related SORs
(“system of systems”). This is
variously referred to as “Capability Management” or “Knowledge Management” and
it demands robust structural guidance from the methodology.
The cost of not
getting the requirement right is always too high – usually catastrophically so
(see Annex to this note and, in particular, the CHAOS report in Section
A1). The cost of ensuring
the quality of the requirement by means of a genuine RE approach is
disproportionately low, compared to the benefits it confers. Typically, it is less than 1% of the
project whole life cost[3]
– and this is able to be reduced even further (by at least an order of
magnitude) if RE is employed to create generic requirements for use
across system domains (eg Warships, Aircraft, Information Systems, etc) and if
it is employed sufficiently early in the life of the project.
3.5 Applying
RE to Existing Projects
The earlier RE is
applied, the lower will be the overall cost of getting the requirement right
and the greater will be the cumulative risk reduction throughout the project
lifecycle. That said, RE could be introduced
at any stage into an existing project in order to identify and remove
the un-quantified requirements-driven risk that are invariably present in all
“non-engineered” project specifications.
Any resources consumed in doing this will be justified, by several
orders of magnitude, as a result of the consequential performance improvements
and time & cost savings.
A minimum risk
starting point for anyone contemplating the use of Requirements Engineering,
but who are nevertheless nervous about adopting what might be regarded as a new
approach, would be to have a formal Subjectivity Audit. There is always delivered value from such
an audit and there is invariably a revelation in appreciation of the actual
quality of the subject specification compared with the previously perceived
quality.
From a client
organisation’s point of view a formal audit of a preferred supplier’s tender
can deliver more value than going out to competitive tender to a number of
companies. For one thing the
process is considerably less expensive and, for another, the problem of trying
to compare disparate approaches does not arise. Formal subjectivity audits also benefit the manufacturer as
well as the client. For example
when a manufacturer receives a specification, from a client for a proposed
system, its arrival invariably triggers a rash of clarification questions. A formal audit produces a complete set
of such questions, in addition to its other benefits, and prepares the way for
further development in a manner that is not possible by any means based on
visual inspection of documents.
3.7 How RE Reduces Risk Throughout the Project Life Cycle
Table 1 shows the principal benefits to be
gained at various stages of the project life cycle. Although the table follows
the MoD’s “CADMID” cycle, the benefits can be read across to other cycles.
The main points of this paper can be summarized
as:
·
The requirement provides the greatest source of project risk.
·
The consequences of requirements-driven risk are unacceptable.
·
Procedural guidance, metrics and complexity management are necessary to
minimize the risk.
·
A good Requirements Engineering methodology will address all three of
the above issues.
·
The cost of minimizing requirements-driven risk by means of a genuine RE
methodology is disproportionately low, compared to the benefits it confers.
·
It is never too late to start.
If major projects
are ever to be delivered to time, cost and performance, a RE
methodology, with the attributes described in Section 3.3 above, must be
applied to all projects, regardless of
their current stage, to obviate requirements-driven risk.
For those who have little or no experience of producing high quality
specifications it needs to be recognized that, just as for any other
engineering discipline, the development of a sound RE methodology is a
non-trivial exercise. It demands
considerable and dedicated effort.
Such development cannot be expected to be undertaken as a part-time
activity – even by the experienced.
It should not be expected that an effective methodology can be produced
on the back of any single project.
In order to ensure that it will provide adequate guidance a RE
methodology needs to have been developed by widely experienced, professional,
requirements engineers and scientists - and it needs to have been well tried
and tested before it is permitted to be used in anger.
It is strongly recommended that a common and proven RE methodology, with
the attributes described in Section 3.3 above[4], together with appropriate consultancy
and software support, is applied to all major enterprises, and that the metrics
so derived be used as standard performance indicators in risk reduction.
-------------||-------------
Table: PRINCIPAL
BENEFITS OF APPLYING RE
|
Project
Phase |
Benefits |
A good RE methodology
assists the user to: |
|
Concept |
ü Reduction in nugatory
research & development. ü
Early
identification of capability gaps. ü
Progressive
reduction in cost, time & effort necessary to develop successive user
requirements documents in application areas. ü
Flexibility
to adapt user requirements to new operational scenarios or technologies
across linked systems. ü
Early
metrics for risk management across capability areas. |
· manage
complexity by structuring and linking information at all levels and across
all systems; ·
acquire and manage knowledge; ·
identify generic, re-usable requirements within
application domains; ·
measure
requirements quality. |
|
Assessment |
ü Identification and
reduction of requirements-driven risk in project approval submissions. ü
Objective,
defensible requirements for business cases. ü
Progressive
elimination of requirements-driven risk from the Through-Life Management Plan. ü
Objective
assurance for all stakeholders of the degree to which their concerns have
been addressed. ü
Assistance
with the management of technology risk by avoidance of early solution lock-in. |
· demonstrate objectively the correctness,
completeness, coherence and consistency of the user requirement at any time
& from any viewpoint; ·
identify
and manage constraints imposed by existing systems and infrastructure, the
environment, legislation etc; ·
derive
and link system requirements from the user requirement to provide a solution-independent
framework for the assessment of technical proposals. |
|
Demonstration |
ü Optimization of
performance, time and cost requirements. ü
Reduced
contracting risk for customer and supplier. ü
System
will work if built to specification. |
· monitor and demonstrate the impact across
all linked requirements of proposed changes; ·
identify
and remove ambiguity, incompleteness and inconsistency from the SOR; ·
dynamically
model the complete requirement against required scenarios before design
commences. |
|
Manufacture |
ü
Right first
time ü
System
architecture confidence ü
Product
data management & enterprise integration ü
Minimize
acceptance risks |
· design the system against a SOR which has
been demonstrated to be correct, complete, coherent & consistent; ·
model
the impact of proposed and existing equipment on the system before design
commitment; ·
use the
solution-independent, structured requirement as a central reference for
distributed design development activities; ·
ensure
the existence of objective
acceptance tests against every requirement, regardless of level (also
essential for removing ambiguity); |
|
In-Service |
ü Optimized availability
within resource constraints. ü
Reduced
contracting risk for support. ü
Identification
and management of in-service capability shortfalls. |
· identify the impact of support decisions on
other aspects of the requirement such as availability, and vice-versa; ·
maintain
a current operational baseline against which in-service decisions and contracts
should be based; ·
model
the impact of changes to the operational base-line requirement (eg change of
use, new scenarios/threats) or system parameters (eg capability updates,
changes to associated systems) |
|
Disposal |
ü Identification and management
of disposal risks |
· Ensure that stakeholder viewpoints are
engineered into the overall SOR in such a way that the impact on each
viewpoint of changes to user or system requirements and constraints will be
identified |
-------------||-------------
A1. The Standish Group Report - CHAOS
This report is based on a
study of US IT projects. The US spends
about $250B/year on IT application development for approximately 175,000
projects.
The report states that 31%
of all projects are cancelled and 53% cost 189% of their original estimate.
Only 9% of
projects are completed on time and to budget and many of these are a mere
shadow of their original specification.
Projects completed by the
largest American companies have only 42% of the originally proposed features
and functions. Smaller companies
do much better. Large companies
overrun their time budget by an average of 230% and their cost budget by an
average of 178%.
The three main reasons that
a project will succeed are given in the report as: User involvement, executive
management support and a clear statement of requirements.
An opinion poll
identified incomplete requirements as the main reason why projects are
cancelled.
“The lost opportunity costs
are not measurable, but could easily be in the trillions of dollars”!
AND THINGS ARE NOT
GETTING BETTER: A high percentage of executive managers
(48%) were prepared to admit ‘there are more project failures now than five and
ten years ago’.
- 43% never delivered;
- 47% never used.
- Of
those delivered, 66% required significant modification.
Problems were caused
by:
- Poor/misunderstood User
Requirements (i.e. building the wrong system);
- Inability to cope with impact
of change/new requirements;
- Poor, or no user documentation,
deriving from poorly expressed requirements.
“Projects go over budget and more money is simply poured in. There is little thought about why this is happening, or whether more money is the correct thing.
Poor communications, vague
specifications and inadequate project management were named as the causes,
with 35% of projects being at serious risk.”
“The failure of Ariane 501 was caused by the complete loss of guidance and attitude information 37 seconds after start of the main engine ignition system (30 seconds after lift-off). This loss of information was due to specification and design errors in the software of the inertial reference system”.
“VIXEN, an intelligence gathering system … was
cancelled two weeks ago. Siemens
was paid the £50m, but the MOD wants its money back. Siemens has refused because it considers the equipment
completed to order. The project
appears to have foundered on the rocks of rapidly changing international
politics. The UK’s defence needs
look different today than in 1987 when the contract was awarded. Much of the disagreement rests on how
well Siemens was able to react to the army’s changing needs.
…..Siemens claims it built in extra
flexibility, and tests were approved in 1995 and 1996. But the MOD disputes this, claiming: ‘tests
conducted in 1995 and 1996 demonstrated that VIXEN has operational
shortcomings….”
CSA
comment: If the
VIXEN requirement had been properly engineered, then changes to the SOR to
reflect “the Army’s changing needs” would have been recorded unambiguously,
together with objective tests and acceptance criteria against each
requirement. This would have left
no room for argument over the conclusions. Instead, the only beneficiary of the dispute is the legal
profession!
“Services giant EDS is heading for a legal
showdown with the holiday firm Airtours following the collapse of their £100
million outsourcing deal….”
‘Only a tiny proportion of these project
disputes reach the courts,’ said Stuart Chapman, partner at technology law firm
Pinset Curtis…… ‘The usual problem is that the customer and supplier have
different perspectives at the procurement stage [of what is required]. Once they get down to the project, they
try to reconcile these different views and that’s where the pain starts…….’
A7. NAO Report “Accepting
[Military] Equipment Off Contract And Into Service” 11 February 2000
“…in one third of the projects surveyed, the
contract acceptance criteria did not fully reflect the staff requirement”.
CSA
Comment: The
International Standards Organization (ISO) defines a requirement as a statement
of need, together with the tests(s) and criteria by which it will be
demonstrated to have been met.
Without an objective acceptance test and criteria, no single requirement
is complete. Any RE methodology
must ensure that such tests exist for every requirement in the SOR,
regardless of level.
- Poor overall Leadership;
- Inadequately identifies needs
of individual probation services.
(Home Office Report
quoted in Times, 02 Nov 00).
- Caused by “Poor business case,
derived from unclear requirement”.
(Quote from IPT Requirements
Management Team, 21 Sep 00).
A10. Railtrack
Network Management Centre (NMC) – Report by Project Manager (Brown & Root)
In
1996 Railtrack commissioned a consulting systems house to undertake the
requirements capture for the project and produce a User Requirements
Specification. The usual steps
were gone through; interviews of users, workshops, feedback, peer reviews and
finally user review of the specification.
The consultant was duly paid and we planned the phase which would
translate the User Requirements into a specification of system requirements
against which we could invite industry to tender.
That
was when we discovered that no-one in the company really understood the User
Requirements Specification inasmuch as it might be used to define a set of
system requirements. The
consultant had gathered a lot of information, much of it necessarily anecdotal,
analysed it, divided it up into a number of academically pure and unrecognisable
processes and written it down in a notation that was entirely unintelligible to
the people who, alone, could have validated it. Six months of project programme had been used up in the
process and this put a number of constraints on the ensuing phases.
What
the client needs from the consultant who is going to undertake the requirements
capture process, especially from end-users, is a demonstration of how simple it
can be, not how clever and complex it all is.”
CSA Comment: To ensure the correctness, completeness, coherence and
consistency of a SOR, each requirement must be stripped to its “atomic” form
and captured and manipulated by the requirements engineer in a structured
database that is capable of managing high degrees of complexity in many
dimensions. Obviously, presenting
the end-user with “screen-shots” from such a database will lose the contextual
information and natural language presentation necessary for the end-user to
understand the requirements statements. This is equally true of the less
capable databases found in requirements management tools. CSA’s G-MARC approach avoids this
problem by making its requirements engineering database transparent to the end
user. At the stroke of a single
key, the G-MARC software operator can display the natural language passage(s)
within the original (source) document(s), or original statements by
contributors, which gave rise to the atomic requirement. Hence, the end-user is presented with
the atomic requirement together with all the associated contextual information
in a language that he can readily understand and relate to.
Furthermore, the structure revealed by the engineering of
the User Requirement is used by G-MARC to capture System Requirements as they
are identified (many are identified naturally during the development of the
User Requirement – it would be foolish to throw them away, only to have to
re-discover them later). This
means that, by the time the project is ready to “translate the User Requirement
into a specification of System Requirements” the structural links between the
various layers of concern (user, facilities, infrastructure, equipment etc) are
already known. This both speeds up
the development of the SRD and obviates the risk that the System
Requirements, which are often developed by different people, will not derive directly and explicitly
from the User Requirement (see examples 6. & 7. above).
The following is an extract
from the National Audit Office’s Major Projects Report dated 22 November 2000:
“2.5. The Defence Procurement Agency has also
developed and approved a performance indicator that is the percentage of the
total procurement cost falling before Main Gate. In the longer term, the Defence Procurement Agency plans to
monitor the trend in the performance indicator over time to place emphasis on spending more in the earlier stages of projects …….”
“2.17. The Strategic Defence Review
suggested that up to 15 per cent of the total procurement cost of a project be
spent before Main Gate”. Paragraph 2.17 then goes on to say
that this is not a hard and fast target, owing to variations in procurement
approach. Nevertheless, paragraph
2.18 states:
“2.18. ..…The ten pre-Main
Gate projects in Major Projects Report 2000 included five collaborative
programmes and one off-the-shelf buy.
Approved assessment phase cost was never more than six per cent on
any of the ten projects, although two of the projects, BOWMAN and MLS were
forecast at 31 March 2000 to spend 15 per cent and 19 per cent respectively of
their total procurement costs on assessment work. Nor is there any indication that more recently approved
projects which have yet to pass Main Gate are forecast to spend a higher
proportion during the assessment phase.”
CSA Comment: applying RE to projects as early as possible will shift the balance of
expenditure forward by significantly de-risking later phases and greatly reduce
the overall project cost. Since it
rarely represents more than 1% of the overall project cost, the arguments for
applying RE are overwhelming.
-------------||-------------
[1]
complete = no missing requirements. Coherent = statements grouped by level of
detail. Consistent = no
contradictions. Correct = accurate, atomic, objective statements.
[2] These definitions are consistent
with those in the MOD’s Smart Procurement/Acquisition Management System
Document “Guidance to the Writing of User Requirements Documents”.
[3] The Strategic Defence Review
recommended that up to 15 per cent
of the total procurement cost of a project be spent before Main Gate - MOD
currently averages 6% (NAO Major Projects Report 2000 refers – see Annex A).
[4] There is now a MOD-proven and
commercially available methodology & tool set (GMARC) that meets these
criteria. GMARC is supported by
highly qualified and experienced requirements engineers at CSA. By addressing the quality of the SOR in
a logical and repeatable way and providing a full range of supporting metrics,
the GMARC methodology facilitates the minimization of requirements-driven risk
to any enterprise.