Agile Attitudes- Let's Look at the Embedded Software Quality Numbers

Fri Feb 11 11:37:57 EST 2005

Agile Attitudes
Volume 2, Issue 3                                     Feb. 8, 2005
         A free bi-weekly email newsletter
                Brought to you by
             Agile Rules consulting
               www.agilerules.com
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Welcome to Agile Attitudes, a newsletter of ideas, insights and
technical tips that help people find better ways to develop
software. Feel free to share this with anyone - just be sure you
send or print the whole thing, including the copyright notice.

Directions for managing your subscription are below.

                 O><O><O><O><O><O
XP Day - Toronto is a one-day conference happening on Saturday, 
Feb 19th. The cost to attend is only $299 (Canadian). For info, see
http://www.xpday.info 

                 O><O><O><O><O><O
 Let's Look at the Embedded Software Quality Numbers
           by Nancy Van Schooenderwoert

Agile software development is all about feedback loops. Programmers 
get feedback on how well their code is working by running sets of 
tests, ideally several times per hour. Management gets feedback on 
how well the software team is doing because they exercise the code 
delivered at each iteration – typically one or more times in a 
month. Customers get feedback on how usable the code is at every 
release, and they give feedback to the project team.

Even with all this feedback going on, some bugs do still happen. 
Perhaps none would if everyone on the team used XP practices 
perfectly – always writing tests for every bit of code, always 
refactoring promptly, always being careful to really understand 
what the customer is asking for (if the customer's not writing 
acceptance tests). 

The most fundamental feedback loop is the one that tells the 
developers how well their coding practices are working. I don't 
mean merely a unit test of the function just written – I mean 
feedback that the coding practices and techniques are performing 
as expected. The best indicator of that is the bugs that get 
through your defenses.

I'm accustomed to bug statistics that are collected at each of the 
waterfall phases; so many defects inserted in the requirements 
stage, so many in design, etc. but when you use XP those early 
phases are rolled into a tight ball. If your design-code-test 
cycle is 3 minutes long then stopping to log bugs at every compile 
is just goofy. 

What I do is log them at the next natural pause – the point where 
unit testing is declared complete and the code is ready for 
integration with the rest of the software. Then I track the number 
of bugs that make it past unit-test, the number that make it past 
integration test, and so on to the number of bugs that get 
reported by customers. These are the bugs that have a story to 
tell.

By looking into the nature of each bug – doing a root cause 
analysis – I then determine which phase actually originated the 
bug. For example, a bug caught in QA might have been introduced 
in any of the prior phases.

Here is a set of those metrics from an XP team building embedded 
software:

Defect inserted in:   How many    % of total
Requirements           2            7
Design                14           47
Code                  14           47
TOTAL                 30          100

These numbers cover approximately an 18 month period, so the team 
was averaging about 1.6 bugs per month. 

Here's one way to compute defect rate:
(Total Defects/ Total lines of code delivered) * 100  gives the 
% defect rate

In the example of the above team, the defect rate was 0.17% for 
the whole project, written mainly in C.

How does this compare with the rest of the software industry, 
embedded or otherwise? Very good question! There are practically 
no statistics publicly available. There is advice generally given 
that says you can stop testing and ship your code when you get it 
down to 15 defects per 1,000 lines of code. That's a defect rate 
of 1.5% - an order of magnitude worse than the XP team's example 
described above! 

What is your software's defect rate? Are you willing to share that 
info? I'm serious about collecting data on embedded software 
defect rates, whether agile or not. You can reach me at 
nancyv at agilerules.com.

                 O><O><O><O><O><O
More articles on Agile software topics at http://www.agilerules.com
Within our company we have a sub-specialty in embedded systems.
Our site has articles on embedded XP and we support a discussion
list focused on the use of agile methods for building embedded
software. The list signup info is at
http://www.agilerules.com/mailinglists.phtml

                 O><O><O><O><O><O
To help you get started with in-depth research into Agile Attitudes
topics, we have added a Library section to our web site at
http://www.agilerules.com/library.phtml
Order using our links and receive discounts up to 30%!

                 O><O><O><O><O><O
If you enjoyed this issue or found it useful, forward it to a
friend! Help spread the word about better ways to build software.
Invite your friends and colleagues to join our growing reader
community at http://www.agilerules.com/mailinglists.phtml

                 O><O><O><O><O><O
Looking for a speaker for your next corporate or society meeting?
We present dynamic, informative programs on topics of interest
to managers and technical staff in their transition to more
flexible, robust ways to create software.

                 O><O><O><O><O><O
Want to reprint this issue in your company or society newsletter?
For permission to reprint any of the articles, contact us at
info at agilerules.com.

                 O><O><O><O><O><O

If you would like to receive an email as soon as we know of
an event in the Boston area of interest to the agile software
community, you can sign up for the announcements list at
http://www.agilerules.com/mailman/listinfo/agileannounce

                 O><O><O><O><O><O

Your feedback is welcome! Send feedback to info at agilerules.com

To manage your subscription:
http://www.agilerules.com/mailman/listinfo/agileattitudes

                 O><O><O><O><O><O
    Brought to you by Agile Rules consulting
      162 Marrett Road, Lexington MA 02421

      Copyright (c) 2004, 2005 Agile Rules
               info at agilerules.com

                 O><O><O><O><O><O
Privacy notice: We will not release a subscriber's address to any
third party for any reason. This is a strictly opt-in newsletter.
No one is ever subscribed without their explicit request.