AgileAttitudes Article
 
 
 

Vol 02 Issue 03- Lets Look at the Embedded Software Quality Numbers

  
 
Back to the list of articles Agile Attitudes Volume 2, Issue 3 Feb. 8, 2005 A free bi-weekly email newsletter Brought to you by Agile Rules consulting www.agilerules.com ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Welcome to Agile Attitudes, a newsletter of ideas, insights and technical tips that help people find better ways to develop software. Feel free to share this with anyone - just be sure you send or print the whole thing, including the copyright notice. Directions for managing your subscription are below. O><O><O><O><O><O XP Day - Toronto is a one-day conference happening on Saturday, Feb 19th. The cost to attend is only $299 (Canadian). For info, see http://www.xpday.info O><O><O><O><O><O Let's Look at the Embedded Software Quality Numbers by Nancy Van Schooenderwoert Agile software development is all about feedback loops. Programmers get feedback on how well their code is working by running sets of tests, ideally several times per hour. Management gets feedback on how well the software team is doing because they exercise the code delivered at each iteration – typically one or more times in a month. Customers get feedback on how usable the code is at every release, and they give feedback to the project team. Even with all this feedback going on, some bugs do still happen. Perhaps none would if everyone on the team used XP practices perfectly – always writing tests for every bit of code, always refactoring promptly, always being careful to really understand what the customer is asking for (if the customer's not writing acceptance tests). The most fundamental feedback loop is the one that tells the developers how well their coding practices are working. I don't mean merely a unit test of the function just written – I mean feedback that the coding practices and techniques are performing as expected. The best indicator of that is the bugs that get through your defenses. I'm accustomed to bug statistics that are collected at each of the waterfall phases; so many defects inserted in the requirements stage, so many in design, etc. but when you use XP those early phases are rolled into a tight ball. If your design-code-test cycle is 3 minutes long then stopping to log bugs at every compile is just goofy. What I do is log them at the next natural pause – the point where unit testing is declared complete and the code is ready for integration with the rest of the software. Then I track the number of bugs that make it past unit-test, the number that make it past integration test, and so on to the number of bugs that get reported by customers. These are the bugs that have a story to tell. By looking into the nature of each bug – doing a root cause analysis – I then determine which phase actually originated the bug. For example, a bug caught in QA might have been introduced in any of the prior phases. Here is a set of those metrics from an XP team building embedded software: Defect inserted in: How many % of total Requirements 2 7 Design 14 47 Code 14 47 TOTAL 30 100 These numbers cover approximately an 18 month period, so the team was averaging about 1.6 bugs per month. Here's one way to compute defect rate: (Total Defects/ Total lines of code delivered) * 100 gives the % defect rate In the example of the above team, the defect rate was 0.17% for the whole project, written mainly in C. How does this compare with the rest of the software industry, embedded or otherwise? Very good question! There are practically no statistics publicly available. There is advice generally given that says you can stop testing and ship your code when you get it down to 15 defects per 1,000 lines of code. That's a defect rate of 1.5% - an order of magnitude worse than the XP team's example described above! What is your software's defect rate? Are you willing to share that info? I'm serious about collecting data on embedded software defect rates, whether agile or not. You can reach me at nancyv@agilerules.com. O><O><O><O><O><O More articles on Agile software topics at http://www.agilerules.com Within our company we have a sub-specialty in embedded systems. Our site has articles on embedded XP and we support a discussion list focused on the use of agile methods for building embedded software. The list signup info is at http://www.agilerules.com/mailinglists.phtml O><O><O><O><O><O To help you get started with in-depth research into Agile Attitudes topics, we have added a Library section to our web site at http://www.agilerules.com/library.phtml Order using our links and receive discounts up to 30%! O><O><O><O><O><O If you enjoyed this issue or found it useful, forward it to a friend! Help spread the word about better ways to build software. Invite your friends and colleagues to join our growing reader community at http://www.agilerules.com/mailinglists.phtml O><O><O><O><O><O Looking for a speaker for your next corporate or society meeting? We present dynamic, informative programs on topics of interest to managers and technical staff in their transition to more flexible, robust ways to create software. O><O><O><O><O><O Want to reprint this issue in your company or society newsletter? For permission to reprint any of the articles, contact us at info@agilerules.com. O><O><O><O><O><O If you would like to receive an email as soon as we know of an event in the Boston area of interest to the agile software community, you can sign up for the announcements list at http://www.agilerules.com/mailman/listinfo/agileannounce O><O><O><O><O><O Your feedback is welcome! Send feedback to info@agilerules.com To manage your subscription: http://www.agilerules.com/mailman/listinfo/agileattitudes O><O><O><O><O><O Brought to you by Agile Rules consulting 162 Marrett Road, Lexington MA 02421 Copyright (c) 2004, 2005 Agile Rules info@agilerules.com O><O><O><O><O><O Privacy notice: We will not release a subscriber's address to any third party for any reason. This is a strictly opt-in newsletter. No one is ever subscribed without their explicit request. </plaintext> </td id="bodytable_r1_c3_body"> </tr id="bodytable_r1"> </table id="bodytable">