Food for Thought: How Do You Define Software Quality? An e-newsletter published by Software Quality Consulting, Inc. December 2004, Vol. 1 No. 4 To view a web version of this newsletter, click on the following link: http://www.swqual.com/newsletter/vol1/no4/vol1no4.html. --------------------------------------------------------------------------------- Welcome to Food for Thought(TM), an e-newsletter from Software Quality Consulting (http://www.swqual.com/). I've created free subscriptions for my valued business contacts. If you find this newsletter informative, I encourage you to continue reading. Feel free to pass this newsletter along to colleagues by clicking this Forward to a Friend link. If you've received this newsletter from a colleague and would like to subscribe, please click this Enter New Subscription link. If you don't wish to receive this newsletter, click the SafeUnSubscribe(TM) link at the bottom of this newsletter, and you won't be bothered again. Your continued feedback on this newsletter is most welcome. Please send your comments and suggestions to info@swqual.com. Wishing you and your family a safe and joyous holiday and a prosperous New Year... --------------------------------------------------------------------------------- In This Months' Topic, we discuss the difficulty of defining and measuring a most important term -- software quality. Regular features to look for each month are: - Monthly Morsel: Hints, tips, techniques and reference info related to this month's topic. - Calendar: Conferences, workshops, and meetings of interest to software engineers, QA engineers and anyone interested in software development. ---------------------------------------------------------------------------------- ***How Do You Define Software Quality?*** Software quality has been a constant topic of concern in the software industry. Everyday, in almost every part of the world, company's large and small struggle with issues related to software quality. Think about the situation at your company: - How many discussions have you had with your Development Team, your QA Team, with Sr. Management, with the Board of Directors, etc., about the quality of your software? - Does your company have a definition of software quality and if so, is it measurable? - How satisfied are your customers with the quality of your software? When was the last time you measured customer satisfaction? Most software companies spend considerable time and money - directly and indirectly - dealing with this issue. Software quality is often a crucial factor in many critical business decisions made on a daily basis. Yet, we have a hard time defining exactly what software quality means. We often use release criteria as a substitute for a good definition. YOU CAN'T IMPROVE WHAT YOU CAN'T MEASURE We talk about issues related to software quality all the time but yet we don't have a definition of precisely what it means. Especially since we should all know by now that we can't improve what we can't measure. Without a clear, concise, measurable definition of software quality, how can we make good business decisions regarding the use of limited engineering resources, areas of focus for quality improvement, tools and techniques that might be appropriate for quality improvement, etc. Well, we can't. This leads me to state what I believe is a basic principle of software quality definitions: Regardless of how software quality is defined, for the definition to be meaningful, it must be measurable. There are many different viewpoints on the issue of defining software quality. And they seem to fall into a few different groupings. Here are a sampling of several definitions from highly respected software engineering gurus: Conformance to Requirements Definition: Roger Pressman defines software quality as: - "Conformance to explicitly defined functional and performance requirements, explicitly documented development standards, and implicit characteristics that are expected of professionally developed software. - When we examine an item based on its measurable characteristics, two kinds of quality may be encountered: quality of design and quality of conformance. Quality of design refers to characteristics designers specify for items. Grade of materials, tolerances, performance specifications all contribute to the quality of design. Quality of conformance is degree to which design specs are followed during manufacturing. Again, greater degree of conformance, higher level of quality of conformance." [1] Customer-centric Definitions: Watts Humphrey says: "The principal focus of any software quality definition should be the users' needs. Crosby [6] defines quality as 'conformance to requirements.' While one can debate the distinction between requirements, needs, and wants, quality definitions must consider the users' perspectives. The key questions then are, who are the users, what is important to them, and how do their priorities relate to the way you build, package, and support your products?" [2] Al Davis defines quality as: "Quality isn't about zero defects or measurable improvements in defect rates, nor is it about meeting documented requirements. It's no more and no less than satisfying customer needs (whether or not the needs are properly documented)." [5] Conformance to Requirements AND Customer-centric Definitions: Capers Jones defines quality as: "The term quality will be used to mean software that has these six attributes: - Low levels of defects when deployed, ideally approaching zero defects - High reliability, or capability of running without crashes or strange results - A majority of users expressing satisfaction with software when surveyed - A structure that minimizes bad fixes or insertion of new defects during repairs - Effective customer support when problems do occur Rapid repairs for defects, especially for high-severity defects" [3] The IEEE Glossary of Software Engineering Terminology [4] defines quality as: (1) The degree to which a system, component, or process meets specified requirements. (2) The degree to which a system, component, or process meets customer or user needs or expectations. Touchy-feely Definitions: Jerry Weinberg defines quality as: "Quality is value to some person." [9] James Bach, Ed Yourdon and others have pushed the notion that software quality needs to be "Good Enough". To claim that any given thing is Good Enough is to agree with the following propositions: - It has sufficient benefits. - It has no critical problems. - The benefits sufficiently outweigh the problems. - In the present situation, and all things considered, further improvement would be more harmful than helpful." [7] "-ilities" Definitions: Robert Glass says: "Quality is a collection of 'ilities': - reliability - the ability to operate error free - modifiability - the ability to have enhancement changes made easily - understandability - the ability to understand the software readily, in order to change/fix it - efficiency - the speed and compactness of the software - usability - the ability to use the software easily - testability - the ability to construct and execute test cases easily - portability - the ability to move the software easily from one environment to another" [8] Are you totally confused by all of these different definitions? I sure am... GIVE ME A DEFINITION I CAN USE! How is it possible that so many smart people can have such different views of such a simple term? Maybe quality isn't so simple after all. Or, have we made it more complicated than it needs to be? As part of research I'm doing for a book, I recently asked some of my fellow consultants the following questions: Are definitions that are not measurable worthwhile? Here's what Karl Wiegers had to say: "I think definitions that aren't measurable are still worthwhile if they help create a shared focus of a project team on its technical and business objectives so they can work together toward a valuable outcome. Even if not quantitatively measurable, there still needs to be some way to assess whether a product or project has achieved its quality objectives. Therefore, a non-measurable definition can be worthwhile but is not adequate for making a definitive evaluation of whether a suitable level of quality is achieved." Should there be only one definition or many definitions depending upon the situation and context? Karl Wiegers said: "I doubt we'll ever have a single definition that fits all situations. A key aspect is that there are multiple dimensions of quality that need to be defined for any product. That's why vague definitions like Weinberg's 'quality is value to some person' aren't sufficiently helpful. Conformance to requirements (Crosby) assumes that the requirements are correct and complete, which isn't something I see very often!" What's a definition of software quality that makes the most sense to you? Here's what Robin Goldsmith said: "A system's quality is the extent to which it meets weighted stated and implied exterior, interior, and future REAL business requirements of all affected internal and external stakeholders consistent with standards of design, workmanship, and performance." Johanna Rothman told me: "I use Weinberg's 'Quality is value to someone'. I find that while it seems vague, it allows me to have a richer discussion about who all the someones are and what they find valuable. This definition also helps me define the context for quality in a given project. Once you start identifying your someones and what they find valuable, you can determine how to quantify that into release criteria, so you meet a minimum standard of project quality for your specific project." Johanna raises an interesting point - many organizations confuse quality and release criteria. (An example of release criteria - some number of P1 bugs). Release criteria should be thought of as a measurable objective derived from your definition of quality. A discussion with my colleague Bill Eventoff helped confirm a thought that I had about the definition. While there are those who believe conformance to requirements is not sufficient (because the requirements are usually flawed and don't reflect user needs), it is important to be able to measure the ability of an organization to build a product that conforms to requirements - even if those requirements are flawed. Getting the requirements right can be measured separately from the organization's ability to build a product that conforms to requirements. QUALITY OF BUILDING VS. QUALITY OF SPECIFYING Maybe the reason we don't agree on definitions of software quality is because we have made the term so all-encompassing that a meaningful definition has become difficult if not impossible. The ability to consistently develop software that is traceable to written requirements is good. If the requirements are flawed but the software matches the requirements, is that not a quality product? I would argue it is. What we need to do is separate the specifying from the developing. Let's measure the quality of developing while we work on improving the quality of specifying. And besides, if we had good quality of developing, then wouldn't the quality of specifying be captured in customer satisfaction surveys? THE BOTTOM LINE Every organization that develops software needs a working definition of software quality. How can you create a good definition? You need to take into consideration many factors such as customer's needs, industry expectations, regulatory requirements, business objectives, etc. With a good understanding of these factors, you can then develop your own definition as follows: First, your definition needs to meet the principle stated above - to be useful, it must be measurable. Second, recognize that, as James Bach observed, "quality is situational". In some situations, quality may be more important to the organization and to customers than in other situations. (Caveat - While this is generally the case for most software companies, it often isn't the case when software is life-supporting or life-threatening). Third, use the following distinctions to separate specification and development quality: - Development Quality is the ability to develop software based on and traceable to documented requirements. - Specification Quality is the ability to specify product requirements that represent real user needs and expectations and are consistent with overall business objectives. Then, using these distinctions: Formulate your definition for Development Quality based on using pieces of the definitions from Pressman and Jones and include as many of the "-ilities" as you deem appropriate. - Use activities such as Design Reviews, Code Inspections, requirements traceability, test results, and number of P1 bugs, etc., as measures. Formulate your definition for Specification Quality by using information from Karl Wiegers [10], Robin Goldsmith [11] and others. - Use activities such as Focus Groups, Usability Studies, and Customer Satisfaction surveys as measures. By defining and then measuring development quality and specification quality, we can focus process improvement activities on those areas where the problems really are - as indicated by these measures. Isn't this better than the meaningless definitions we often have? Using this paradigm, every software company would have different definitions. When it comes to software quality, one size clearly doesn't fit all. Well, that's my story and I'm sticking to it. Happy holidays! ------------------------------------------------------------------------------- ***Monthly Morsel*** Every month in this space you'll find additional information related to this month's topic. - Internet references: Click here to view a Wiki page on the subject of what is quality (http://c2.com/cgi/wiki?WhatIsQuality) ASQ Software Division Website (http://www.asq.org/perl/index.pl?g=softwareforum) Software Quality Group of New England Website (http://www.sqadvice.com/SQGNE_Home.htm) - Books and articles: [1] Pressman, R., Software Engineering: A Practitioner's Approach, McGraw-Hill, 4th edition, 1997. [2] Humphrey, W., A Discipline for Software Engineering, Addison-Wesley, 1995. [3] Jones, C., Software Quality: Analysis and Guidelines for Success, ITP, 1997. [4] IEEE Standard Glossary of Software Engineering Terminology, IEEE Std 610.12-1990, 1990. [5] Eickelman, N. and Hayes, J. eds., "New Year's Resolutions for Software Quality", IEEE Software, Jan-Feb 2004, pp. 12-13. [6] Crosby, P., Quality Without Tears, McGraw-Hill Book Company, 1984, p.64. [7] Bach. J., "Good Enough Quality - Beyond the Buzzword", IEEE Computer, August 1997. [8] Glass, R., "Revisiting the definition of software quality", StickyMinds Weekly Column From 10/08/01. [9] Weinberg, G., Quality Software Management - Systems Thinking, Dorset House, 1991. [10] Wiegers, K., Software Requirements, Microsoft Press, 1999. [11] Goldsmith, R., Discovering the REAL Business Requirements for Software Project Success, Artech House, 2004. --------------------------------------------------------------------------------- ***Calendar*** Every month, you'll find news here about local and national events that are of interest to the software community ... - Software Quality Calendar There are many organizations that sponsor monthly meetings, workshops, and conferences of interest to software professionals. Find out what's happening (http://www.swqual.com/links/upcoming.html) - Workshops Offered by Software Quality Consulting Software Quality Consulting offers workshops in many topics related to software process improvement. Get more info (http://www.swqual.com/seminars/courses.html) --------------------------------------------------------------------------------- ***About SQC*** Software Quality Consulting provides consulting, training, and auditing services tailored to meet the specific needs of clients. We help clients fine-tune their software development processes and improve the quality of their software products. The overall goal is to help clients achieve Predictable Software Development(TM) - so that organizations can consistently deliver quality software with promised features in the promised timeframe. To learn more about us and how we can help your organization, visit our web site (http://www.swqual.com/) or send us an email (info@swqual.com). I hope this newsletter has been informative and helpful. Your comments and feedback are most welcome. Send me your feedback... Happy Holidays, Steve Rakitin info@swqual.com Food for Thought and Predictable Software Development are trademarks of Software Quality Consulting, Inc. Copyright (c) 2004. Software Quality Consulting, Inc. All rights reserved. Graphic design by Sage Studio