Views

Enterprise:Engineering:SQA

Contents

[edit]

Navigation

Home · Main Page · Enterprise  ·


[edit]

Related categories



[edit]

About this page

We apologize for that little information on this page, which is still under construction. Please stay tuned.
Image:Construction_worker.gif

[edit]

Quality Management and Quality Assurance practice

On this page various resources and information of interest on Quality Management can be found, with special emphasis on Software Quality Assurance topics.

[edit]

Organizations

The Quality Assurance Institute was founded in 1980 in the US. Dedicated to improving quality, productivity, and effective solutions for process management in the information services profession. The American Society for Quality Assurance (ASQ) offers technologies, concepts, tools, and training to quality professionals, quality practitioners, and everyday consumers. The Usability Professionals' Association promotes the usability approach in product development. "I Six Sigma" grants Certifications and Awards (Balridge, etc) to individuals and organizations for achievements in Quality Assurance
  • Quality Assurance Institute [1]
  • Quality Assurance Institute Canada [2]
  • American Society for Quality Assurance [3]
  • The Usability Professionals' Association [4]
  • "I Six Sigma" promotes the adoption of Six Sigma methodology [5]


[edit]

Quality Management Resources

Some general resources for deploying the Quality function in enterprise:
  • Quality Assurance Institute Bookstore and training
  • General resources for QA & process improvement [6]
  • Deming Continuous Quality Improvement (CQI) resources [7]
  • BALDRIGE, SIX SIGMA, & ISO - Understanding the Options [8]
  • All about quality Gurus - lives and ideas [9]
  • Six Sigma, Quality Management & ISO Articles [10]
  • The Deming Center for Continuous Quality Improvement (CQI) [11]
  • Lloyd Register Quality Assurance (LRQA) ISO 9001:2000 assessment resources [12]
  • The W. Edwards Deming Institute [13]
  • Understanding and Implementing an ISO 9001:2000 or related Quality Management System [14]
  • TQM and Six-Sigma resources for CEOs [15]
  • ISO 9001:2000 resources courtesy of Elsmar Cove [16] and the standard ISO 9000:2000 local copy ISO 9001:2000 local copy
  • Deming's 14 points [17]


[edit]

Software quality assurance resources

Software Quality Assurance (SQA) has many facets, from processes, standards and methodologies to specialized toolsets. SQA involves defect-prevention as well as defect detection and correction and relies of strict configuration management procedures, coding standards, engineering methodologies and validation-verification (V&V) processes. The real problem in the world of SQA is the 'quagmire' of existing methodologies, frameworks, standards, conformance criterias, maturity models, many of them similar in many respects to each other. Due to market pressure, organizations producing or integrating software products tend to see a competitive factor in aligning themselves with a recognized generic- or software- quality-assurance framework. Software Quality Assurance is pervasive and is encompasses all activities related to product development, including Project Management, Requirements Management, Configurations Management and Engineering Practices. For this reason, some large organizations tend to adopt generic process-improvement framework, like ISO 9001:2000, CMMI,RUP or Fujitsu/DMR Macroscope. Other organizations tend to introduce in their processes best practices derived from SQA-specific standards issued by IEEE or ISO. Introducing a Software Quality Assurance framework and formally assessing compliance to it becomes important for organizations offering products or services to the financial sector, due to stringent security requirements. These organizations' products and processes should comply and be formally audited against such criteria as:
  • The "Evaluation Assurance Levels" (EAL) criteria defined by the "Common Criteria" ISO/IEC 15408 standard [18])
  • The Financial Institution Shared Assessments Program (FISAP) for IT outsourcer assessment [19]


Many software organizations have product testing departments which are named "Quality Assurance". In fact, Quality Assurance is much more than testing, it is mainly defect prevention by encompassing all phases and processes in the product life-cycle.

Here are a few pointers to general Software Quality Assurance and Software Testing resources.
  • The Software Quality Page [20]
  • Testing and Quality Control resources [21]
  • SQA resources, mainly related to ISO/IEC 12207, courtesy of Abelia [22]
  • Software QA Automated Testing Resources and the the AppTest Manager automated test-suite [23]
  • Software QA resources, mainly for testing and bug-tracking, and testers' forum [24]
  • Software Testing Hotlist with many Software testing and SQA resources [25]
  • SQA resources courtesy of Software Quality Engineering [26]
  • Interesting Software testing resources [27]
  • Software QA and Testing Resource Center [28]
  • SQA&T reference list [29]
  • Recommended Software Quality Assurance and Testing Tools [30]
  • The "Better Software" magazine [31]
  • US Air Force (USAF) Software Technology Support Center (STSC) [32]
  • Difference between Quality Management & Quality Audit
  • Balancing Time to Market and Quality [33]
  • Quality as a business decision [34]
  • Defensive programming [35]


[edit]

SQA Organizations

Here are a few pointers to organizations and interest groups dedicated to software quality assurance:
  • Systems and Software Consortium, Inc. (SSCI) [36]
  • International Software Testing Institute [37] with SIG's and Test Project Management resources
  • Society for Software Quality (SSQ) [38]
  • Society of Quality Assurance (SQA) [39]
  • The Build Security In (BSI) initiative promoting SQA practices for ensuring software security


[edit]

Recommended SQA Practices

Any organization that feels the need to introduce or improve its software quality-assurance practice should tailor it for its own reality. It is not a very good idea to implement a SQA "recipe" that was maybe a success story for other organization. Instead, once the basic principles are well understood, the implementation should start with those factors that would bring maximum business benefit at lowest cost.

References:

[edit]

Managing Defects in Software

Beside trying to prevent and detect as many as possible defects, managing the metrics related to defect type and density is useful for the control and continuous improvement of SQA processes. Software organizations have noticed that the type and density of defects being found in each phase of a product lifecycle should follow some patterns if the SQA processes are under control. Any anomaly relative to these expected defect patterns represent a signal that preventive and corrective measures are needed to bring back under control the deviant processes. Both the IBM-proprietary Orthogonal Defect Classification (ODC) methodology and the ISO/IEC 9126 standard take this approach. Large software-intensive organizations (e.g. IBM or Motorola) use the defect-type-density measures for controlling the SQA processes.

References on Orthogonal Defect Classification (ODC) and ISO/IEC 9126:
  • ODC overview and links [48]
  • ODC home page at IBM [49]
  • ODC detailed information courtesy of IBM [50]
  • Managing Software Quality With Defects [51] local copy
  • ISO/IEC 9126 - a standard for the software evaluation through quality metrics [52] part of SQuaRE project local copy
  • Measures and Measurement for Secure Software Development [53]


[edit]

Bug- and Issue-Tracking tools

Among the many free and commercial bug-tracking tools, Bugzilla and Mantis are the most popular in the open-source category. However, the selection of a particular bug-tracking toolset is a matter of correcting evaluationg the organization's needs, including integration within the already existing systems. Many organizations have ditched commercial toolsets (ClearCase/ClearQuest, Microsoft VSS or Merant PVCS/Tracker) in favor of open-source solutions, much easier to integrate within reasonable budgets and time-frames.

References:
  • Bug-tracking software directory [54]
  • Comparison of issue tracking systems [55]
  • APTest bug-tracking tools directory [56]
  • Defect-tracking tools list [57]
  • The Bugzilla Test Server [58] and RedHat bug-tracking system
  • Bugzilla home page [59] and guide
  • Mantis [60]
  • GNATS [61], the GNU project's bug-tracking toolset
  • Integrating Bugzilla and Subversion [62]
  • CVS and Bugzilla integration : [63] and [64]
  • Integration of Bugzilla with CVS and MediaWiki [65]


[edit]

Software Quality & Engineering standards

There are many process-improvement and quality frameworks, some of them pertaining more to the IT management practice than to the Software Quality Engineering. Other standards, e,g, the ANSI/IEEE ones address specific SQA practices like documentation, unit-testing, verification and validation or code inspections.

Specific Software engineering standards are focused on specific areas of the practice:
  • Processes - focusing on Software Engineering mechanisms and supporting activities (e.g. Life-Cycle, V&V, CM)
  • Artifacts - guidelines for outputs from processes and activities (e.g. requirements, design or test specifications)
  • Methodology - process/activity guidelines and procedures (e.g. unit-testing, quality-metrics assessment)


References on Software Quality Frameworks:
  • List of Software Engineering standards [66]
  • The Frameworks quagmire (in 1997, it worsened eversince) [67]
  • International standardization in software and systems engineering [68] local copy
  • International Standards for Software and Systems Engineering [69] local copy
  • Software Engineering Standards overview local copy
  • Looks like a joke but there are good reasons why standards live forever [70]


[edit]

Cleanroom Software Engineering

The Cleanroom Software Engineering, both an engineering methodology and management process, aims at delivering zero-defect, certified reliability products through:
  • Application of the SWEBOK Software Engineering methodology
  • Rigurous requirements gathering and analysis
  • Incremental development using small-teams responsible for specification, development and certification
  • Formal verification of the design against the specification
  • Extensive testing based on use scenarios, automatic test vectors and statistical defect control


[edit]

ISO/IEC 12207

The ISO/IEC 12207 standard aims at implementing the Total Quality Management principles in Software Engineering practice. To this avail, the standard establishes a top-level architecture for the Software Life Cycle, based on a interdependent processes.
  • Overview of International Standard ISO/IEC 12207 - Software Life Cycle Processes [71]
  • ISO/IEC 12207 Software Lifecycle Processes - a primer by Lewis Gray [72]
  • Information on ISO/IEC 12207 and its mapping to IEEE Software Engineering Standards Committee (SESC) Standards [73]
  • ISO 12207 and Related Software Life Cycle Standards [74]
  • Guidebook to ISO/IEC 12207 [75]


The ISO/IEC 12220 series provides detailed guidelined on the supporting processes of ISO/IEC 12207
  • ISO/IEC 12220-1: Overview Document;
  • ISO/IEC 12220-2: Software Configuration Management;
  • ISO/IEC 12220-3: Software Project Management;
  • ISO/IEC 12220-4: Software Quality Assurance;
  • ISO/IEC 12220-5: Software Verification and Validation;
  • ISO/IEC 12220-6: Software Reviews and Audits.


[edit]

ISO/IEC 14764

ISO/IEC 14764 Software Engineering Software Life Cycle Processes Maintenance covers the software-product maintenance domain, which is different from both operations and development domains. Maintaining ab organization's software portfolio requires a specific management system satisfying the customers' service criteria while maximizing the economic benefits. The Software Life Cycle begins with development and continues with maintenance and operation, one key characteristic of maintenance deliverable being the extremely short time-to-delivery, as it is usually the case with dealing with support tickets. While ISO/IEC 12207 covers the development process (including of new releases), ISO/IEC 14764 and IEEE STD 1219 covers the maintenance of a software product after delivery, mainly the necessary processes and activities to efficiently correct issues customer-reported issues.

References:
  • Harmonization of ISO/IEC 14764 and IEEE STD 1219 [76]
  • Software Maintenance Maturity (SMmm) Model to Evaluate and Improve the Quality of Software Maintenance Process [77]


[edit]

ISO/IEC 15504

The ISO/IEC 15504 standard, known as SPICE (Software Process Improvement and Capability Determination) is a "framework for the assessment of software processes", derived from ISO 12207 and using many of the CMMI concepts. The standard does not define a methodology, instead provides a set of auditable processes and activities as gudelines for areas to be assessed in order to determine an organisation's capabilities for delivering software.

References:
  • Introduction to Software Engineering Standards - the SPICE project [78]
  • ISO SPICE resoources [79]
  • The SPICE document suite [80]
  • ISO/IEC 15504 (SPICE) hotlist [81]
  • CMM, CMMI and ISO/IEC 15504 (SPICE) [82]


[edit]

ISO/IEC 15939

The ISO/IEC 15939, "Software Engineering - Software Measurement Process", was developed based on the Practical Software Measurement (PSM) guidelines pioneered by the US DoD. However, compared to ISO/IEC 15939, the PSM provides much additional details on the processes presented in the standard, including detailed steps and guidance on how to meet these processes' requirements. The guidance provided by PSM is very detailed, including sample measures, lessons learned, case studies, and implementation guidance. The PSM and the ISO/IEC 15939 standard were used as input for the Measurement and Analysis (MA) process area of the CMMI and the Software Engineering Measurement and Analysis (SEMA) initiative.

[edit]

ANSI/IEEE SESC SQA standards

The following ANSI/IEEE standards cover the process areas involved in the development phase of the Software Life Cycle:
  • The IEEE Software Engineering Standards Collection 2003 on CDROM
    • ANSI/IEEE Standard for Software Quality Assurance Plans [83]
    • ANSI/IEEE Standard for Software Test Documentation [84]
    • ANSI/IEEE Standard Dictionary of Measures to Produce Reliable Software [85]
    • ANSI/IEEE Standard for Software Unit Testing [86]
    • ANSI/IEEE Standard for Software Verification and Validation Plans [87]
    • ANSI/IEEE Standard for Software Reviews and Audits [88]
  • The IEEE Software Engineering Standards Collection - 1993 edition [89]


[edit]

Code Review and Inspections

Inspecting delivered artifacts in all phases of the development cycle is essential for minimizing the number of issues (and hence the maintenance costs) once the products are shipped in the field. The areas targeted of inspections vary and comply to generic requirements (e.g. security, requirements tracing) or specific (e.g compliance to coding standards or testing requirements). The success of the code-review practice is conditioned by the implementation of sound inspection practices for requirements and design documentation. One example of code-review topic to look at is the presence of diagnostic hooks and logging that would allow for efficient troubleshooting of issues by support personnel. Another example would be to inspect for adherence to a set of secure-coding practices, as required by the Common Criteria for product compliance to an Evaluation Assurance Level. Manual code-reviews are notoriously slow, error-prone and resource-intensive and, for this reason, a few improved, semi-automated practices emerged:
  • First and above all use an "unforgiving" programming language (e.g Java) that minimizes the probability of undetected errors.
  • Use static code-analysis tools that would catch a great number of potential issues and would also indicate hot-spots, usually characterized by high cyclomatic complexity scores.
  • Focus manual code-review first on the high-risk areas, usually those flagged by the analysis tools with high cyclomatic-complexity scores.
  • Use a system of classification for the types of errors found during the reviews, possibly finding inspiration in the Orthogonal Defect Classification model.
  • Based on previous code-review results, take as hypothesis that some programmers might have personal propensities for certain types of errors and check first for those.


Code Review references :
  • Security Code Review Guidelines [90] and [91]
  • Goodies for Peer Reviews [92]
  • Construx peer review guidelines [93]
  • Formal Inspections - a software acquisition gold practice [94]


Software "cyclomatic complexity" metrics :
  • Software complexity metrics & models (Halstead & McCabe) [95]
  • Structured Testing Methodology using Cyclomatic Complexity Metric [96] local copy


Code Review and SQA Process Support tools :
  • Codestriker - a CVS-compliant code-review tool[97]
  • Jupiter - a Code Review plugin for Eclipse [98]
  • CodePro Analytix - an automatic code-review tool for static code and complexity metics analysis, coverage and JUnit test generation
  • Tools for code analysis, inspections and review [99]
  • Wikipedia page on static code analysis tools
  • Process Assets for Software Peer Reviews and Inspections [100] - templates, checklists, processes
  • Linux SQA tools directory - configuration Management, Code Review, Issue Tracking [101]
  • The Hammurapi Quality Assurance Open Source tools
    • The set of available too,s [102]
    • The Hammurapi Manifesto
    • The Hammurapi "Overarching QA principles" [103]


[edit]

Software quality assurance articles

A few interesting articles on Software Quality ASsurance, touching many topics of the domain:

[edit]

Software Testing

Beside the measures meant to prevent defects in software, by carefuly following a ser of best practices during the development phase, the testing processes are essential for minimizing the number and impact of undetected defects in shipped software.

[edit]

Testing Practices and resources



[edit]

Risk-based testing

Pareto's 80-20 principle states that as much as 80% of the consequences can be traced back to only 20% of the causes. Pareto stated his principle though in an economic context, observing the economic disparities, more precisely observing than 80% of the wealth is owned by 20% of the population. Dr. Joseph Juran, working in the US in the 1930s and 40s stated this universal principle, as the "vital few and trivial many" and naming it "Pareto's principle". Pareto's Principle, serves as a reminder to focus 80 percent of time and energy on 20 percent of work that is really important, i.e. "work smart" on the things that matter. The Risk-based testing takes further this principle, defining a methodology of finding the most important defects as early as possible at the lowest price. The Risk-based testing would rely on a risk analysis and on the evaluation of business impacts defects may have, based on the framework defined in ISO/IEC 9126.

References:
  • Risk based testing primer by Hans Schaefer [120]
  • Heuristic Risk-Based Testing [121]
  • Troubleshooting Risk-Based Testing [122]
  • Incomplete list of testing tools [123]
  • A Strategy for Risk-Based Testing by Stephane Besson [124]
  • Incomplete tests are worse than none at all [125]


[edit]

Test Maturity Model (TMM)

  • Test Maturity Model overview [126]
  • TMM compared to other test improvement models [127]
  • TMM resources [128]
  • The TMMi Foundation dedicated to improving test processes and practice
  • TIM - a Test Improvement Model [129]
  • Using the Software-TMM [130]


[edit]

Software Test automation



[edit]

Embedded Software Quality Assurance

Embedded software drives many mission-critical systems (e.g. telephony switches, car brake-systems or even fighter jets) and, for this reason, the quality assurance practice is of paramount importance.

References:

[edit]

ISO/IEC 10303 (STEP) and Model-based design validation & verification

The standard ISO 10303, named STEP (Standard for the Exchange of Product Model Data), describes how to represent and exchange digitally the product design and manufacturing information. It is divided in several parts and is widely used in the industry for realizing Integrating manufacturing systems. Having a standard representation of product model data enables the automated model verification.

References:
  • Information and tutorials about ISO 10303 STEP [139]
  • Technical report ISO/TR 10303-303(E) - Application protocol - Configuration controlled 3D design of mechanical parts and assemblies local copy
  • STEP - An International Standard for Data Exchange - presentation [140]
  • The ISO10303-AP233 Exchange standard [141]
  • NASA STEP resources - using STEP (ISO 10303) for CAD/CAM/CAE Data Exchange [142]
  • The DoD Architecture Framework (DODAF) and STEP AP233 [143] local copy
  • The Model-based Development & Verification Environment [144]
  • Data Exchange Standards and Ontologies for Engineering [145]
Retrieved from "http://www.mindtrek.dnsalias.net/components/com_mambowiki/index.php/Enterprise:Engineering:SQA"