Push for Quality

This is a DRAFT page still under review by the Global Projects Committee

This page is maintained by the Global Projects Committee to help assist Project Leaders with information about successfully running an OWASP Project. It will be updated from time to time, and changes will be discussed and announced on the OWASP-Leaders list.

Overview
OWASP created the project assessment criteria to define the quality levels for OWASP Projects with the purpose of evaluating all OWASP projects. The overall goal was to ensure that consistent quality levels are maintained by OWASP projects. This benefits both the external audience and those working on projects. The criteria allows the external audience to determine the quality of any OWASP project they are considering. For project members, it provides a method to measure the quality of their project in relation to other OWASP projects. Additionally, the criteria allows for excellent contributions to be recognized and projects which need further work to be identified.

Currently, OWASP projects fall into three primary categories:
 * Tools
 * Documents
 * Activities and Research

The Tools and Documents categories are easily understood. The Activities and Research category is less obvious and is used for projects which either have multiple sub-projects or have deliverables which fall into both the tools and documents category. Thus, Activities and Research can be used for parent projects that cover multiple smaller sub-projects. Some examples will make this clearer:


 * OWASP ESAPI
 * Java
 * .Net
 * PHP


 * OWASP Guides
 * Testing Guide
 * Development Guide
 * Code Review Guide
 * ASDR (Application Security Desk Reference)


 * OWASP OpenPGP Extensions for HTTP - Enigform and mod_openpgp

All existing projects and their current ratings are here. Any new OWASP project and its deliverables will be assessed based on the criteria below as well as any Season of Code project. The goal is to eventually have all OWASP project deliverables, past and future, assessed under a version of this criteria. The initial set of assessment criteria was created for the OWASP Summer of Code 2008 and was designated version 1.0. The current version below was derived from version 1.0 and is version 2.0. Labelling any new criteria with a version number allows for graceful transitions to occur should any criteria change.

Assessing a project
Any OWASP project will consist of two critical pieces:
 * the project's OWASP Wiki page
 * one or more project deliverables

Each of these pieces will be have different methods with which they are reviewed. For OWASP project wiki pages, please see the Project Wiki Pages section of the Guidelines for OWASP Projects.

For project deliverables, OWASP has established the assessment criteria with three designations of quality: Alpha, Beta and Release. As project deliverables move up the quality ladder from Alpha to Beta and finally to Release quality, the amount of rigor required increases. In general, the project lead will determine the goal quality level of their project and work towards fulfilling the criteria for that level. Once a project lead has completed the prerequisites and criteria for the goal level, they request that their project be reviewed. The level will determine who and how those reviews occur.


 * Alpha quality: The review consists of the Global Project Committee (GPC) verifying that the project pre-assessment check-list is complete. Alpha projects are the most free and open since anyone with solution to an application security problem can propose a new OWASP project. Alpha is where all new projects begin.
 * Beta quality: The review will first be conducted by the project's reviewer (more on this below). After the reviewer completes the reviewer action items, the GPC will validate the project's assessment.
 * Release quality: The two project reviewers will complete their action items (more on this below). After the reviews are complete, the Global Projects Committee and OWASP Board will validate the project's review.

Pre-Assessment Checklists versus Reviewer Action Items:
 * Pre-Assessment checklists should be completed by the project lead prior to asking for an assessment.
 * Pre-Assessment checklists were designed to be completed in minutes if all items are complete.
 * Reviewer Action Items should be completed by project reviewer(s) after the project lead indicates the project is ready to be assessed.
 * Reviewer Action Items were designed to require some significant time commitment from the reviewer since the questions are subjective and require a good deal of understanding of the project.

Prerequisites for project assessment
Depending on the quality level criteria, the project lead may have prerequisites before the project deliverable(s) can be assessed by the criteria below


 * Alpha quality: No prerequisites.
 * Beta quality: 1 reviewer is required.
 * Release quality: 2 reviewers are required. Second review has special requirements.

Notes on reviewers


 * Ideally, the project lead will suggest project reviewer(s).
 * Ideally, reviewers should be an existing OWASP project lead or Chapter leader.
 * If the project lead is unable to find the required reviewer(s), the Global Projects Committee can assist in identifying reviewers for the project.
 * It is recommended that an OWASP board member or Global Projects Committee member be the second reviewer on Release quality deliverables. The board has the initial option to review a project, followed by the Global Projects Committee.
 * The Global Projects Committee confirms the assignment of reviewers to a project.

Alpha Quality Tool Criteria
Pre-Assessment Checklist:
 * 1) Is your tool licensed under an open source license?  (see Project Licensing section of the Guidelines for OWASP Projects)
 * 2) Is the Project Identification template on the project complete and accurate?
 * 3) Is the source code and any documentation available in an online project site? (e.g. Google Code or Sourceforge site)
 * 4) Is there an OWASP mail list for the project?
 * 5) Is there a statement of the application security issue the tool addresses on the OWASP project page?

Beta Quality Tool Criteria
Pre-Assessment Checklist:
 * 1) Is there an installer or stand-alone executable?
 * 2) Is there user documentation on the OWASP project wiki page?
 * 3) Is there an "About box" or similar help item which lists:
 * 4) Project Name
 * 5) Short Description
 * 6) Project Lead and contact information (e.g. email address)
 * 7) Project Contributors (if any)
 * 8) License
 * 9) Project Sponsors (if any)
 * 10) Release status and date assessed as Month-Year e.g. March 2009
 * 11) Link to OWASP Project Page (NOTE: I am unsure if this is needed --Mtesauro 21:06, 30 March 2009 (UTC))
 * 12) Is there documentation on how to build the tool from source including obtaining the source from the code repository?
 * 13) Is the tool documentation stored in the same repository as the source code?
 * 14) Are the Alpha pre-assessment items complete?

Reviewer Action Items:
 * 1) Is an installer for the tool available and easy to use? How close does it reach the goal of a fully automated installer?
 * 2) Is the end user documentation complete, relevant and presented on the OWASP wiki page?
 * 3) Does the tool have an “About box” or similar help item which allows the end user to get an overview of the state of this tool? Is this information readily available and easy to find?
 * 4) Does the documentation on building the source provide the necessary information and detail to allow someone to build the tool? Is there sufficient detail and information for the target user? Is there any domain specific knowledge that is assumed and not provided?
 * 5) Is the tool's documentation available with the source code and would it readily discoverable by a new user of the tool?

Release Quality Tool Criteria
Pre-Assessment Checklist:
 * 1) Does the tool include documentation built into the tool?
 * 2) Does the tool include build scripts to automate builds?
 * 3) Is there a publicly accessible bug tracking system?
 * 4) Have any limitations of the tool been documented?
 * 5) Is there a conference style presentation that describes the tool in at least 3 slides?
 * 6) Is there a one sheet overview document about the project?
 * 7) Are the Alpha and Beta pre-assessment items complete?

Reviewer Action Items:
 * 1) Does the tool substantially address the application security issues it was created to solve?
 * 2) Is the tool reasonably easy to use?
 * 3) Does the documentation meet the needs of the tool users and is easily found?
 * 4) Do the build scripts work as expected? Can you build the tool? The goal is a “One-click” build.
 * 5) Is the bug tracking system usable? Is it hosted at the same place as the source code? (e.g. Google Code, Sourceforge)
 * 6) Have you noted any limitations of the tool that are not already documented by the project lead.
 * 7) Have all the Beta Reviewer Action Items been completed?  These will need to be completed if they have not already occurred during a previous assessment.

Alpha Quality Document Criteria
Pre-Assessment Checklist:
 * 1) Is your document licensed under an open source license? (see Project Licensing section of the Guidelines for OWASP Projects)
 * 2) Is the following present on the OWASP project wiki page:
 * 3) Description of the document
 * 4) Project leader information
 * 5) Contact information
 * 6) A list of contributors or company logo/name if a team effort (NOTE: How is this going with the sponsorship model? - from Leo)
 * 7) A roadmap or timeline listing the steps to achieve the purpose of the project
 * 8) The "Alpha Quality Document" project tag - e.g. OWASP Alpha Quality Document
 * 9) The project information template.  See the Project Wiki Pages section of the Guidelines for OWASP Projects
 * 10) Is there an OWASP mail list for the project?
 * 11) Is there a statement of the application security issue the document addresses on the OWASP project wiki page?

Beta Quality Document Criteria
Pre-Assessment Checklist:
 * 1) Are all document contents (articles) present and listed on the OWASP project wiki page?
 * 2) Is there an “About” section or similar item in the document which lists: (NOTE: Is there a general template/structure for docs? - from Leo)
 * 3) Document Name
 * 4) Author(s)
 * 5) Contributor(s)
 * 6) Contact email address
 * 7) Current version and/or release data
 * 8) Project's web page address (NOTE: Should this exactly match the template material - from Leo)
 * 9) Is the tool documentation stored in the same repository as the source code? (NOTE:  Think this is a copy/paste error --Mtesauro 21:44, 30 March 2009 (UTC))

Reviewer Action Items:
 * 1) Is the document considering the OWASP Writing Style?
 * 2) Does the tool have an “About box” or similar help item which allows the end user to get an overview of the state of this tool? Is this information readily available and easy to find? (NOTE:  Think this is a copy/paste error --Mtesauro 21:44, 30 March 2009 (UTC))
 * 3) Is the tools documentation available with the source code and would it readily discoverable by a new user of the tool? (NOTE:  Think this is a copy/paste error --Mtesauro 21:44, 30 March 2009 (UTC))

Release Quality Document Criteria
Pre-Assessment Checklist:
 * 1) Have any limitations been documented?
 * 2) Is there a conference style presentation that describes the tool in at least 3 slides?
 * 3) Is there a one sheet overview document about the project?
 * 4) Is the document respecting OWASP Writing Style and OWASP Template for Docs?
 * 5) Are the Alpha and Beta pre-assessment items complete?

Reviewer Action Items:
 * 1) Does the document substantially address the application security issue(s) it was created to solve?
 * 2) Verify grammar, writing style and document template compliance.
 * 3) Have you noted any limitations of the document that are not already documented by the project lead.
 * 4) Have all the Beta Reviewer Action Items been completed?  These will need to be completed if they have not already occurred during a previous assessment.