Talk:Testing Guide Introduction

The main issue I have is that the concept of "testing" proposed by this introduction ("testing is a process of comparing the state of something against a set of criteria") is significantly different from the generally accepted idea of testing. For example, the IEEE glossary defines test "An activity in which a system or component is executed under specified conditions, the results are observed or recorded and an evaluation is made of some aspect of the system or component." (http://www.fda.gov/ora/inspect_ref/igs/gloss.html). Most evidently (and for me, confusingly), this contrasts with the definition of manual inspection and reviews, threat modeling and source code review as testing techniques. However, I do understand that the objective of the guide is to provide a testing framework, rather than a simple testing technique/methodology (even if it seems to me that the text shows here and there the shift from the narrower to the wider meaning). In this context, those techniques could certainly be presented as techniques that aid (focus, guide, etc.) the proper testing activity. To make a long story short, I guess this boils down to renaming "testing techniques explained" to something like "techniques of the testing framework explained" and reframing the presentation of those techniques.

Second, even if it is hinted somewhere, it is never clearly stated that, in general, testing cannot find all vulnerabilities in an application. I would add a paragraph on this in the principles of testing section.

Third, the final section of source code review tools doesn't seem to be on a par with other parts of the guide: it promises to highlight fundamental issues but only briefly mentions the problem of false positives (what about false negatives?) and presents an (unexplained) example in C/C++ (which is not exactly the first language one would think of when discussing source code review tool for web applications). I would either expand it or move it as a paragraph of the section of "need of balanced approach", or merge it with the section titled "a note about web application scanner" to make it a more general "note about automated tools".

More down-to-the-text comments:
 * Is there any guideline for references: sometimes they are used inline, sometimes they presented at the end of the paragraph.
 * The first paragraph is somewhat disorganized. I would restructure it to say (in this order):
 * objectives (first sentence, the line starting with "This framework aims", and the discussion on the economics of insecure software)
 * two challenges (from "Writing the testing project has proven" to "integrated in the software development lifecycle")
 * The guide is good: "However, OWASP is able to take" to the end of the paragraph and "Many industry experts..."
 * Comments: the paragraph before "Principles of testing"
 * I would rename some titles:
 * Scope of this Document -> Why having a testing program?
 * The software development life cycle process -> When to apply testing?
 * The scope of what to test -> What to test?

Marco 04:58, 9 February 2007 (EST)