Selection of penetration testing methodologies: a comparison and evaluation

Classification 
Suitability 
Candidate 
Framework 
Methodology 
Other
Penetration Testing 
Specific
Security 
Assessment
ISSAF 




OSSTMM 






OTG 




BSIMM 




PTES 




MSF 




Pre-evaluation Classification
*
Post-evaluation Classification
+
Legend
68

---

Selecting candidates for the evaluation of quality was determined by two criteria. First, whether or not a 
particular candidate classified as either framework or methodology. Candidates that fall under a methodology or 
framework were deemed in-scope, thus eliminating all other candidates. The second criterion was to examine the 
boundaries of a particular candidate for its scope, in other words, whether or not a particular candidate is focused 
entirely on penetration testing as opposed to an overall security assessment. The research being undertaken has a 
primary goal of evaluating penetration testing methodologies and frameworks explicitly rather than assessing the 
security posture of an organisation in its entirety, therefore candidates that are categorised as penetration testing 
explicitly are preferred over security assessment specific candidates. As a result the two remaining candidates 
are ISSAF and OTG.
Next, quality characteristics were nominated (see figure 2), for the purpose of evaluating the refined subset of 
frameworks. Two factors were taken into consideration for selection of quality characteristics. First, the field of 
study or context from which a characteristic definition was drawn (in particular that of information systems was 
preferred); and second, whether or not a particular quality characteristic was directly applicable to the field of 
penetration testing. 
Figure 2: Penetration Testing Quality Model (adapted from ISO/IEC 25010:2013). 
 
From the revised taxonomy shown in table 3, the selected quality metrics are applicable to both frameworks.
Both frameworks display evidence of the quality characteristics applicable to penetration testing, therefore the 
six quality characteristics selected are considered suitable, thus will be used to facilitate this research in 
evaluating efficiency for the two chosen candidates, ISSAF and OTG. Note that reliability, whilst a valid 
characteristic, was not tested in this evaluation due to lack of delivery of an expected real-world case study. 
Table 3: Quality Matrix of Selected Penetration Testing Frameworks. 
69

---