Penetration Testing is the final word in proving that technical compliance and good security practices are in place – or so it should be. But how do you know if you’re getting a good service or not? What if the consultant performing the test is inexperienced? What is the impact on quality if the consultant is overworked? What if the consultant is an expert ‘hacker’, but terrible at report writing?
The trouble with asking questions like these is that there’s no tick box to check when choosing your supplier. An easier method is to ask if someone has CHECK or PCI accreditation. However, neither of these is a guarantee of quality.
Let’s just think about what quality means for a minute. Let’s quantify some aspects of this opaque practice.
Is it good quality for the consultant to do a quick portscan, and not cover all 65k ports for example? Doing a full port scan takes time, and usually turns up nothing, a quick portscan wouldn’t find. Maybe the consultant is on a tight timetable, with a test for another company pending. Is it good quality, to identify ‘autocomplete’ on an application as low risk, because that’s the standard classification, without taking in to account the context of the application and the business – e.g. a banking application? Is quality running an exploit on a vulnerable service, and then forgetting to clean it up afterwards, leaving the system in much less secure state than when you started?
Every penetration test is different, and just as there is no single IT specialist that knows every plat-form, every application and every network equally as well, security consultants have their strengths and weaknesses too. Is it reasonable to assume that an expert at testing Solaris, AIX, and other Unix flavours is also going to be equally as good on Windows? Or that a consultant who is great at testing networks, will be as good testing applications?
The truth is that most consultants have favourite platforms which they know at a deep level, and are either just competent or even incompetent with other platforms. Testing Windows RPC services for example, typically seems to provide a great challenge to many consultants. Just as you wouldn’t use a tractor on a racetrack, or a Ferrari in a field, you wouldn’t put a Unix expert on a windows test, or an Oracle expert on a MSSQL assignment. If you don’t take the time up front to properly determine the skills of the people you engage, not only have you wasted a significant amount of money, but you may have led yourself and your company in to a false sense of security.
Consultants hate report writing
The secret is out – consultants hate writing reports. Who can blame them? The ‘art’ of penetration testing is much more fun. The challenge and cerebral exercise attached to performing this, most technical of technical disciplines is alluring and perhaps slightly addictive to many. Imagine how boring it is to then type up a long document, cutting and pasting examples and screenshots, laboriously typing up the executive summary, and entering portscan results. It’s not fun, I’m sure, but let’s be clear about one thing, as a customer you don’t care if the technical assessment was the best in the world, if the report is incomplete, misleading, or just wrong. You don’t ‘see’ the assessment – you see the report!
The report IS the deliverable
Remember, it is the Executive Summary that you will show to your manager, the remediation ad-vice that you will give to your team, and the classified vulnerabilities that your auditor will review. The report is the deliverable, and when it is done poorly, then the brilliance of the assessment is somewhat eclipsed to say the least.
Strangely, it is known that consultants can fail to report on issues they did actually find! Perhaps this phenomenon is caused sometimes by time pressures, and sometimes by the consultant not reading the output of an automated tool correctly.
No doubt you’ve read, or at least skimmed through the “Methodology” paper on your suppliers web site, or their glossy brochure. It is designed to demonstrate a deep understanding of the assessment process. But aside from the comfort factor you may have felt after reading it, is the methodology actually applied, do the consultants follow it rigorously, or at all? A consultant can do an excellent job without following the company methodology, but by not having a structure to work with, there is a good chance the results will be inconsistent at best, and dangerously incomplete at worst.
5 steps to choosing a supplier
When choosing a supplier, there are a number of steps you can take which will give you a greater understanding of that vendors capabilities.
Ask if their consultants have passed an independent penetration testing assessment. There are some services that will independently test a consultant and rate their strengths and weaknesses in great detail. Always meet the consultants doing the proposed assessment and satisfy yourself about their competence. Ensure that only the consultants interviewed are the one’s carrying out the assessments. It’s easy to wheel in a star consultant to win the business, but follow through with a trainee. Ask to speak with reference sites, and actually follow through. Make sure the work carried out for those customers resembles your own. Ask questions about their methodology to discover if it is an active programme, or just marketing.
Finally, remember that companies don’t perform penetration tests, people do. So no matter which company you go to, it always boils down to the person you have working on your account. Make sure you always have the best people for the job in place, and remember that the best person for one job, may not be the best for another. Understanding the strengths and weak-nesses of your team is a fundamental part of good management. Extending this principal to your suppliers is just as valuable.