One of our biggest frustrations we have in our industry is coming across organizations that have been taken advantage of by former security contractors. The biggest and most scandalous trend in today's InfoSec marketplace is the resale of canned or massaged Nessus and Qualys scan reports for ridiculous sums of money. We've seen some clients pay over $20k for a single scan which probably delivered nothing more than $10 in value to their organizations. The reason this is happening so often, is because many organizations don't know what to ask or look for when it comes to hiring security talent. Hopefully, we can rectify these issues in today's blog post.

In this post, we'll go over some of the most important questions you should be asking your next penetration tester. We feel that the answers to these questions can help you identify faux security practitioners and make the right decision when selecting your next security contractor. Of course, this list isn't comprehensive, but we think it's a good start.

1. How much of your testing is manual versus automated?

This is probably one of the most important questions you should ask every security practitioner. The ratio that they advertise will give you a good sense of how capable an individual is when it comes to assessing your systems. If their answer is anything less than 80% manual, then you've probably got a (bad) tool monkey on your hands.

Good security practitioners usually boast about how "hard core" their penetration testing techniques are and can geek out for hours. They'll go in-depth into how they found an exotic 0-day vulnerability in some vendor's software or talk about the latest technology. If you're not technically savvy, a good measuring stick is whether or not the security practitioner sounds like Sheldon from the "Big Bang Theory". Enthusiasm like this is a sure sign that you've got someone highly capable on your hands.

If however, the talk primarily revolves around the use of tools and scanners, then that's a sure sign that you're probably dealing with a Nessus reseller and you'll probably be paying $20k for something that could have cost your organization $1.2k.

Another good sign of capability is whether or not they're asking the right questions in your discussions. They should seem curious, have an appetite to learn more about your business, its technology, processes, etc. Good hackers are like good detectives, turning over every stone.

Why is this important? Don't get us wrong, tools are a great complement to the penetration testing process but they miss out on a lot of things. Tools are only capable of finding really simple vulnerabilities and this is because they don't have the ability to think outside the box as your talented security practitioner would. If we relied on tools alone, our clients would have not been able to address over %60 of the most critical vulnerabilities in their systems.

Having the capability to perform manual testing requires a deep and intimate understanding of technology. Being able to hack a system requires practitioners to apply that same knowledge creatively.

2. What tools will you be using?

Yes, some of you security aficionados may be cringing right now but I feel this is a valid concern. A good security practitioner will be quick to point out that it's not about the tools, it's about the process. When they eventually get around to answering your question you may hear some of their favourites like BurpSuite, Nmap, or MetaSploit, to name a few. Mention of any of these tools is a sign that your practitioner may know a thing or two about penetration testing.

If you get a very short response like "Nessus" or "Kali", then chances are they're going to click a big green button and send you the invoice the next day. You should also ask them why they use these tools. For example:

BurpSuite is used for low-level web application testing. It includes an automated scanner but capable web application security practitioners spend their time performing manual testing in the "Repeater" tab.
Nmap is a sophisticated network port scanner that is capable of performing many tasks including host and service discovery, fingerprinting, and vulnerability identification.
MetaSploit is the swiss army knife for penetration testers. It includes an impressive arsenal of exploits but most pen-testers use it for it's powerful and flexible exploit development framework.

If they take the time to answer your question thoroughly then that demonstrates a solid understanding of the tool's capabilities and uses. If you are left confused, then chances are they don't even know what the tool is used for. For example, Nmap is not a SQL injection scanner and anybody that says so should be shown the door.

3. Has your firm ever provided solutions integration to our organization?

If so, then you may have a conflict of interest on your hands. Why do I say that? I can't tell you the countless number of times a managed security services provider has claimed they've detected and blocked malicious activity. Only to find out later that we were able to breach the perimeter either because their security operations centre team were sleeping on the job or their security solution was incorrectly configured. Or perhaps, the time one of my clients was told by their software development solutions provider that they've already performed security testing and "don't need another QA test". Only to be surprised later that their software solution was full of critical vulnerabilities.

If they've provided the solution, they shouldn't be testing its security for you. Period. Get a third-party that specializes in security assessments to test your systems. Someone who's neck isn't on the line if they identify a critical vulnerability that may set back the project. After all, it's your sensitive information assets that are at risk, not theirs.

4. How do you approach a penetration test?

A firm with no methodology is a firm without a plan of attack. A firm without a plan of attack is a firm that runs random scans and will most likely down your infrastructure. Do you want someone randomly scanning your enterprise with all the dangerous settings enabled, having no understanding of what they're doing, hoping to hit it big? I'd hope not.

Despite what some penetration testers may say, penetration tests do follow a general process or lifecycle:

Dicovery/Reconnaissance: good firms will boast about their intelligence-driven methodology. This is the phase where they get everything they need to know about your organization, its systems, personnel, online services, etc. Believe it or not, organizations unknowingly volunteer many precious nuggets of information to the public domain which can significantly increase an attacker's chances of successfully penetrating your organization's perimeter without using an exploit.
Scanning/Probing: this is where your practitioner's capabilities are put to the real test. Both a manual and automated process is used to identify vulnerabilities. If they're doing their job right, the automated testing should run in parallel to and complement their manual testing. This is where the tester is required to apply their technical expertise and creative thinking to identify potential vulnerabilities in your systems, services, software, etc.
Exploitation: once they've identified an issue it's time to try and exploit it. A large number of security practitioners fail in this phase because they are heavily tool reliant and don't know how to get around minor road blocks such as target platform variances or anti-virus. Exploiting vulnerabilities will involve compromising the confidentiality, integrity, and in some cases availability of a system. So make sure your security practitioner highlights those risks and the possible impacts to your business in your conversations with them.
Post-Exploitation: once a vulnerability is exploited, the access or information obtained in the previous phase is used to gain additional access to a system or resource and the cycle repeats itself. If your security practitioner mentions "pivoting" or "passing the hash" then it's a good indication they know what they're doing.

5. How will you be reporting your findings?

A mature security practice will ask for a few things in advance of their penetration test:

Multiple emergency points of contact. Let's face it - a real penetration test involves risky activities that may inadvertently down a system or two that your organization may rely on to power its business. This is why it's absolutely essential to exchange contact information for individuals that are available 24/7, prior to any penetration testing activity, in order to minimize the impact of an unsuccessful exploitation attempt.
The frequency of communication should also be agreed upon. This is something you can negotiate with your penetration tester. We highly recommend communicating updates once at the beginning, and once at the end of the week.
The method of communication is also crucial. You wouldn't want John Doe on the Internet knowing your soft spots. Encrypted email (PGP) or physical onsite presence are both acceptable ways of communicating findings. Anything less, would be uncivilized :).
Finally, the report document. This is where a lot of firms don't do a good job. In some cases, we've seen security practitioners deliver an Excel spreadsheet - yikes! At the end of the day, you want to be able to communicate the urgency of your findings to C-level executives so they can make good business decisions. If your practitioner is going to deliver a report that cannot be ingested by your executives, then you've completely thrown away the value of the security engagement. Ask for a template report, evaluate it's effectiveness in communicating findings at both a technical and executive level. This will also help you easily identify scanner generated reports. If they refuse, then this is a sign that you'll probably be getting someone else's report.

Conclusion

You may be asking yourself, "how much should a good penetration test cost?". A good penetration test doesn't come cheap and miracles don't exist in the InfoSec market. If it sounds too good to be true, then it probably is. In order to answer this question accurately we need to look at scope. If you want to do everything under the sun (penetration test, WiFi assessment, secure source code review, etc.), then be prepared to spend hundred's of thousands of dollars if not millions over a long period of time (years).

If you're new to this game and have a limited budget, a better approach is to test things incrementally. Getting a penetration test done is a great first step. Organizations should budget around $40k minimum if they want to get a good overview of their general attack surface. As your security practice matures, look into implementing security awareness programs, assessing applications or solutions prior to production deployment, or training your software developers to code securely. A small step in the right direction can go a long way towards filtering out the unsophisticated threats.

Hopefully this brief overview of what to look for in a penetration test will help your organization select their next top candidate. If you have any other questions please feel free to contact us via email or telephone +1-613-255-0000 and we'd be more than happy to assist.