How to Evaluate Web Application Security Scanners

Ask 20 penetration testers which web application security scanner they prefer and use and you will get 20 different answers, if not more. Every web vulnerability scanner has its own pros and cons and what works for Mr X does not necessarily work for Mr Y, almost like everything else. Therefore you shouldn't simply base your purchasing decision, or build a web application security program on what colleagues say or think. You have to do some testing and come up with your own decision.

This guide will explain how to evaluate web application security scanners and help you choose the right web security tool that fits your requirements.

Make a List of Requirements

Before getting your hands dirty with web security scanning, compile a list of the requirements you currently have. The most typical requirements are:

  • Automate most of the tasks; there are several advantages to automating the identification of web application vulnerabilities. Most common reason management can think of is to save time, but it is not just about that, as seen from the article Why Web Vulnerability Testing Needs to be Automated.
  • Lowering the costs of web application security by doing in house scanning rather than hiring a seasoned expensive penetration tester or service.
  • Increase the coverage; as opposed to a penetration testers, an automated web application security scanner has an extensive set of heuristic web vulnerability checks that are frequently updated by a number of researchers and security experts. This allows users to identify all type of web application vulnerabilities in custom made web applications. Some scanners also have a vulnerability database for known web applications such as WordPress and Joomla that also comes in handy if your business or customers are using such web applications.

In short, automated web application security scanners are mostly required to save time and to ensure that all technical web vulnerabilities are identified. Once you listed down your requirements proceed to gathering as much information as you can about the websites and web applications you will be scanning.

Document all the Web Applications to be Scanned

The next step in the selection process is to document the web applications you will be scanning with the automated web security scanner. During this stage it is important to identify most common factors of the web applications. For example if something is implemented on 2% of the web applications you are scanning, then you can test such parts manually. But if something is implemented on 60% of the web applications, or more, then its testing should definitely be automated. The below list of questions will help you build a list of web applications common factors;

1. Which development frameworks are used to build the web applications?

2. Is there any type of authentication mechanism used on the web applications?

3. What type of database backend server is used?

4. Are URL rewrite rules being used?

5. Do you need a client side certificate to access the web applications?

6. Are there any client-side scripts that need to executed and tested?

7. Are Custom 404 error pages being used?

8. Are there any types of protection mechanisms implemented on the web applications, such as Anti-CSRF mechanism?

Once you have the list of common factors you can shortlist the web application security scanners you would like to evaluate. For example;

1. If all your web applications are developed with PHP and .NET, you do not need to test automated tools that cannot scan and identify vulnerabilities in such web technologies. For example some web application security scanners cannot automatically crawl, scan and identify vulnerabilities in web applications built with .NET, hence there is no need to test such scanners.

2. If authentication is implemented, the automated web security scanners you want to test must support the type of authentication mechanism that is being used, and can be configured to automatically login to the website during a web security scan.

3. If URL rewrite rules are being used for search engine friendly URL's, the scanner should be able to identify such URL structure and scan the website properly without getting stuck in some sort of infinite loop, or without requiring extensive configuration changes.

4. If the web applications you want to scan have some sort of mechanism such as anti-CSRF, the scanner should still be able to automatically scan the web application without missing any parts of it. Note: Such mechanisms are typically barriers for most of the automated tools.

Testing Web Application Security Scanners

Once you have completed all the paperwork it is time to get your hands dirty and start scanning websites. Download the trial or demo version of each web vulnerability scanner you would like to evaluate. If the trial is not available for download, contact the software vendor.

Before You Start a Web Security Scan

If you are new to automated tools such as web application security scanners, there are some things you need to know about before launching a security scan. Here is the list of points:

How Web Application Security Scanners Work

Before doing anything else, one needs to understand how such automated tools work. First the scanner will crawl the target website or web application and identify all possible attack entry points and parameters. During this stage the crawler will access every link it discovers, including links in client-side scripts etc. During the scanning stage the scanner will send specially crafted HTTP requests which include a payload that is used to test if a website if vulnerable or not.

Although automated scanners are designed not to be intrusive, there is still a small probability that such attacks might tamper your web application or database, depending on how secure your web application already is. In light of this, we move to the next point.

Always Scan Realistic Test and Staging Web Applications

First and foremost it is important to scan realistic web applications, i.e. web applications that are similar, or ideally identical to the real live ones you will be scanning. Unfortunately many evaluate web application security scanners against vulnerable web applications such as DVWA (Damn Vulnerable Web Application) and OWASP WebGoat. These type of web applications have been built for educational purposes and might be completely different than the web applications your business or customers use.

Therefore do not base your findings on any of such demo sites. Don't forget a web application security scanner can easily cover thousands of different vulnerabilities where about these simple test systems will only show how the scanner behaves according to 10 of these thousands vulnerability cases.

Therefore you should make your tests against a test or staging website if you would like to see how the web application security scanner performs. It might take some time until you get used to the automated tools you are using so stay safe to scan test websites. What's for sure is that with time you will get used to the tools you are using and soon should be able to scan live websites. In fact it is recommended to scan both staging and live websites because some vulnerabilities might only be introduced when switching from the staging server to the live server.

Evaluating Web Application Security Scanners and the Results

Now that you know what you need and how to evaluate the software, it is time to fire up the scanners. Below are some points which if followed should help you determine which is the best automated web application security scanner that fits your requirements.

Web Application Coverage

During a scan check the list of all crawled objects. Ensure that the sitemap representation of your web application includes all the files and their variations, scripts, client scripts, input parameters, directories etc which are on your website. If not all objects are listed, it means that the crawler is not able to automatically crawl all of the web application, thus might not identify all vulnerabilities.

Web Vulnerability Reporting

Once the automated web security scans are finished, start comparing the findings of both scanners.

  • Which web application security scanner detected the most vulnerabilities?
  • Which web application security scanner reported less false positives?
  • Are there any vulnerabilities that were not detected by any of the scanners (false negatives)?

By finding the best ratio between all of the three points mentioned above, you should find the ideal automated web security scanner. Definitely you do not want a scanner that reports hundreds, if not thousands of vulnerabilities but most of them are false positives.

You might ignore false positives or mark them out manually. However this mechanism might not scale well because when you actually start using your scanner regularly on large scan web applications, and there are bigger problems with false positives, such as a lot of extra time being spent on this process (i.e. verifying false positives), you and your users will start losing faith in the scanner. To learn more about false positives and the impact they have on a penetration test, refer to the article The Problem of False Positives in Web Application Security and How to Tackle them.

On the other hand you do not want a scanner that does not detect much of the most commonly exploited vulnerabilities. The best automated scanners are typically those scanners which can report all of the most commonly exploitable web application vulnerabilities, such as SQL injection and Cross-site scripting while reporting the less false positives and which require the least configuration.

After all each automated scan should always be accompanied by a manual penetration test, so those one-off vulnerabilities which are rarely, or never exploited in a real life attacks can be identified manually. But at least, the bulk of the work is done and developers can already start concentrating on remediating the reported vulnerabilities.

Integration with Other Web Security and Development Tools

Another important feature, or set of features to look for in web application security scanners is how easy they can be integrated. For example does it output the scan results to standard formats that can be parsed by other tools, such as XML? Can an automated web security scan be launched via command line or batch files? The more of such features it has, the easier it is to integrate web application security scanning in the SDLC process of web applications.

And for those penetration testers who always like to dig deeper into web applications, can the scanner import the output of other web security tools typically used during manual security audits, such as other proxy servers, or http analysers? Supporting such kind of tools is an important factor to consider as well, since the more integration capabilities a scanner has, the more of the process can be automated.

Choosing your Web Application Security Scanner

As we have seen there are several criteria you should look into when choosing a web application security scanner. What is important is that you should focus on your needs rather than relying on others' opinion. Ensure that the automated tool you are about to choose can scan the web applications you want to audit and allows you to automate as much as possible, to save time, money and other resources.


Dead accurate, fast & easy-to-use Web Application Security Scanner