Web Vulnerability Scanners Comparison

The latest independent web application security scanners benchmark results have been published. How did Netsparker fare when compared to the other web vulnerability scanners? In short, Netsparker was:

The only scanner that identified all the vulnerabilities
One of the only two scanners that reported zero false positives

None of the other web vulnerability scanners in the comparison, including the open source ones performed as well as Netsparker. For more detailed information about these comparisons, including results of the vulnerability detection rates, read on. This post also explains how the vulnerability scanner tests were conducted and displays the results of each individual test.

Table of Content

What is the Web Application Security Scanner (DAST) Benchmark?
The Benchmark Results – Global Results
The Benchmark Results – Individual Tests Results
Are Web Security Scanner Comparisons Useful & Realistic?
1. Which is the Best Web Application Security Scanner?
2. Can Netsparker Identify Security Flaws in Your Web Applications and APIs?
Past Comparisons Between Automated Web Application Security Scanners
Other Comparisons with Netsparker

What is the Web Application Security Scanner (DAST) Benchmark?

It is a test that compares the features, coverage, vulnerability detection rate and accuracy of automated web application security scanners, also known as web vulnerability scanners or Dynamic Application Security Testing (DAST) solutions.

Individual tests were conducted by the independent information Security Researcher and Analyst, Shay Chen. He compared both commercial and open source vulnerability scanners, but in these results we are only focusing on the commercial solutions.

Shay has been conducting benchmark tests and improving the platform since 2010. So far he has released six (2010, 2011, 2012, 2013/2014, 2015, 2017/2018). His work is considered the de facto comparisons results by the application security industry.

How Are Tests Performed?

Shay Chen and his team built The Web Application Vulnerability Scanner Evaluation Project (WAVSEP), a testbed that they scan to see how every scanner performs. In it the scanners are tested against realistic setups, with crawling the most basic HTML website to identifying security vulnerabilities typically found in modern Single Page Applications (SPA). The WAVSEP is an open source project and new tests are incorporated every year. You can download it from the WAVSEP GitHub repository.

Testing Vulnerability Management & Assessment Solutions

This year Shay and his team went a step further. They have been installing and integrating DAST solutions in real-life enterprise SSDLC (Secure Software Development Lifecycle) processes to get a better understanding of how they can expand the WAVSEP testbed and test the scanners. So the have implemented automated vulnerability scanners in financial, hi-tech and telecom organizations.

They wanted to test more than security vulnerability detection rates. They wanted to see how these tools can really help business improve their vulnerability management and triaging processes, and their information security programme. As Shay himself explains:

Some of these experiences led us to develop test cases aimed to inspect issues in proclaimed features that we noticed didn't work as expected in actual implementations, and some to the creation of comparison categories that are apparently crucial for real-world implementations.

The Negative Impact of False Positives

Shay and his team also talked about the importance of accurate scan results in the report, after their first-hand experience with scanners in real-life environments. Quoting from the official benchmark results:

Weeding out a reasonable amount of false positives during a pentest is not ideal, but could be performed with relative ease. However, thousands upon thousands of false positives in enterprise SSDLC periodic scan scenarios can take their toll.

False positives occur in scan results to the detriment of the web application security industry. So much so, that large organizations, that have hundreds or even thousands of web applications, limit their efforts to a handful of mission-critical websites and ignore the rest. I was quite shocked to learn this, though it is unsurprising because many hacks and data leaks that happen every year.

False Positives Make Scaling Up Web Security Impossible

If a solution reports false positives, it is impossible – unless you have an army of people – to scale up your efforts and secure all your web applications. Even if you have the budget for such an undertaking, there is still the troublesome problem of human error.

This is why we developed Netsparker's proprietary Proof-Based Scanning^TM, technology that automatically verifies detected vulnerabilities – proving they are real flaws, and not false positives. The benefits of such technology are plentiful, and since the scan results are accurate, you can easily scale up your efforts. In a real-life environment, with thousands of web applications, you can start the vulnerability triage process and fix them within a matter of hours.

Evaluation Criteria

In the 2017/2018 benchmark tests, Shay and his team included several previously uncovered aspects of scanners and new tests to check the detection capabilities of previously uncovered vulnerabilities. This included OS Command Injection, and repurposing XSS via RFI tests that can also be used for Server Side Request Forgery (SSRF) evaluation.

The Benchmark Results – Global Results

You might notice that vendors such as Qualys, Tenable Nessus, Retina and Nexpose are not mentioned in these comparisons. We have checked with Shay and he confirmed that he contacted all vendors but not all of them wanted to contribute and participate towards these benchmarks.

How Many Vulnerabilities Did the Scanners Detect?

This matrix lists what percentage of all vulnerabilities each web application security scanner identified. Missing data or scores are represented with 'N/A'.

	Netsparker	WebInspect	AppSpider	Acunetix	Burp Suite	AppScan
OS Command Injection (New)	100	N/A	99.11	78.57	93.3	N/A
Remote File Inclusion/SSRF (New)	100	100	82.67	64.22	74.67	N/A
Path Traversal	100	91.18	81.61	94.12	78.31	100
SQL Injection	100	98.46	95.39	100	97	100
Reflective XSS	100	100	100	100	97	100
Unvalidated Redirect	100	95.51	100	100	76.67	36.67
Average %	100.0	97.0	93.1	89.5	86.2	84.2

Clearly, Netsparker beats the competition in terms of vulnerability detection. It was the only scanner to identify all the security issues, followed by HP WebInspect at 97% and Rapid7 AppSpider at 93.1%.

Note: Missing data or scores were the result of lack of support (in some cases even a lack of response) from some vendors. Only the tests for which scanners had a result were used to calculate the global average.

How Many False Positives Were Reported?

This matrix lists what percentages of all false positives each web application security scanner identified.

	Netsparker	AppSpider	WebInspect	AppScan	Acunetix	Burp Suite
OS Command Injection (NEW)	0	0	0	0	0	0
Remote File Inclusion / SSRF (NEW)	0	0	0	0	0	16.67
Path Traversal	0	0	0	0	0	12.5
SQL Injection	0	0	0	0	0	0
Reflective XSS	0	0	0	0	0	0
Unvalidated Redirect	0	0	11	11	11	0
Total %	0.0	0.0	1.8	1.8	1.8	4.9

Netsparker and Rapid7 AppSpider were the only solutions that reported zero false positives, while Burp Suite was the one that reported the most false positives.

Graph with Global Detection & False Positives Rates

This graph is a visual representation of the global results, illustrating both the vulnerability detection and false positives rates side by side for each vendor.

The Benchmark Results – Individual Tests Results

OS Command Injection Detection

The OS Command Injection vulnerability tests is one of the new tests. Netsparker was the only scanner to detect all the vulnerability instances in the test.

Remote File Inclusion / SSRF

This was also one of the new tests included in the WAVSEP benchmarking tests. Netsparker and WebInspect were the only two scanners that detected all the vulnerabilities in this test. AppSpider followed with 82.67%, and then Burp Suite with 74.67%. Though Burp Suite also had 16.67% false positives.

Path Traversal

This time Netsparker and Appscan led the field, both of which detecting all the Path Traversal vulnerabilities.Acunetix WVS and HP WebInspect came third and fourth, followed by AppSpider. Burp Suite was the scanner that detected the least at 78.31% and also reported 12.5% false positives.

SQL Injection

This is one of the classic tests; the SQL injection vulnerability. In this test Netsparker, Acunetix WVS and Appscan detected all the vulnerabilities. HP WebInspect followed with 98.46%. None of the scanners reported any false positives in this test.

Reflective Cross-site Scripting (XSS)

All scanners but Burp Suite detected all the cross-site scripting vulnerabilities.

Unvalidated Redirect

In the unvalidated redirect vulnerability tests three of the scanners, WebInspect, Acunetix and AppScan reported vulnerabilities. AppScan also performed very poorly with a detection rate of only 36.67%. On the other hand, Netsparker, AppSpider and Acunetix detected all the vulnerabilities.

Are Web Security Scanner Comparisons Useful & Realistic?

As a rule of thumb, nothing beats a live environment test. Though it impossible to test all the web security scanners available on the market. So, these comparisons are incredibly useful because they highlight who the market leaders are – those vulnerability scanners that can detect the most vulnerabilities and generate accurate results.

Once you determine which two or three solutions you’d like to test, request a trial from the vendor to test the vulnerability scanner. In fact, at Netsparker we always encourage prospects to test our web security solution by scanning a staging copy of their web applications, as explained in How to Evaluate Web Application Security Scanners.

To do such test it is really easy - register for a trial of Netsparker Enterprise or install Netsparker Standard on a Microsoft Windows virtual machine.

Which is the Best Web Application Security Scanner?

The best web vulnerability scanner is the one that detects the most vulnerabilities in your web applications, is easiest to use and can help you automate most of your work. Finding vulnerabilities in a web application is not just about the duration of the scan, but how long it takes to setup the scan (pre-scan) and verify the results (post scan). How long it takes you to complete the whole process including the triaging of vulnerabilities and testing of fixes. Therefore, when you evaluated solutions, you should ensure that automated vulnerability confirmation is part of the equation.

Read Shay Chen’s full report: Evaluation of Web Application Vulnerability Scanners in Modern Pentest/SSDLC Usage Scenarios.

Can Netsparker Identify Security Flaws in Your Web Applications and APIs?

Netsparker can scan any type of web application, regardless of the technology it was built with. It uses a Chrome based crawling engine and can identify vulnerabilities in legacy, and custom built, modern HTML5, Web 2.0 applications and Single Page Applications (SPA). It also has vulnerability checks for popular frameworks, libraries and popular open source software such as WordPress, Joomla! and Drupal.

Though I would still encourage you to download a demo and launch a vulnerability scan. The Netsparker vulnerability scanner is very easy to use and most of the pre-scan configuration is automated. It is an all in one vulnerability management solution, with multi user support and integration capabilities. Though to test it all you need to do is specify the URL and credentials (to scan password protected websites), and launch a vulnerability can.