Ha.ckers blog published Larry’s new report: “Accuracy and Time Costs of Web Application Security Scanner Report”.
Unfortunately Larry never contacted us so we didn’t know that he was doing such a test. However as soon as the report was out we conducted the very same test as methodology was straight forward.
Also we sent an email to Larry and offered fully-functional trial version for him to conduct the test as well. Anyone with full or demo version of Netsparker can repeat these tests easily.
Two charts exactly like Larry’s report, we added Netsparker to the results.
As you can see after NTO Spider, Netsparker is the best scanner when “Trained” and the second best trainer in “Point and Shoot” right after NTO and IBM AppScan.
We delivered what we claimed and the report was false-positive free.
Although 184.108.40.206 release caused some LFI bugs which already has been addressed and will be deployed in the next release. This caused [High Possible] LFI vulnerabilities. None of them were confirmed (obviously!) but it was quite irritating, for full-disclosure I wanted to point out that clearly. We spotted some instances unconfirmed possible LFI issues in all Permanent XSS locations . Since we considered [High Possible] as vulnerability, we’ll consider these as False Positives. We addressed this problem in v220.127.116.11, use “Help > Check Update” to update Netsparker.
Actually Netsparker has not many training options because it doesn’t require any. It picks up many URL Rewrites automatically, it detects Custom 404’s on the fly, 99% of the time you don’t need to tweak it. It just works. In this case all we did was changing the start URL of the scan. So our training time was something between a second and a minute. Depending on how fast we can copy & paste a URL.
Larry calculated the overall human time/cost with the following formula:
Training time + (# False Positives * 15min) + (# False Negatives * 15min)
This is the original chart (in minutes, lower is better):
However since none of the scanners in the test has a confirmation engine like Netsparker he excluded the fact that even though all of the issues are not false positives you still have to analyse them, otherwise you wouldn’t know if they are false-positive or not.
This is not an issue with Netsparker as we can confirm vulnerabilities. Out of 103 identified vulnerabilities we confirmed 87 of them, so we confirmed 84% of all identified issues. This could’ve been much higher if the test websites were using MySQL, ORACLE or MS SQL or even Postgres (we have limited support) instead of MS Access.I’ll discuss this further at the end of this post.
So I’ve revised Larry’s function and made it more realistic by adding one more criteria, time to confirm that a vulnerability is not a false positive. 15 minutes would have been harsh as some issues could be really obvious so I used 3 minutes for per identified vulnerability which is quite naive.
Training time + (# False Positives * 15min) + (# False Negatives * 15min) + ( # Identified none FP Vulnerabilities * 3 min )
Updated and more realistic results (in minutes, lower is better):
Netsparker identified 7 new vulnerabilities that all of the other scanners missed:
UPDATE: 2 of the issues removed from zero.webappsecurity.com because they were duplicates, we didn’t notice it in the first analysis.
We observed that Netsparker missed a remote code evaluation vulnerability according to the Larry’s results.
I don’t know how Larry confirmed all vulnerabilities but this is certainly not exploitable :)
From “http://testphp.acunetix.com/comment.php” and “phpaction” parameter.
if ($_POST["phpaction"] == "printf(md5(acunetix_wvs_security_test));exit;//") eval($_POST["phpaction"]);
So it’s not actually a vulnerability it’s just a PoC vulnerability to demonstrate Acunetix’s related checks.
UPDATE: I want to make it clear that this is obviously not an intentional code to block other scanners. This is just a test page for Acunetix scanner for their customers and demo versions. Which make a lot of sense to do such a thing, all I wanted to point out that this issue should be excluded from the test and leaving such an issue in the report might raise questions about other issues. I’m not quite sure if there are any other issues like this in the report as we haven’t investigated every single one of them. My apologies to our friends in Acunetix if I seemed like accusing them, that definitely wasn’t my intention.
We were expecting good results and we got it. Although I still think Netsparker would have performed better in more realistic scenarios. For example I haven’t notice any Full-Blind SQL Injection (time based) vulnerabilities in the whole test.
One of the most unrealistic things about the report is the amount of false positives possibilities in the test websites. If you haven’t use any of these scanners just ask anyone and they’ll tell that they definitely report more than 2-3% false-positive issues in every scan.
Report proves that Netsparker’s Confirmation and False-positive Free Scanning feature is a real time saver.
You can request a demo and try Netsparker’s fully-functional evaluation version.