URL rewriting in web applications is one big headache for both automated web security scanners and their users. If you scan a website that uses URL rewrite technology and do not configure URL rewrite rules in your scanner, the parameters in URLs won't be scanned. And if such parameters are vulnerable the scanner won't report such vulnerabilities, leaving them open to potential exploitation.
Failing to configure URL rewrite rules in scanners also mean that the web security scan might take forever to finish. Some scanners will even report false positives and go haywire if you don't configure URL rewrite rules, yet many users still do not configure them. So we automated the whole process of configuring URL rewrite rules in web security scanners.
This blog post gives an insight of how this automated technology works. It also explains why we automated the process of configuring URL rewrite rules in our web application security scanners.
Addressing the Problem of Configuring URL Rewrite Rules in Automated Web Security Scanners
Many users fail to configure URL rewrite rules in their automated security scanners because in most software, they are very difficult to configure. Most of the time you need to have access to the web server's configuration files, and need to know how to write regular expressions. As always, we wanted to automate as much of this process as possible, so users can scan all of the websites' parameters without getting bogged down into configuring their web security scanner.
Manually Configuring URL Rewrite Rules in Netsparker Is Easy
To configure URL rewrite rules in both Netsparker Desktop and Netsparker Cloud you do not need to have any information on how the web server is configured, and you do not need to know how to write regular expressions. All you need to do is use the URL rewrite rules wizard to specify the parameter name and type, the rest is all automated.
Even with such wizard in place many users still do not configure URL rewrite rules. Most users just want to launch a web security scan and get a report with all the vulnerabilities to fix. Only the very few really go into the detail of configuring several different aspects of the web security scan. So as per our usual, we thought of automating the process of configuring URL rewrite rules while still allowing the old school geeks to configure their own rules.
Limitations of the Existing URL Rewrite Rules Configuration
To understand how this new technology works, first you have to understand the limitations of the existing solution. At the moment Netsparker users can select one of the below options when scanning a web application which has URL rewrite enabled:
Use Heuristic URL Rewrite Support
When this option is selected the scanner tries to automatically determine if URL rewrite technology is being used on the target website, so if it is the scan is limited. Therefore if during a scan the scanner detects the below directory structure it will stop after a number of tries to avoid going into a loop:
This solution has a number of limitations. First of all, the scanners will not scan the parameters in URLs and it only recognizes numeric parameters, such as the ones in the example above.
Use Custom URL Rewrite Rules
When this option is enabled you have to configure URL rewrite rules in Netsparker scanners so the scanner can scan all the parameters in the URLs. The problem with this solution is that even though when compared to the configuration of other scanners it is a very easy solution, users still need to have some basic knowledge of the link structure of the target website, hence manual configuration is involved. And as you might have noticed by now, manual configuration is in Netsparker's bad books.
Automatic Detection and Configuration of URL Rewrite Rule of Target Websites and Web Applications
To automate the configuration of URL rewrite rules we improved the Heuristic URL Rewrite engine. Therefore if you want to scan a website or web application which has URL rewrite enabled you can either configure the URL rewrite rules manually or select the option Use Heuristic URL Rewrite Support.
Screenshot: Heuristic URL Rewrite Rules configuration in Netsparker Desktop
Screenshot: Heuristic URL Rewrite Rules configuration in Netsparker Cloud
By default, when Netsparker Desktop or Netsparker Cloud crawl more than 60 URLs that have the same pattern, they will automatically create a custom URL Rewrite rule and scan the parameters in the URLs. Below is a list of some of the different type of parameters which Netsparker can automatically identify;
Example URL: http://www.example.com/user/robert
Example URL: http://www.example.com/userid/1
Multiple Numeric Parameters (for example typically used as dates in URLs)
Example URL: http://www.example.com/blog/2016/05/13/netsparker-rocks
Parameters with Prefixes
Example URL: http://www.example.com/users/user_robert/
Parameters with Suffixes
Example URL: http://www.example.com/users/robert_user/
Multiple Parameters in Same URL
Example URL: http://www.example.com/users/123/robert/
Slug Based Parameters
Example URL: http://www.example.com/blog/pci-dss-good-bad-insecure/
Fine Tuning the Automatic Detection and Configuration of URL Rewrite Rules
As seen in the above screenshots the Netsparker scanners have three new settings in the Heuristic URL Rewrite Support configuration. These settings can be used to fine tune the automated detection and configuration of URL rewrite rules and below is an explanation of what each of the setting is used for.
Note: The default settings work well in most of the cases and should only be changed should you notice any performance issues.
Maximum Dynamic Signatures
Use this option to specify how many URLs Netsparker should crawl before it determines that URL rewrite rules are needed to scan the target website. The scanners determine if URL rewrite rules are needed or not by analyzing the patterns of the crawled URLs.
If there are more than 60 URLs (default value) that have a similar pattern, for example http://www.example.com/category/[x] then the scanners will automatically create the neccessary URL rewrite rules to crawl the website and scan the parameters in the URLs.
Use this setting to specify which URL extensions should be analyzed by the heuristic URL Rewrite technology in case they are URL Rewrite URLs. For example a URL such as http://example.com/hello-world.html can also be a URL Rewrite URL. The default extensions are HTM and HTML. Therefore by default the heuristic URL Rewrite technology will include these URLs when analyzing URLs:
http://example.com/hello-world.html (Allowed extension)
http://example.com/hello-world.htm (Allowed extension)
http://example.com/hello-world (No extension)
A link such as http://example.com/hello-world.asp will not be analyzed since the asp extension is not listed in the Analyzable Extensions setting.
Use this option to specify the characters used on the target website to separate the blocks in a URL. The default list has the following characters: / _ $ . , ; | :
Therefore when using the default list, the scanner will split a crawled URL such as http://www.example.com/user/robert_abela in 6 blocks, as highlighted in the below screenshot.
Reporting the Detected URL Rewrites on Target Website
Once Netsparker Desktop or Netsparker Cloud automatically detect and configure URL rewrite rules for a target website they will report it in a new node URL Rewrite in the Knowledge Base node, as seen in the below screenshots.
Automatically configured URL rewrite rules are reported in the Knowledge Base node in Netsparker Desktop
Automatically configured URL rewrite rules are reported in the Knowledge Base node in Netsparker Cloud
For more detailed information about this technology refer to the whitepaper Automating the Configuration of URL Rewrite Rules in Netsparker Web Application Security Scanners.