URL rewrite rules and automated web vulnerability scanners are not exactly the best of friends; typically URL rewrite technology hinders automated vulnerability scans. This article explains what are URL Rewrite rules and URL Rewrite technology, why URL Rewrite technology hinders automated web vulnerability scans and what Netsparker did to enable users to use Netsparker Web Application Security Scanner to easily scan websites which use URL rewrite technology. In this article:
Web application developers use URL Rewrite Rules to hide parameters in the URL directory structure and is typically used to make it easier for search engines to index all the pages on a website. Another advantage of using URL rewriting technology is that all symbols such as question marks and equal signs are not used in URLs, thus making URLs easy to remember.
A URL rewrite rule is like a translator; it explains the web server software how to change the URL that users can see and use in a web browser to a format it understands. Example follows:
When you browse a movie collection library, the URL typically looks something like:
Using a URL rewrite rule the web server converts the above URL to a specific format as shown below, so it can retrieve the data from the back end database to show the movie details to the website visitor:
From the above example we can determine that the subdirectory movie in the first URL is actually a parameter in the file library.php that accepts inputs, which in this case is the movie name fight-club.
A common problem web vulnerability scanners have when scanning web applications that use URL rewriting technology is that scanners are unable to identify parameters in the URLs, and would assume that they are directories rather than parameter names or values, therefore such parameters are not scanned.
For example when scanning the URL http://www.example.com/movie/fight-club/ the scanner would think that both movie and fight-club are directories, while in reality movie is a parameter and fight-club is a value.
As a matter of fact the above problem can lead to prolonged scans and incorrect scan results For example if the web vulnerability scanner is scanning a movie database that contains 100,000 films, since the scanner is unable to identify that there is a parameter and a value in the URL it would think that they are all different pages, therefore it will try to crawl them and scan them all.
If memory problems and other exceptions are not handled properly by your scanner this could also lead to the software crashing on you, leaving you with no results and a number of wasted hours. If you do not configure URL rewrite rules in Netsparker it will heuristically identify the pattern and will limit the scan to avoid having prolonged scans and incorrect results.
Since URL rewrite technology has become really popular in web applications, many commercial web vulnerability scanners allow users to configure the scanner so it can identify the parameters in the URLs and scan them.
But even though web vulnerability scanners can be configured to scan websites using URL Rewrite Rules, there are several other problems users typically face;
Therefore unless you are the developer of the web application itself or have a deep understanding of the web application, and unless you have direct access to the configuration files it is virtually impossible for someone to configure configuring URL rewrite rules on the scanner and will be a very hard and time consuming task.
And even if you manage to configure URL rewrite rules in your web vulnerability scanner you are in for more problems, or better there are a number of limitations to how the scanners scan the web application.
As a security precaution typically web applications do not accept HTTP requests which are already “translated”, such as http://www.example.com/library.php?movie=fight-club. In fact by default .NET web applications do not accept such HTTP requests. It can get even worse when scanning MVC web applications because such applications use a different approach to URL rewriting.
As a matter of fact, while Netsparker can scan MVC web applications many other web vulnerability scanners cannot, even when URL rewrite rules are configured.
But once you configure the URL rewrite rules in your scanner, it typically sends such type of HTTP requests, i.e. translated queries. In this case even though the web application security scanner reports that the scan ran successfully, in reality most of the HTTP requests were denied and the parameters in URLs were not scanned, thus providing a false sense of security.
As we have just seen in this article, many web vulnerability scanners have several shortcomings when it comes to scanning web applications that use URL rewrite technology. They also give a false sense of security since parameters are not actually scanned and are very difficult to configure.
On the other hand when we implemented URL rewrite rules support in Netsparker we wanted to ensure that they are very easy to configure and that all parameters in the URLs are actually scanned correctly. Both Netsparker scanners also have a heuristic technology that can identify the URL rewrite configuration on the target web application and automatically configure themselves, therefore if you do not know about the URL rewrite configuration, or do not have access to the web application configuration simply enable this heuristic technology.
During the web vulnerability scan Netsparker sends normal HTTP requests to the web application just like an attacker would, to ensure that such requests are accepted by the web application and that all parameters in the URLs are properly scanned to identify any potential vulnerabilities they might be vulnerable to. With Netsparker Web Application Security Scanner it is also possible to scan pages which have more than one parameter in the URL.
For more details on how to configure URL rewrite rules in Netsparker using the user friendly wizard read Configuring URL Rewrite Rules Support in Netsparker.