URL Rewrite Rules

Web application developers use URL Rewrite Rules to hide parameters in the URL directory structure. This makes it easier for search engines to index all the pages on a website, web browsers are supplied the URL in a format they understand and makes them easy for users to remember.

  • For example, when you browse a movie collection library, the URL typically looks something like this: http://www.example.com/movie/fight-club/. Using a URL rewrite rule, the web server converts this URL to a specific format, so it can retrieve the data from the back end database and display the movie details to the website visitor:
  • http://www.example.com/library.php?movie=fight-club.
    In this example, we can determine that the subdirectory ('/movie') in the first URL is actually a parameter in the library.php file that accepts inputs, which in this case is the movie name ('fight-club').

During the scan, Netsparker sends normal HTTP requests to the web application to mimic an attacker, ensure that such requests are accepted by the web application and all parameters in the URLs are properly scanned. It is also possible to scan pages which have more than one parameter in the URL.

Problems with URL Rewrite Rules

This table lists and explains the problems that can occur when automated web vulnerability scanners attempt to scan websites that employ URL rewrite technology and rules.

Problem

Description

Parameters in URLs are not scanned

A common problem web vulnerability scanners have when scanning web applications that use URL rewriting technology is that scanners are unable to identify parameters in the URLs. Scanners assume that the URLs are directories rather than parameter names or values, and leave them unscanned.

For example, when scanning the URL http://www.example.com/movie/fight-club/ the scanner would think that both movie and fight-club are directories, while in reality movie is a parameter and fight-club is a value.

Prolonged vulnerability scans

This problem can lead to prolonged scans and incorrect scan results. For example, if the web vulnerability scanner is scanning a movie database that contains 100,000 films, since the scanner is unable to identify that there is a parameter and a value in the URL, it would think that they are all different pages, so it will try to crawl them and scan them all.

If memory problems and other exceptions are not handled properly by your scanner, this could also lead to your software crashing, leaving you with no results and a number of wasted hours.

If you do not configure URL rewrite rules in Netsparker, it will heuristically identify the pattern and will limit the scan to avoid having prolonged scans and incorrect results.

Configuring URL rewrite rules is a difficult process

Since URL rewrite technology has become really popular in web applications, many commercial web vulnerability scanners allow users to configure the scanner so it can identify the parameters in the URLs and scan them.

But even though web vulnerability scanners can be configured to scan websites using URL Rewrite Rules, there are several other problems users can face:

  • Configuring of URL rewrite rules support is very difficult
  • User must know how to write regular expressions
  • User should have access to web server configuration files

Unless you are the developer of the web application itself or have a deep understanding of the web application, and unless you have direct access to the configuration files, it is impossible to configure URL rewrite rules on the scanner. And, even if you do know how to do it, configuring rewrite rules is a very difficult and time consuming task.

Web applications are not properly scanned for vulnerabilities

Assuming you manage to configure URL rewrite rules in your web vulnerability scanner, there are further problems, or at least, there are a number of limitations to how the scanners scan the web application.

As a security precaution, web applications do not accept HTTP requests which are already “translated”, such as http://www.example.com/library.php?movie=fight-club. By default .NET web applications do not accept such HTTP requests. The problem is worsened when scanning MVC web applications because such applications use a different approach to URL rewriting.

While Netsparker can scan MVC web applications, many other web vulnerability scanners cannot, even when URL rewrite rules are configured.

Once you configure the URL rewrite rules in your scanner, it sends a type of HTTP requests called translated queries. Even though the web application security scanner reports that the scan ran successfully, most of the HTTP requests were denied and the parameters in URLs were not scanned, providing you with a false sense of security.

By default, Netsparker can heuristically identify URL rewrites in the target website.

Netsparker Standard can also automatically configure the settings (see Automatic Configuration of URL Rewrite Rules In Netsparker Web Applications Security Scanners).

However, if you can configure the URL rewrite rules manually as explained here, the scan will be more efficient.

Specifying the Parameter Type

Specifying the correct parameter type makes the web application security scan more accurate.

  • For example, imagine specifying the following pattern: "/{PARAM}/{ID}", where "{PARAM}" is the parameter that will be scanned and "{ID}" is its value. If the parameter type is not specified, leaving the default 'Any', this means that both the below URLs will match this URL Rewrite configuration:
  • The above matching is too generic and might lead to incorrect scan results. For example Netsparker might not scan some files because after collecting enough samples (i.e. /product/1, /product/2, /product/3 etc) it will assume that /products/date.js is just another value of the product parameter, due to the generic matching.
  • In this case, if you set the parameter type to 'Integer', you avoid this problem, since Netsparker will only expect integers when scanning such parameters and when it notices something else, such as /product/date.js it will recognize it is a file and scan it.

Encoded URLs

If you are manually configuring URL Rewrite Rules, and your website URLs use encoded values, always specify the decoded value.

  • For example, suppose the Website URL is:
    http://www.example.com/user/john%2dDoe.
  • The correct URL rewrite rule should contain the decoded character as follows:
    http://www.example.com/user/{firstname}-{lastname}
  • This rewrite rule is incorrect because it contains the encoded character:
    http://www.example.com/user/{firstname}%2d{lastname}

How to Configure URL Rewrite Rules in Netsparker Enterprise

  1. From the main menu, select Scan, then New Scan. The New Scan window is displayed.
  2. In Scan Options section, select the URL Rewrite tab.
  3. Select an option: None, Heuristic or Custom:
    • If None is selected, then no rules will be applied

    • If you keep Heuristic as the default, then:
      • Netsparker Enterprise will automatically populate the fields:
        • Root Path Maximum Dynamic Signatures
        • Sub Path Maximum Dynamic Signatures
        • Block Separators
        • Analyzable Extensions

    • If you select Custom:
      • Select Enable Heuristic URL Rewrite Detection so Netsparker will attempt to automatically determine other URL Rewrite rules. Both Custom rules and Heuristic rules will be in effect. Or disable, so that only the Custom rules will be in effect.

      • In the Placeholder Pattern and RegEx Pattern fields, enter the relevant information
      • Click New to enter further rules
  1. Click Launch.

How to Configure URL Rewrite Rules in Netsparker Standard

  1. Open Netsparker Standard.
  2. From the Start a New Scan dialog, in the Scan Settings menu, click URL Rewrite.
  3. Use Heuristic URL Rewrite Support is the default option. Check the Use Custom URL Rewrite Rules option.

  1. Check Enable Heuristic Rule Detection, so the scanner still tries to automatically detect any additional URL Rewrites on the target website and use them in conjunction with the ones you configured.
  2. Click New to launch the URL Rewrite Rules wizard.
If you wish to configure the URL Rewrite rules manually in Netsparker, without using the wizard you can simply click on the Placeholder Pattern and RegEx Pattern input fields to populate them manually.
  1. In the first step of the wizard, specify a URL that matches the URL rewrite rule you want to add, such as "http://www.example.com/movie/fight-club/"

  1. In the second step of the wizard, you have to specify which of the path segments is a parameter and its type, by following the below procedure:
    • Check the URL path segment that contains a parameter value
    • Specify the parameter name
    • Specify the parameter type from the drop down menu in the Parameter Type column

  1. If there are multiple parameters in the URL, specify all of them. In this example (http://www.example.com/2014/07/movie/fight-club/), you would specify the year, month and movie, as illustrated.

  1. Click Finish so the placeholder pattern and regular expression are automatically generated. Click on any of the values to manually modify them, for example to manually write a regular expression.

Configured URL rewrite rules in Netsparker Desktop

If required, click Test to test the URL Rewrites Rules. When running a test, put an example URL next to the Test button.

This video explains how to configure URL Rewrite Rules in Netsparker.

Netsparker

Dead accurate, fast & easy-to-use Web Application Security Scanner

GET A DEMO