Support
Working with Scans

URL Rewrite Rules

This document is for:
Invicti Standard, Invicti Enterprise On-Premises, Invicti Enterprise On-Demand

Web application developers use URL Rewrite Rules to hide parameters in the URL path structure. This makes it easier for search engines to index all the pages on a website, while web browsers are supplied the URL in a format they understand and makes them easy for users to remember.

  • For example, when you browse a hardware store online library, the URL typically looks something like this: http://www.example.com/tools/hammer/. Using a URL rewrite rule, the web server converts this URL to a specific format, so it can retrieve the data from the back end database and display the tool details to the website visitor:

http://www.example.com/library.php?tools=hammer.

In this example, we can determine that the subdirectory (‘/tools’) in the first URL is actually a parameter in the library.php file that accepts inputs, which in this case is the tool name (‘hammer’).

During the scan, Invicti sends normal HTTP requests to the web application to mimic an attacker, ensure that such requests are accepted by the web application, and all parameters in the URLs are properly scanned. It is also possible to scan pages which have more than one parameter in the URL.

For further information on how URL Rewrite Rules work in Invicti, see How Invicti handles URL rewriting.

Problems with URL Rewrite Rules

This table lists and explains the problems that can occur when automated web vulnerability scanners attempt to scan websites that employ URL rewrite technology and rules.

Problem

Description

Parameters in URLs are not scanned

A common problem web vulnerability scanners have when scanning web applications that use URL rewriting technology is that scanners are unable to identify parameters in the URLs. Scanners assume that the URLs are directories rather than parameter names or values, and leave them unscanned.

 

For example, when scanning the URL http://www.example.com/tools/hammer/ the scanner would think that both tools and hammer are directories, while in reality tools is a parameter and hammer is a value.

Prolonged vulnerability scans

This problem can lead to prolonged scans and incorrect scan results. For example, if the web vulnerability scanner is scanning a tool database that contains 100,000 tools, since the scanner is unable to identify that there is a parameter and a value in the URL, it would think that they are all different pages, so it will try to crawl them and scan them all.

 

If memory problems and other exceptions are not handled properly by your scanner, this could also lead to your software crashing, leaving you with no results and a number of wasted hours.

If you do not configure URL rewrite rules in Invicti, it will heuristically identify the pattern and will limit the scan to avoid having prolonged scans and incorrect results.

Configuring URL rewrite rules is a difficult process

Since URL rewrite technology has become really popular in web applications, many commercial web vulnerability scanners allow users to configure the scanner so it can identify the parameters in the URLs and scan them.

 

But even though web vulnerability scanners can be configured to scan websites using URL Rewrite Rules, there are several other problems users can face:

  • Configuring of URL rewrite rules support is very difficult
  • User must know how to write regular expressions
  • User should have access to web server configuration files

Unless you are the developer of the web application itself or have a deep understanding of the web application, and unless you have direct access to the configuration files, it is impossible to configure URL rewrite rules on the scanner. And, even if you do know how to do it, configuring rewrite rules is a very difficult and time-consuming task.

Web applications are

not properly scanned

for vulnerabilities

Assuming you manage to configure URL rewrite rules in your web vulnerability scanner, there are further problems, or at least, there are a number of limitations to how the scanners scan the web application.

 

As a security precaution, web applications do not accept HTTP requests which are already ‘translated’, such as http://www.example.com/library.php?tools=hammer. By default .NET web applications do not accept such HTTP requests. The problem is worsened when scanning MVC web applications because such applications use a different approach to URL rewriting.

 

While Invicti can scan MVC web applications, many other web vulnerability scanners cannot, even when URL rewrite rules are configured.

 

Once you configure the URL rewrite rules in your scanner, it sends a type of HTTP requests called translated queries. Even though the web application security scanner reports that the scan ran successfully, most of the HTTP requests were denied and the parameters in URLs were not scanned, providing you with a false sense of security.

By default, Invicti can heuristically identify URL rewrites in the target website. Invicti can also automatically configure the settings. However, if you can configure the URL rewrite rules manually as explained here, the scan will be more efficient.

Specifying the Parameter Type

Specifying the correct parameter type makes the web application security scan more accurate.

  • For example, imagine specifying the following pattern: ‘/{PARAM}/{ID}’, where ‘{PARAM}’ is the parameter that will be scanned and ‘{ID}’ is its value. If the parameter type is not specified, leaving the default ‘Any’, this means that both the below URLs will match this URL Rewrite configuration:
  • http://www.example.com/products/18
  • http://www.example.com/products/date.js
  • The above matching is too generic and might lead to incorrect scan results. For example Invicti might not scan some files because after collecting enough samples (i.e. /product/1, /product/2, /product/3 etc) it will assume that /products/date.js is just another value of the product parameter, due to the generic matching.
  • In this case, if you set the parameter type to ‘Integer’, you avoid this problem, since Invicti will only expect integers when scanning such parameters. Then, when Invicti detects something else, such as /product/date.js, it will recognize it is a file.

Encoded URLs

If you are manually configuring URL Rewrite Rules, and your website URLs use encoded values, always specify the decoded value.

  • For example, suppose the Website URL is:

http://www.example.com/user/john%2dDoe.

  • The correct URL rewrite rule should contain the decoded character as follows:

http://www.example.com/user/{firstname}{lastname}

  • This rewrite rule is incorrect because it contains the encoded character:

http://www.example.com/user/{firstname}%2d{lastname}

URL Rewrite Fields

This table lists and describes the fields in the URL Rewrite tab.

Field

Description

Root Path Max Dynamic Signatures

If a URL block in the root path contains more items than this limit, it will be identified as a URL rewrite parameter. It must be between 1 and 10,000.

This field is displayed only in the Heuristic tab.

Sub Path Dynamic Signatures

If a URL block in the sub path contains more items than this limit, it will be identified as a URL rewrite parameter. It must be between 1 and 10,000.

This field is displayed only in the Heuristic tab.

Block Separators

Enter separators to use to split the URL into blocks.

This field is displayed only in the Heuristic tab.

Analyzable Extensions

If the URL contains a file extension, it will be analyzed only if the respective extension is in this list.

This field is displayed only in the Heuristic tab.

Enable Heuristic URL Rewrite detection

Invicti will try to automatically detect other URL rewrite rules if this option is set.

This field is displayed only in the Custom tab.

Placeholder Pattern

This contains the relative path with placeholders for URL rewrite parameters.

This field is displayed only in the Custom tab.

RegEx Pattern

This is a regular expression used for matching the URL rewrite parameters.

This field is displayed only in the Custom tab.

How to Configure URL Rewrite Rules in Invicti Enterprise
  1. From the main menu, select Scan > New Scan
  2. In the Options section, select URL Rewrite.
  3. Select an option: None, Heuristic, or Custom:
  • If None is selected, then no rules will be applied
  • If you keep Heuristic as the default, then:
    • Invicti Enterprise will automatically populate the fields:
  • If you select Custom:
  • Select Enable Heuristic URL Rewrite Detection so Invicti will attempt to automatically determine other URL Rewrite rules. Both Custom rules and Heuristic rules will be in effect. Or disable, so that only the Custom rules will be in effect.
  • In the Placeholder Pattern and RegEx Pattern fields, enter the relevant information
  • Click New to enter further rules
  1. From the Exclusions, click New, if required. 
  2. Complete as required and select Launch.
How to Configure URL Rewrite Rules in Invicti Standard
  1. Open Invicti Standard.
  2. From the Start a New Website or Web Service Scan dialog, in the Scan Settings menu, click URL Rewrite.
  3. Heuristic URL Rewrite Support is the default option. Select Custom from the drop-down.
  1. Check Enable Heuristic Rule Detection, so the scanner still tries to automatically detect any additional URL Rewrites on the target website and use them in conjunction with the ones you configured.
  2. Select Copy Rules to copy custom URL rewrite rules listed below.
  3. Select Exclusions to exclude certain URLs from URL Rewrite Detection. 
  4. Complete as required and select Save on the dialog box.
  1. Select New to launch the URL Rewrite Rules wizard.
  2. In the first step of the wizard, specify a URL that matches the URL rewrite rule you want to add, such as ‘http://www.example.com/tools/hammer/’

If you wish to configure the URL Rewrite rules manually in Invicti, without using the wizard you can simply click on the Placeholder Pattern and RegEx Pattern input fields to populate them manually.

  1. In the second step of the wizard, you have to specify which of the path segments is a parameter and its type, by following the below procedure:
    • Check the URL path segment that contains a parameter value
    • Specify the parameter name
    • Specify the parameter type from the drop-down menu in the Parameter Type column
  1. If there are multiple parameters in the URL, specify all of them. An example of this is (http://www.example.com/departments/equipment/tools/hammer/), as illustrated.
  1. Select Finish so the placeholder pattern and regular expression are automatically generated. Click on any of the values to manually modify them, for example, to manually write a regular expression.

If required, click Test to test the URL Rewrites Rules. When running a test, put an example URL next to the Test button.

Invicti Help Center

Our Support team is ready to provide you with technical help.

Go to Help Center This will redirect you to the ticketing system.