Manual Crawling with Netsparker Desktop in Proxy Mode

Netsparker Desktop has a built-in proxy that allows you to manually crawl a target and scan it. Manual crawling is a process that is used to scan parts of a web application that for some reason cannot be crawled automatically. During a manual crawl the scanner will only scan the URLs that you feed through the proxy, though you can also combine both the automated and manual crawl as this posts explains. Manual crawling is typically used when:

  • You want to scan sections of the web application that were not automatically crawled,
  • You want to scan a limited number of URLs and parameters,
  • You want to launch a Controlled Attack.

Manual Crawling with Netsparker Desktop

To do a manual crawl with Netsparker Desktop web application security scanner follow this procedure:

Step 1: Configure a Browser to Proxy the Traffic through the Scanner

By default when Netsparker's proxy is started all the popular browsers such as Internet Explorer, Google Chrome and Mozilla Firefox will automatically proxy traffic through it. Therefore you do not need to manually configure the browser's proxy settings.

If not, open the scanner's Options from the Tools drop down menu, click to the Internal Proxy node and tick the option Register as System Proxy. If you are using a browser that is not automatically proxying the traffic through Netsparker's proxy, configure it to proxy the traffic to port 10010, Netsparker's proxy default port.

Tip: When the proxy is started the listening port will be shown on the Proxy button.

The listening port of the built-in proxy is shown on the button

Step 2: Start Netsparker Desktop Scanner in Proxy Mode

Once the browser is configured start the Netsparker Desktop web application security scanner and specify a target URL in the Start a New Website or Web Service Scan dialog. The target URL will be used to filter the requests received from the web browser. Therefore if for example you want to scan, enter such URL and if you browse pages from other domains Netsparker will not add them to the Scan Scope.

Note: All the requests captured from proxy will also be filtered according to the configured Scan Scope.

To start Netsparker's proxy select Manual Crawl (Proxy Mode) from the Start Scan drop down button in the Start a New Website or Web Service Scan dialog.

Select Manual Crawl (Proxy Mode) to start the proxy server and do a manual crawl

Step 3: Start Browsing the Pages You Want to Scan

You can start browsing the pages you want the scanner to scan via your web browser. At this stage you can notice the pages being added to the sitemap as you browse them.

The scanner will start crawling the pages you access via the web browser

Scan the Manually Crawled Pages

Once you have crawled all the pages click the Resume button from Netsparker's toolbar or press F5 so the scanner proceeds with attacking the pages listed in the sitemap.

Resume the scan once you have manually crawled the pages that need to be scanned using a web browser

Using Both Automated and Manual Crawling in a Web Security Scan

To crawl a website automatically and then add URLs from a manual crawl select the option Crawl and Wait  from the Start Scan drop down button in the Start a New Website or Web Service Scan dialog.

Use the Crawl and Wait option to use both automated and manual crawling when scaning a website

In this mode Netsparker will crawl the website automatically and will stop before starting the attack phase, allowing you to switch on the proxy by clicking the Start Proxy button in the toolbar. At this stage follow the procedure above to configure a browser to proxy the traffic through Netsparker Desktop and browse the pages you want to add to the sitemap for the scan.


Keep up to date with web security news from Netsparker