SUPPORT

24/5 Hotline Support Service

+44 (0)20 3588 3841

Open a Support Ticket

support@netsparker.com

Manual Crawling in Proxy Mode

Netsparker Standard has a built-in proxy that allows you to manually crawl a target and scan it. Manual crawling is a process that is used to scan parts of a web application that cannot be crawled automatically. This may be for several reasons.

It could be because:

  • It uses a third-party plugin such as Flash or Silverlight (currently unsupported by Netsparker)
  • It uses forms that required DOM simulation
  • If the website does not provide a link to a sub-directory and it also needs to be scanned

Use manual crawling when you want to:

  • Scan sections of the web application that were not automatically crawled
  • Scan a limited number of URLs and parameters
  • Launch a Controlled Attack

During a manual crawl, the scanner will only scan the URLs that you feed through the proxy, though you can also combine both the automated and manual crawl as this post explains.

For further information, see Manual Crawling and Security Scanning.

How to Run a Manual Crawl with Netsparker Standard

There are four steps:

  1. Start Netsparker Standard in Proxy Mode
  2. Configure a Browser to Proxy the Traffic Through Netsparker
  3. Start Browsing the Pages You Want to Scan
  4. Scan the Manually Crawled Pages

Step 1: Start Netsparker Standard Scanner in Proxy Mode

  1. Next, start Netsparker Standard, and from the Home tab, click New. The Start a New Website or Web Service Scan dialog is displayed.

  1. In the Target Website or Web Service URL, enter the target URL. The target URL will be used to filter the requests received from the web browser, so only those requests related to the scan will be added. Therefore, if, for example, you want to scan http://php.testsparker.com, enter this URL. If you browse pages from other domains in your web browser, Netsparker will not add them to the scan scope.

  1. Note that all requests captured from proxy will also be filtered according to the active Scan Scope. To start Netsparker’s proxy, from the Start Scan dropdown, select Manual Crawl (Proxy Mode).

Step 2: Configure a Browser to Proxy the Traffic Through Netsparker

  1. By default when Netsparker’s proxy is started, all the popular browsers such as Internet Explorer, Google Chrome and Mozilla Firefox will automatically proxy traffic through it. Therefore you do not need to manually configure the browser’s proxy settings
  2. If you are using a browser that is not automatically proxying the traffic through Netsparker’s proxy, configure it before starting the scan (enable Use Custom Proxy) to proxy the traffic to port 10010, Netsparker’s proxy default port.

  1. When the proxy has started, the listening port will be shown on the Proxy button.

Step 3: Start Browsing the Pages You Want to Scan

  1. Using your web browser, start browsing the pages you want the scanner to scan.
  2. If you look at the Sitemap panel, you'll notice these browsed pages are being added to as you browse them.

Step 4: Scan the Manually Crawled Pages

Once you have crawled all the pages, click the Resume button on the Scan tab, so the scanner proceeds with attacking the pages listed in the Sitemap.

How to Combine Automated and Manual Crawling in a Web Security Scan

  1. To crawl a website automatically, but also add URLs from a manual crawl, open the Start a New Website or Web Service Scan dialog, and select the option Crawl and Wait  from the Start Scan dropdown button.

  1. In this mode Netsparker will crawl the website automatically and then stop before starting the Attack phase. At this point, from the Scan tab, click Start Proxyto switch on the proxy.

  1. At this stage follow the steps in How to Run a Manual Crawl with Netsparker Standard to configure a browser to proxy the traffic through Netsparker Standard and browse the pages you want to add to the sitemap for the scan.

Using Selenium for Manual Crawling

Selenium testing framework allows you to record and play back the browsing of a web application. It is very popular with developers, QA engineers and others who are involved in the development and testing of web applications.

It is possible that you already have Selenium scripts to test your web application. These might take the form of certain flows within your application, such as multiple step forms or shopping cart-like functionality. You can use the Selenium IDE Firefox browser extension or any other driver to replay the recordings and capture all the browsed pages and parameters in Netsparker, and get them scanned automatically.

How to Use Selenium and Netsparker for the Manual Crawling of Web Applications

  1. Open Netsparker Standard.
  2. Start Netsparker Standard Scanner in Proxy Mode.

  1. Play the Macro on Selenium IDE.
    • Click Selenium IDE from the tools dropdown menu of your browser.
    • Click Play Entire Test Suite.

  1. Start the Automatic Vulnerability Scan.
    • Switch back to Netsparker when the micro is finished.
    • Check the Sitemap to confirm that the scanner captured the links.
    • Click Resume to resume the scan so Netsparker can start attacking the parameters.

For more information, see Scanning URLs in Selenium Playbacks with Netsparker Desktop.

Netsparker

Dead accurate, fast & easy-to-use Web Application Security Scanner

GET A DEMO