Netsparker's discovery service enables you to become aware of your enterprise's online collateral, web applications, and services.
- As soon as you register with Netsparker Enterprise, the system begins the discovery process with your commercial email, immediately suggesting websites that might also belong to you.
- Once you start adding websites, the system makes new suggestions based on those websites.
- Netsparker analyzes your configuration and data, then suggesting further websites that might also belong to you.
This topic explains how Netsparker Enterprise discovers web assets and services.
To manage the Discovery Service in Netsparker Enterprise, see Managing Discovery Service in Netsparker Enterprise. In order to create websites via the discovery service, see Creating Websites via Discovery Service.
How the Discovery Service Works
There are three main resources that Netsparker uses to discover your web assets.
- Your Email's Domain
- Second-Level Domain of Existing Websites
- Knowledge Base
As soon as you register with Netsparker Enterprise, the system begins the discovery process with your email address, immediately suggesting websites that might also belong to you.
The service takes the domain name (e.g. netsparker from firstname.lastname@example.org) and starts querying. At the same time, the service queries the IP address of discovered websites. For example, the service queries the IP address of Netsparker and lists the results in the discovered websites section.
Second Level Domain
While the domain name of your email address is used to query the discovery service, Netsparker further uses this domain address to look for additional websites. For example, when the service discovers netsparker.com from your email address, it also starts looking for websites, such as api.netsparker.com and test.netsparker.com. Then, it lists these websites under the Discovered Websites.
Links in the Knowledge Base
Although Netsparker aims to crawl every part of the target web application to identify vulnerabilities, the scanner still allows you to determine the scope of the scan. Even if you do, Netsparker lists these websites to inform you which links remain uncrawled.
You can use these links and feed the discovery service so that Netsparker tries to find additional websites. Once you do that, you will be able to see additional websites in the discovered websites.
When you add, for example, freecsstemplates as a second level domain into Netsparker, the discovery service start querying and populating the discovered websites as the following:
Public Data Sources
Three main sources act as information for Netsparker to discover websites that may be related to you. But, which service does Netsparker use to inquiry in order to list these websites?
Firstly, the Discovery Service is a separate service that works completely independent from Netsparker Enterprise and currently runs here: https://services.netsparker.cloud/
Netsparker queries the discovery server and lists the results in the application. There are 2 main public sources where the Discovery Service collects this data: Certificate Transparency Logs and Project Sonar Reverse DNS.
Certificate Transparency Logs
This is a registration system in which all certificate authorities have to register every SSL certificate they sign. In this registration system, logs are kept as binary. For example, in the following query, logs of record number 696712242 (which is associated with netsparker.com) can be seen: https://ct.googleapis.com/rocketeer/ct/v1/get-entries?start=696712242&end=696712242
The parsed view of this record is as follows: https://crt.sh/?id=1509541883
Discovery Service downloads these logs, parses them and saves them into the database. By doing this, for example, when “www.google.com” is added to Netsparker Enterprise, Discovery Service gets the Organization (O) and Subject Common Name values from this website's SSL certificate and filters websites that match the organization name or subject common name in that SSL certificate from the backend database and shows them as a discovered website in the UI.
For example, these records will be listed under the Discovered Websites section when www.google.com is added from the New Websites page:
Project Sonar Reverse DNS
Project Sonar is a project that compiles public data on the internet under different categories. The Discovery Service uses the Reverse DNS data of this project. This data is also parsed and saved into the backend database like CT logs.
Just as DNS resolves the IP address from a domain name, reverse DNS is the opposite, resolving the domain name from an IP address. In this way, if there are multiple websites running on the same IP address, it becomes possible to detect all of them by doing a reverse DNS lookup.
For example, the screenshot below shows that the IP address of “netsparker.com” is 184.108.40.206, but when we make a reverse DNS query to this IP address, we can detect another domain name:
All the data compiled by using this technique is in Discovery Service’s backend database. Thus, using the IP addresses of newly added websites, the Discovery Service can discover other sites related to this website.
Discovery Service FAQ
Question: I have example.com. However, Netsparker Discovery Service could not find this domain. Why?
- As specified above, the Discovery Service is a separate service that works completely independent from Netsparker Enterprise. Netsparker inquiries third-party databases to identify websites that may be related to you.
- Secondly, the Discovery Service does not provide a 100% guarantee that Netsparker will discover all of your websites. If only third-party databases have information related to your website, Netsparker can discover and list them.
- Also, please note that the Discovery Service can find those websites that are public.
Question: In order to utilize the Discovery Service in Netsparker On-Premises, which URL/port should I permit?
While using Netsparker On-Premises, you should select the Enable Discovery Service under the General Settings.
Also, you should enter https://services.netsparker.cloud to the Discovery Service URL so that Netsparker can carry out the query to discover websites.