WhatWeb: Web Technology Identifier

WhatWeb is a next-generation web scanner that identifies websites and recognizes a vast array of web technologies. It uses over 1700 plugins, each designed to detect specific components, including content management systems (CMS), blogging platforms, statistic/analytics packages, JavaScript libraries, web servers, embedded devices, and much more. This makes it an invaluable tool for reconnaissance in web application security assessments.

Key Features

• Extensive Plugin Database: Over 1700 plugins for identifying diverse technologies.
• Flexible Output: Supports various output formats, including JSON.
• Aggressiveness Levels: Allows users to control the invasiveness of scans.
• Customization: Supports custom headers and other request modifications.

Installation

WhatWeb can be installed using RubyGems, or by cloning its Git repository.

Using RubyGems

This is the simplest way to install WhatWeb if you have Ruby installed.

gem install whatweb

From Git Repository

For the latest development version or if you prefer a source installation:

git clone https://github.com/urbanadventurer/WhatWeb.git
cd WhatWeb
bundle install

Common Commands and Options

Scan a Single Target

whatweb <target_url_or_domain>
# Example:
whatweb example.com

Scan Multiple Targets

whatweb <target1_url> <target2_url>
# Example:
whatweb example.com example.org

Scan from a File

Provide a file containing a list of target URLs, one per line.

whatweb -i target_list.txt

Aggressiveness Levels (-a)

WhatWeb offers different levels of aggressiveness, controlling how intrusive the checks are.

• -a 1: Stealthy (default) - Performs basic, passive checks.
• -a 3: Aggressive - Includes more intrusive checks, potentially revealing more information but also increasing the chance of detection.
• -a 4: Full - Attempts all plugins and checks, maximizing information gathering.

Verbose Output (-v)

Increases the verbosity of the output, showing more details about the scanning process.

whatweb -v example.com

Output to JSON (--log-json)

Exports the scan results into a JSON file, which is useful for programmatic processing or integration with other tools.

whatweb --log-json output.json example.com

Advanced Example Usage

Here’s a comprehensive example demonstrating various WhatWeb options for an aggressive scan, suppressing errors, disabling colored output, logging to JSON, and including a custom HTTP header:

whatweb -a 4 -q --no-errors --colour=never --log-json=/dev/stdout --header "X-Bug-Bounty: portal-username=l0l4da" https://<target_url>

• -a 4: Sets the aggressiveness level to 4 (Full scan).
• -q: Suppresses verbose output (quiet mode).
• --no-errors: Hides error messages in the console.
• --colour=never: Disables colored output, useful for scripting or logging.
• --log-json=/dev/stdout: Outputs results in JSON format directly to standard output.
• --header "X-Bug-Bounty: portal-username=l0l4da": Adds a custom X-Bug-Bounty header to all requests.
• https://<target_url>: The URL of the target web application.

Testing WhatWeb Requests with mendhak/http-https-echo

To verify that WhatWeb is sending requests with specific headers or in a particular way, you can direct it to an HTTP echo server. The mendhak/http-https-echo Docker container is excellent for this purpose, as it reflects back all the request information it receives.

1. Run the http-https-echo container: Start the echo server, mapping a host port to its internal port.

   docker run -it --rm -p 443:8443 mendhak/http-https-echo

   This command maps port 443 on your host to port 8443 (HTTPS) on the container. The --rm flag ensures the container is removed when you stop it.

2. Point WhatWeb to your local echo server: Now, execute your WhatWeb command, targeting localhost (or the IP of your Docker host) on the mapped port.

   whatweb -a 1 -q --no-errors --colour=never --log-json=/dev/stdout --header "X-Custom-Header: test-value" https://localhost

   Note: When testing HTTPS locally with http-https-echo, you might need to use curl -k (or equivalent) for the client sending requests to localhost to ignore self-signed certificate warnings. If WhatWeb has issues with self-signed certs, try using http://localhost if the echo server is configured for HTTP on that port.

   The console output of your docker run command will display the exact request WhatWeb sent, including all headers, paths, and methods. This helps confirm that your WhatWeb command is constructed correctly.

---

Disclaimer: WhatWeb is a reconnaissance tool. Its use should be restricted to legal and ethical purposes, such as authorized penetration testing, bug bounty hunting on explicitly allowed scopes, or security research on systems you own or have explicit permission to test. Unauthorized scanning of web applications is illegal and unethical.