Mass HTTP Enumeration with Metasploit

On most penetration tests (particularly internal tests on large networks), it is useful to be able to connect to a large number of webservers, obtain basic page information (server header and title) so that any interesting or anomalous servers can be easily identified.

This can be achieved using some third party tools, and to an extent can be achieved using scraper. However, the auxiliary/scanner/http/title module will connect to ‘RHOSTS:RPORT’, perform a non-greedy match for the firsttag (which will return the first title that appears on the page), the return HTTP code, the Location header (if present) and the Server header (if present). It will optionally present this on the console if SHOW_TITLES is set and write to notes (http.title) if STORE_NOTES is set. It can be made as quiet as possible to allow background execution without filling up the console with output.

The benefit of using notes can be easily seen using commands such as:

notes -t http.title -S ':port=>443, :server=>"Apache.*?"' -R

and then launching another module specifically aimed at mod_ssl etc.

Note that this is first and foremost a title grabber, not a fingerprinting module. Therefore, no output will be generated if a title cannot be obtained, even if the web server is still alive. It should be used to identify interesting hosts based on HTML title, not to identify webservers which are responding to HTTP.

Output Format

Console

If SHOW_TITLES is set, the code will generate console output in the format shown in the example below:

[*] [46.228.47.114:80] [C:200] [R:] [S:ATS] TITLE

Format	Example	Description
C	[C:200]	The HTTP status code from the server.
R	[R:http://www.mwrinfosecurity.com/]	The contents of the Location header, if present. This is useful to see at a glance if a GET request redirects you to another URL.
S	[S:ATS]	The contents of the Server header, if present.
(everything else)	TITLE	The unescaped contents of the first HTML TITLE tag.

Notes

If STORE_NOTES is set, the code will inject a note into the notes database to cover each of the HTML titles that are obtained, as shown in the example below:

[*] Time: 2015-05-11 17:25:21 UTC Note: host=193.0.255.237 type=http.title data={:code=>200, :port=>80, :server=>"gws", :title=>"Google", :redirect=>nil}

The format is relatively self-explanatory.

Internals

Internally, this is a very simple plugin; it obtains a webpage and uses a capturing regular expression to obtain the title.

Benefits

At first glance, this is one of many tools and scripts that exist to meet this need. However, a dedicated metasploit module does offer some advantages, including:

• It can take advantage of the metasploit framework database. For example, it is possible to identify all hosts with an interesting title or that meets some other basic criteria and very easily set the RHOSTS parameter.
• It can be easily multithreaded; simply set the THREADS option.
• It uses Rex sockets, meaning that it will follow any internal pivoting that is set. This is particularly useful if a compromised host running meterpreter can connect to internal servers that are not otherwise accessible; any interesting web servers can be very rapidly identified in a lightweight manner.

Further Information

The pull request can be found on Github at http://mwr.to/4fe3.