CeWL

CeWL Description

CeWL (Custom Word List generator) is a Ruby based application that scans a specified URL to a specified depth and returns a list of words, which can then be used for password crackers like John the Ripper. If desired, CeWL can follow external links.

CeWL can also create a list of email addresses found in mailto links. These email addresses can be used as usernames in brute force attacks.

Another tool provided by the CeWL project is FAB (Files Already Bagged). The FAB extracts the contents of the author/creator fields from the metadata of some files to create lists of possible usernames. These usernames can be used in conjunction with the CeWL generated password list. FAB uses the same metadata extraction methods as CeWL. The FAB currently handles pre-2007 Office, 2007 Office and PDF formats.

CeWL is useful in security tests and forensic research. CeWL is pronounced “cool”.

Homepage: https://digi.ninja/projects/cewl.php

GitHub: https://github.com/digininja/CeWL

Author: Robin Wood (digininja)

License: Creative Commons Attribution-Share Alike 2.0 UK: England & Wales

CeWL Help

CeWL: a custom wordlist generator.

CeWL will scan the target site and generate the following lists:

• List of all unique words found on the target site
• List of all email addresses found in mailto links
• List of usernames/author information from metadata found in any documents on the site

The program groups words up to the specified group size.

Usage:

cewl [OPTIONS] ... <url>

Options:

	-h, --help: Show help.
	-k, --keep: Keep the downloaded file.
	-d <x>,--depth <x>: Depth to spider to, default 2.
	-m, --min_word_length: Minimum word length, default 3.
	-o, --offsite: Let the spider visit other sites.
	--exclude: A file containing a list of paths to exclude
	--allowed: A regex pattern that path must match to be followed
	-w, --write: Write the output to the file.
	-u, --ua <agent>: User agent to send.
	-n, --no-words: Don't output the wordlist.
	-g <x>, --groups <x>: Return groups of words as well
	--lowercase: Lowercase all parsed words
	--with-numbers: Accept words with numbers in as well as just letters
	--convert-umlauts: Convert common ISO-8859-1 (Latin-1) umlauts (ä-ae, ö-oe, ü-ue, ß-ss)
	-a, --meta: include meta data.
	--meta_file file: Output file for meta data.
	-e, --email: Include email addresses.
	--email_file <file>: Output file for email addresses.
	--meta-temp-dir <dir>: The temporary directory used by exiftool when parsing files, default /tmp.
	-c, --count: Show the count for each word found.
	-v, --verbose: Verbose.
	--debug: Extra debug information.

	Authentication
	--auth_type: Digest or basic.
	--auth_user: Authentication username.
	--auth_pass: Authentication password.

	Proxy Support
	--proxy_host: Proxy host.
	--proxy_port: Proxy port, default 8080.
	--proxy_username: Username for proxy, if required.
	--proxy_password: Password for proxy, if required.

	Headers
	--header, -H: In format name:value - can pass multiple.

    <url>: The site to spider.

FAB Help

FAB: Files Already Bagged

This script can be run on files already downloaded from the target site to create a list of usernames and email addresses based on the metadata they contain.

The FAB currently handles pre-2007 Office, 2007 Office and PDF formats.

Usage:

xx [OPTION] ... filename/list

Options:

	-h, --help: show help
	-v: verbose
	
	filename/list: the file or list of files to check

CeWL Usage Example

Starting the collection of words from the pages of the site https://site.ru, using only the pages, links to which will be found at the specified address (-d 1), to compose a dictionary that will be saved to the dic.txt file (-w dic.txt):

cewl https://site.ru -d 1 -w dic.txt

Starting the collection of words from the pages of the site https://site.ru, using the pages, links to which will be found at the specified address, as well as on the downloaded pages (-d 2), to compile a dictionary that will be saved to the specified file (-w dic.txt), while for each word the frequency with which it occurs (-c) will be shown, a list of found email addresses (-e) will also be compiled, which will be saved to the specified file (--email_file emails.txt) and a list will be created based on the information found in the document meta tags (-a), this list will be saved to the specified file (--meta_file meta.txt):

cewl https://site.ru -d 2 -w words.txt -a --meta_file meta.txt -e --email_file emails.txt -c

FAB launch, during which the document test.pdf located in the current folder will be checked, a field containing the name of the author of the document will be extracted from the meta-information of this document, the data will be displayed:

ruby /usr/share/cewl/fab.rb test.pdf

FAB launch, during which all *.doc documents in the /home/mial/Downloads/ directory will be checked, a field containing the name of the document author will be extracted from the meta information of these documents, the data will be displayed:

ruby /usr/share/cewl/fab.rb /home/mial/Downloads/*.doc

How to install CeWL

Installation on Kali Linux

The program is pre-installed on Kali Linux.

In the minimum versions, the program is installed as follows:

sudo apt install cewl

But the supplied package lacks the FAB utility, if you need it, then install it as follows:

wget https://raw.githubusercontent.com/digininja/CeWL/master/fab.rb
sudo mv fab.rb /usr/lib/ruby/vendor_ruby/

Run like this:

ruby /usr/lib/ruby/vendor_ruby/fab.rb FILE

Also make sure the ExifTool program is installed and that it is in your $PATH, to check it run:

exiftool -ver

Installation on Debian, Linux Mint, Ubuntu

sudo apt install cewl libimage-exiftool-perl

The FAB program is present, you need to run it like this:

fab-cewl FILE

Installation on BlackArch

The program is pre-installed on BlackArch.

sudo pacman -S cewl
gem install mime mime-types mini_exiftool nokogiri rubyzip spider

The FAB program is present, you need to run it like this:

ruby /usr/share/cewl/fab.rb FILES

Also make sure the ExifTool program is installed and that it is in your $PATH, to check it run:

exiftool -ver

CeWL Screenshots

CeWL Tutorials

• Advanced wordlist generating techniques