I run john the ripper on multiple boxes at once by splitting up the input password hash files by type (NT, LanMan, DES, md5, etc.)

Once I’ve let the individual systems run for a while, I run reports using  john - -show. This means I need to combine the john.pot files from each system into a unique file on the system I’m running the reports on.

So, I ran the (fairly standard) commands:

$ cat john1.pot john2.pot john3.pot > combined.pot

$ cat combined.pot | sort | uniq > john.pot
sort: string comparison failed: Illegal byte sequence
sort: Set LC_ALL=’C’ to work around the problem.
sort: The strings compared were `1234567892031276d66b123456789:user’ and `abcdefghijklmnop4ab38:l\4327367qrstuvwxyz’.

Oops! I wasn’t expecting that command to bomb out like that. To fix the sort, do exactly as sort suggests. Type:

$ export LC_ALL=’C’

What’s going on here?

I found this reference webpage on Google. Here’s the relevant bits:

LC_ALL

This variable determines the values for all locale categories. The value of the LC_ALL environment variable has precedence over any of the other environment variables starting with LC_ (LC_COLLATE, LC_CTYPE, LC_MESSAGES, LC_MONETARY, LC_NUMERIC, LC_TIME) and the LANG environment variable.

If the LANG environment variable is not set or is set to the empty string, the implementation-dependent default locale is used.

If the locale value is “C” or “POSIX”, the POSIX locale is used and the standard utilities behave in accordance with the rules in POSIX Locale , for the associated category.

Related posts:

1. John the Ripper Password Cracking Now Obsolete
2. Complete Unix Cron Job Crib Sheet
3. AUC #2: ls

Shopping List
Before we start, make sure you have the following ingredients ready to go:

• Fydor’s nmap – The true king of network scanners
• OpenVAS – The glamor queen of vulnerability assessors

Start Here
The first thing you need to determine is if you will be performing a black box test or a glass box (often referred to as crystal box) test. I will only quickly go over the differences here.

The Fairest Box of All
Black box testing is the “simplest” as you are only provided the IP Addresses in scope of the assessment. That’s it. You are not told how many Windows systems to expect, how many firewalls may be in your way or even if an IPS may shut down your scan if you are too aggressive with your timing options (we’ll touch on that later in this article.) It is difficult to accurately predict how long this may take as the composition of every network is different. An extreme example is if you are scanning through ocean floor fiber it could take quite some time when each packet suffers 250ms+ latency.

Glass box testing is typically much quicker to perform as your client will be providing architectural details of how the network is bolted together. It’s best to ask for network diagrams plus a real live human being to help you decipher them. (I mean seriously, how many UML network diagrams have you seen?) This is important not only because they may use abbreviations unknown to you or how closely the diagram aligns with reality or even all the labels they scribbled on by hand are subject to the lost art that is penmanship. Err, graphmanship.

Scope Up!
Once the transparency of your target environment is agreed upon, you will need to define the scope of which portions of the target organization’s network is within scope for the assessment.

Ask for hosts which must NOT be scanned as they will fall over seizing due to poor IP/TCP stack implementations or other programming bugs. Place into “exclude.list”

Place the authorized IP address ranges and domains you were provided into “inscope.list”

Just a Minor Technicality
Ok, we are now ready to dominate the world with our awesomeness! Or at the very least impress your client with a thorough report…

If you are scanning a class B network (10.10.X.X) or smaller, you will likely want to choose -T4 for your speed setting. If you are attempting to map the Internet, stop now. Fydor has beaten you to it

The Need For Speed
The –n switch will disable DNS lookups, reducing overall start to finish time as you no longer need to wait for all those UDP DNS request / responses to file in.

Setting how fast nmap chucks packets onto the network is controlled with:
-T, –scan-delay and —min-hostgroup
-T5 is balls to the wall and –T0 means you never want it to finish…

A safe choice is –T3 (the default) but again you will likely want to use -T4 to save time.

Top Heavy Servers
Scanning for every in-scope IP Address plus 65,535 port combinations to gather an inventory of live hosts could last longer than your contract engagement. You may need to pick 20 to 30 ports. In a perfect scenario, at least one of these ports will be open on every server that you need to find. Use -p to specify destination ports; a good starting list are the following UDP (U:) and TCP (T:) ports:
-p U:53,111,137,T:21-25,80,135,139,443,445,3389,8080

If your client has no idea what services they expect to find on their network, you now have permission to cry like a baby because they will have problems with your report regardless of what you find. T-T
All seriousness aside, thanks to Fydor’s Internet mapping project I mentioned earlier, you can just substitute the defined list with Fydor’s results by adding:
–top-ports 30 (where 30 is the number of ports you want to scan)

I Know That Look…
nmap is more than just a simple inventory scanner. It can also guess what the target operating system and it’s network accessible service names and versions are. It does this by looking at distinguishing features of a packet or connection (such as what the starting TTL value is or how it responds to an unsolicited RST packet sent to a closed port.) This technique is referred to as fingerprinting. If nmap is unsure about a target host, it will guess (and tell you as much) and if it is completely baffled it will politely ask you to submit a fingerprint to add to the nmap database along with what you determine the target to be (by other means.)

-sV will attempt to match the name and version of each open network service that is discovered
-O will attempt to match the operating system of target boxen

New Implants
nmap has received some really neat upgrades since version 4.something such as the ability to run lua scripts during the scan process. An example:
–script smb-os-discovery

Make sure you are using the latest scripts by running:
nmap —script-updatedb

Putting it all Together

Thus far, we have compiled the following nmap command:

nmap –iL inscope.list —excludefile exclude.list –v –sS –n —reason –T4 —script smb-os-discovery –sV –O —top-ports 30

Sweep 1: I highly recommend swapping out –sS with –sL before you run your final scan. Why? A list of hosts to be scanned is a great piece of information to include in your report to PROVE that you only touched systems that you were expressly authorized to touch. Yes, this is just feeding back information they originally provided you… Hey, this is why consultants get paid the big bucks right?

Sweep 2: swap out –sS with –sP to icmp scan for which hosts you are able to receive echo replies from. Consider sending this list to your client immediately to verify that your are in the ballpark prior to the “real” scan. Obviously, this can be skipped if a networking device (such as a router or firewall) is dropping all your icmp echo request probes.

Sweep 3: -sS for the win! This is your traditional half open SYN scan. Nmap will send out a TCP SYN packet to each IP address and port combination as you specified to determine if it returns a SYN+ACK packet, indicating an open port. If it doesn’t hear back from the target, it’s assumed to be closed. (Yes, this is where the increased timing option pays off.)

Wrapping Up

If you are running up against an IPS blocking your aggressive scans by sending you RSTs or dropping your packets silently, try setting –f to fragment (split up) your packets to a smaller portion of their maximum transmission unit (MTU) or size. This may just be enough to fool (or crash) that pesky IPS into letting them all pass on their merry way for total target box domination. Make sure you have permission to try this from the client first…

This was intended to be a pick up and go article and not dig too deep into how network scanning works nor the intricacies of nmap. If you want WAY more info than I’ve provided here, I highly recommend buying Fydor’s nmap book. If you can’t spare a few bucks, it’s also available online for free.

Stay tuned for Part 2, which covers OpenVAS.

Related posts:

1. Thousands of Sacrificial Lambs
2. Writing a DNS sniffer
3. Detecting bad TCP sessions

A laptop reliability study conducted by Squaretrade reveals most reliable laptop manufactures. The sample size was from over 30,000 devices over a 3 year lifetime. No big surprises here as the cheaper bargain laptops and netbooks round out the highest failure rates and the more expensive systems last longer. The linked PDF is full of pretty graphs which is always a win.

Of personal importance is my current laptop manufacturer of choice the Macbook Pro‘s ranking in the study.

[via Engadget]

Related posts:

1. Dell Mini 10 – My Next Toy Purchase
2. Computer Security At Hotels
3. Network Scanning with nmap

Google’s Chrome OS project, first announced in July, will become available for download within a week, we’ve heard from a reliable source. Google previously said to expect an early version of the OS in the fall.

(Via Linux.com :: Features.)

Related posts:

1. Updated Google Docs Drawing
2. I sure miss grep in M$ Windows
3. alltop

AUC 2 – Arbitrary Unix Command #2

Today I wanted to talk about a command that we use every day: ls. Yes, this isn’t a very arbitrary pick, but I decided sharing useful commands to be more important than obscure.

If you aren’t a regular unix user, the command ‘ls’ is equivalent to the Windows ‘dir’ command when using cmd.exe.

It is so frequently used that nearly 10% of my bash history contains ‘ls’. When considering the unix mantra that every command should do only one thing, but do it well, ls is surprisingly robust.

While you may be quite familiar with this command, I always recommend checking the man page for a unix command as command switches may differ depending on your system (ie: gnu commands) and more importantly you may learn something new and useful. You can do so by typing:

man ls

This should look familiar. The -l switch produces a vertical list (just like dir /p in cmd.exe.) The -a switch includes hidden files, which are typically preceded by a period. I won’t go into detail on how to read the permissions as the internet has many such guides.

Lets say you are interested in only file names of a particular directory. The -1 switch will declutter the output quite nicely and only spit out a list of filenames and not any of the meta information such as file permissions, file ownership, size, and last file modify time.

Ok. So what? Perhaps you are interested in performing a command against every file (not directory) in your current working directory. The first of the two commands above produces a single line of output, space delimited list of filenames. (I have see Internet postings frowning upon the use of xargs to convert newlines into spaces as I have, but still need to discover a better way.)

The second command builds upon the first by embedding it in backticks (usually shared with the tilde key) in use with a for loop. The command string inside the backticks is executed first and produces a space seperated list for the x variable in the for loop to iterate through. The result is a copy and paste-able start for a bash script. I’ve used the head command here as a placeholder example.

Another useful application of ls is to produce a full, recursive directory listing.

Saving the best for last is my favorite ls command. The -t switch sorts by modify time, newest at top and the -r switch reverses the order. This is the perfect way to quickly find your recent files. It’s such a time saver, I rarely use ls -al anymore in favor of the above.

Until next time!

Related posts:

1. AUC #1: paste
2. Complete Unix Cron Job Crib Sheet
3. Quick note on sorting john.pot files

On the hunt for a travel netbook? After testing out early models with squished keyboards I decided to wait for a model with useable size and 10″ seems to be about perfect. I originally thought I wanted a fully loaded netbook until I realized that wasn’t the point. Netbooks are not intended to replace laptops. They are intended to be ultra portable with prices making them nearly disposable. Netbooks are perfect travel companions for photographers who need to dump the days pictures onto a backup external hard drive and review images on a screen larger than the thumbnail preview on the back of their camera.

Since a unit’s size is an important feature, how much juice can you realistically expect from the form factor? Find the list of which models last the longest after the jump.

A recent article was posted with a chart of netbook battery life of popular netbook models. They identified the MSI Wind U115 Hybrid as the pack leader (thanks to it’s 6-cell battery and use of SSD hd to save power) however, other upcoming models that look promising such as the Samsung NC10, the Asus 1000HE (E for extended battery) and the Dell Mini 10 were not tested.

It’s interesting to note how poor the Dell Mini 12 performed. The larger screen and higher resolution is likely to blame. This highlights an important decision future netbook buyers will need to consider: Longer battery life or a better looking screen?

[LINK]

Related posts:

1. Dell Mini 10 – My Next Toy Purchase
2. A Romance Flowchart: When Is It Inappropriate to Use Your iPhone?
3. Study indicates MacBooks are pretty reliable

AUC 1 – Arbitrary Unix Command #1

Up first is the command that took my 15 minutes to remember at work today when I wanted to take a list with one column and seperate it into a list with two columns. Here, a visual will make this easier to explain:

This is a list of a person and their favorite fruit. You want to convert this into a table like this:

But your table is 100,000 lines long and you don’t want to do it by hand, or mess with any complex regular expressions in sed, awk or a scripting language.

The unix command paste does exactly this. Here’s how simple the syntax is:

cat 100k_file | paste – - > lookup_table

Now that’s pretty simple. Just change the number of “-”‘s to how many columns you want.

paste can do some other things but this is how I found it to be useful. If you have another tip on paste, the comment box below awaits.

Related posts:

1. AUC #2: ls
2. Complete Unix Cron Job Crib Sheet
3. Detecting bad TCP sessions

I recently stumbled upon this website (OpenSourceMac.org) which is a clean list of useful open source software that has been written or ported over to Apple OS X. This is very handy to email to your friend which you recently converted to Apple

Related posts:

1. Free OSX Tools I Use Everyday
2. Writing a DNS sniffer
3. iPhone: Harbinger of Doomsday Malware

Cron is a unix program that is used to automatically run commands or scripts at predetermined times. Common uses include running nightly backups, log searching or other system resource intensive tasks when everyone is at home sleeping. Each unix user has their own crontab list. If you are new to scheduling tasks in Sun Solaris, Red Hat Enterprise Linux, Fedora, Ubuntu, Debian OS X, FreeBSD or any other flavor of unix, the following quick cron guide should help.

To check what is currently scheduled for execution for one user, open a terminal and type:

crontab -l -u <username>

Any task that requires root privileges will need to be put in the root uer’s crontab. Let’s add a simple backup scheduled task as an example. To open the cron file as the root user with your default text editor, type:

sudo crontab -e

At the top of the crontab file (or if it is empty) paste in the following:

# +- - - - - - minute [0-59]   | Special Entries: @reboot, @yearly, @monthly,
# | +- - - - - hour [0-23]     |                  @weekly, @midnight, @hourly
# | | +- - - - monthday [1-31]  ---------------------------------------------
# | | | +- - - month [1-12, jan,feb,mar,apr,may,jun,jul,aug,sep,oct,nov,dec]
# | | | | +- - weekday [0-6, sun,mon,tue,wed,thu,fri,sat,sun]
# * * * * * /full/path/to/command args >> ~/command.log

This Crib Sheet will serve as a crontab syntax reference every time you list or edit the crontab but won’t be interpreted by cron since each line starts with a poundsign (#) character indicating a comment line. The first five columns allow you to define the minutes, hours, days of the month, months, and weekdays that you want your job to run. The sixth column is the command or script to execute at the appointed time or times.

Here’s an example crontab line that runs the backup.sh script every twenty minutes on April 13th, 14th and 15th:

0,20,40 * 13-15 4 * /home/tom/scripts/backup.sh >> /var/log/backups.log

The last part may look new if you are unfamiliar with pipes. The double greaterthan signs will append the output of backup.sh into a file called backups.log located in the /var/log/ directory. While you can chain together many unix commands directly in the crontab file, I recommend using a separate file (ie: in your scripts directory) to perform more complicated tasks.

Here’s another cron job example that searches for files in anyones home directory modified in the last 24 hours and puts a report in your home directory:

0 1 * * * /usr/bin/find /home -mtime -1 -type f -exec ls -lh {} \; > \
/home/tom/new_files_report_`date +%Y-%m-%d`.txt

Hope that helps! Comments welcome.

Related posts:

1. AUC #2: ls
2. AUC #1: paste
3. Quick note on sorting john.pot files

Related posts:

1. I sure miss grep in M$ Windows
2. A Romance Flowchart: When Is It Inappropriate to Use Your iPhone?
3. Pretty graphs for the malware detection (ftw?)