Unix Training For Data Scientists

This is a quick introduction to the most useful unix commands for folks doing data manipulation in a unix environment. It is not a comprehensive introduction to Unix operating system (not even when constrained to topic at hand).

All files used in the examples are provided in the files directory of this repository.

For details explanation of the commands described in the following chapters, users should utilize the "man" command to show the manual page for a given command. For example:

$ man grep

The above will show a detail manual page for the grep command.

Text File Tranformation/Manipulation

• Input/output redirection
• File hex dump
• Unix vs. Dos/Windows text files
• File & Directory permissions 🆕
• Translate characters
• Removing funny/weird characters
• Searching files
• Selecting columns from a file
• Selecting top/bottom rows from a file
• Counting lines in file
• File difference
• File checksum
• Sorting files

Misc Commands

• Finding/searching files across directories
• Directory commands
• Timing Execution

General Purpose Data Manipulation "Languages"

• Stream Editor (sed)
• AWK
• PERL
• PYTHON

Hadoop Commands

• Hadoop filesystem
• Yarn commands

Development Environment

• shell
• Source Control
• [Integrated Development Environments (IDE)][ide.md]

Remote Session Management / Connection

• Remote shell (ssh)
• TMUX 🆕
• SCREEN

Virtualization Frameworks

• DOCKER
• Virtual Box