The Unix Shell

The Unix shell

If you are used to working with Windows, you might find working on the cluster a bit overwhelming, and this is because the cluster uses Linux, an operating system that instead of a graphical user interface (with icons, windows, and a start menu like Windows), is managed and accessed through a command-line interface (aka Unix shell) .

Unix-like operating systems such as Linux and macOS operate through the use of Unix commands. These commands consist of text-based instructions that users input into the shell (on Macs this is done with the Terminal app). Essentially, the shell is a program that takes these keyboard instructions and communicates them to the operating system to execute various tasks. This is how users can interact with the operating system, enabling them to perform actions like navigating the file system, handling files and directories (e.g., copying, moving, renaming), overseeing processes, configuring system settings, and accomplishing a wide range of other functions.

As you should know by now, understanding the Unix shell is crucial for working effectively on a cluster, for several reasons:

❯

File and Directory Management: Working with files and directories is a fundamental part of cluster usage. You'll need to transfer data, create directories, copy files, and manage permissions—all of which are done through Unix shell commands.
❯

Job Submission : To run computational tasks on a cluster, you typically submit jobs to a scheduling system. Job submission often involves crafting scripts or commands in the Unix shell to specify job requirements, such as CPU cores, memory, and runtime.
❯

Environment Configuration : Setting up your working environment, including loading software modules, configuring environment variables, and activating specific Python or R virtual environments, is typically done through shell commands.
❯

Monitoring and Management : You'll use Unix shell commands to monitor the status of your jobs, check system resource usage, and troubleshoot issues. This includes using commands like top, ps, and grep to manage running processes.
❯

Scripting : Many cluster tasks involve creating and running scripts, often written in Unix-compatible scripting languages like Bash or Python. Scripting allows you to automate repetitive tasks, manage data pipelines, and control job submission programmatically.
❯

Data Transfer : Transferring data to and from the cluster is a routine task, and Unix shell tools like scp, rsync, and wget are commonly used for secure and efficient data transfer.
❯

Security and Permissions : Understanding Unix file permissions and user/group management is critical to ensure data security and control access to your files and scripts on the cluster.
❯

Customization : You can customize your cluster environment by configuring your shell's profile files (e.g., .bashrc) to set aliases, environment variables, and custom functions that streamline your workflow.
❯

Debugging : When issues arise, knowing how to debug problems through log files and error messages using Unix tools and techniques is invaluable for troubleshooting.

In this section you will learn fundamental skills for working in the cluster thought the shell. There are plenty of tutorials and courses (both online and on site at UCL) that you can take to learn Unix commands. Some of our favorite options are listed below. You don't need to take all these courses, just pick one, but be sure to dedicate enough time to complete it fully before progressing to the next section.

Please note that all the courses below focus on teaching you Unix commands, but they also introduce the concept of Bash Scripting. We will take a more comprehensive look into Bash Scripting in a later section.

A link to the free online Linux Tutorial developed by Ryan Chadwick is below. We like it because it is very thorough but at the same time very easy to read.

TAKE THE COURSE

The UCL Advanced research Computing Center (ARC) offers training materials and workshops related to Unix shell and high-performance computing. Some are online, and some are in person and happen a few times during the year. Visit their Training website for an updated list of training resources and future workshops and events. But in particular, this online "Introducing the Unix Shell" course is one you shouldn't miss.

TAke the course

The Carpentries is an umbrella organization that encompasses both the Data Carpentry and the Software Carpentry organizations, both of which provide training in essential data and software skills to researchers and professionals. While not specific to UCL, they offer a wide range of workshops and lessons covering Unix shell, R and Python for data organization, data cleaning and data analysis, software development practices, version control with Git, and more. The "Introduction to the Command Line for Genomics" course from Data carpentry could be a great option too as it's tailored for genomics.

TAKE THE COURSE

Finally, if you want to do further training, another more intensive, 2 day course, from Software Carpentry taught in person at UCL through the ARC is their "Software Carpentry" worshop. Visit their Training website for information on future dates.

After this section you should be able to

Efficiently navigating the filesystem (cd, ls, pwd...) and understanding relative and absolute paths.
Ability to create, move, copy, rename, and delete files and directories (mkdir, cp, mv, rm...)
Using text editors like vim or nano
Connecting to remote servers using SSH (ssh) and securely transferring files (scp, sftp, rsync..)
Mastering the use of pipes (|) to pass the output of one command as the input to another
Proficiency in text manipulation and processing using tools like grep, sed, awk, and cut to extract, transform, and format data.
Customizing your shell environment with configuration files like .bashrc or .zshrc to set aliases, customize prompts, and define functions

The Unix shell

TO DO

After this section you should be able to