Working remotely

Before we start

I’ve created a small repo containing a simple file structure we will be navigating around today. Before we start you should obtain a local copy of these files by either:

• git clone https://github.com/Sta523-Fa17/filesystem.git

• File > New Project… > Version Control > Git > https://github.com/Sta523-Fa17/filesystem.git

Why the shell

• Programmatically interact with OS (file system)
• Origins in the 1960s and 1970s
• “designed by computer scientists for computer scientists”
• Evolution - interactive command language to scripting programming language
• “quick and dirty” prototyping

Unix Design Philosophy

“Even though the UNIX system introduces a number of innovative programs and techniques, no single program or idea makes it work well. Instead, what makes it effective is the approach to programming, a philosophy of using the computer. Although that philosophy can’t be written down in a single sentence, at its heart is the idea that the power of a system comes more from the relationships among programs than from the programs themselves. Many UNIX programs do quite trivial things in isolation, but, combined with other programs, become general and useful tools.”

The UNIX Programming Environment, Brian Kernighan and Rob Pike

Why should you care?

• Power

• Reproducibility

• Scripting is reproducible - clicking is not.

• Analysis pipelines

Where am I? (pwd, ls, cd)

cr173@saxon [~]$ pwd
/home/vis/cr173

cr173@saxon [~]$ ls -l
total 5
drwx------+  2 cr173 visitor 11 Aug 25 19:53 mail
drwx------+  2 cr173 visitor  2 Dec 12 2011  Mail
drwxr-xr-x+  2 cr173 visitor  2 Aug 25 20:37 Sta523

cr173@saxon [~]$ cd Sta523
cr173@saxon [Sta523]$ pwd
/home/vis/cr173/Sta523

Absolute vs relative paths

In unix paths can either be absolute or relative, and the difference is very important. For portability reasons you should almost never use absolute paths.

Absolute path examples:

/var/ftp/pub
/etc/samba.smb.conf
/boot/grub/grub.conf

Relative path examples:

Sta523/filesystem/
data/access.log
filesystem/nelle/pizza.cfg

Creating or Deleting directories

cr173@saxon [~]$ cd Sta523
cr173@saxon [Sta523]$ ls -l
total 2
drwxr-xr-x+ 2 cr173 visitor 2 Aug 25 20:41 filesystem

cr173@saxon [Sta523]$ mkdir test
cr173@saxon [Sta523]$ ls -l
total 2
drwxr-xr-x+ 2 cr173 visitor 2 Aug 25 20:41 filesystem
drwxr-xr-x+ 2 cr173 visitor 2 Aug 25 20:44 test

cr173@saxon [Sta523]$ rmdir test/
cr173@saxon [Sta523]$ ls -l
total 2
drwxr-xr-x+ 2 cr173 visitor 2 Aug 25 20:41 filesystem

Home directory and ~

~ is a special character that expands to the name of your home directory. If you append a user’s login to ~, it then refers to that user’s home directory.

cr173@saxon [filesystem]$ cd ~
cr173@saxon [~]$ pwd
/home/vis/cr173

cr173@saxon [~]$ cd ~mc301
cr173@saxon [mc301]$ pwd
/home/fac/mc301

Exercise

Make sure that you have cloned the filesystem repository Connect to saxon and change to filesystem directory

cr173@saxon [~]$ cd ~/filesystem

Along with your neighbors, explore and map out all of the files and subdirectories that are contained within filesystem. It will probably be easiest to do this by drawing a tree on a piece of paper.

cr173@saxon [filesystem]$ mv haiku.txt better_haiku.txt
cr173@saxon [filesystem]$ ls -l
total 4
-rw-r--r--+ 1 cr173 visitor 88 Aug 25 21:07 better_haiku.txt
drwxr-xr-x+ 2 cr173 visitor  5 Aug 25 20:56 data
drwxr-xr-x+ 4 cr173 visitor  5 Aug 25 20:53 users

cr173@saxon [filesystem]$ mv better_haiku.txt haiku.txt

Wildcards and the shell

• * - matches any number of characters in a filename, including none.
• ? - matches any single character.
• [ ] - set of characters that may match a single character at that position.
• - - used within [ ] denotes a range of characters or numbers (eg. [0-9]).

More on this later when we discuss regular expressions

cr173@saxon [filesystem]$ ls data/
access.log  hardware.cfg  network.cfg

cr173@saxon [filesystem]$ ls data/*.cfg
data/hardware.cfg  data/network.cfg

Examining files (cat, more, head, tail, grep)

cr173@saxon [~]$ cat ~/Sta523/filesystem/haiku.txt 
All night with no sleep
Caffeine your only partner
Damn thing still won't work
-chimera

cr173@saxon [~]$ head -n 2 ~/Sta523/filesystem/haiku.txt 
All night with no sleep
Caffeine your only partner

cr173@saxon [~]$ tail -n 3 ~/Sta523/filesystem/haiku.txt 
Caffeine your only partner
Damn thing still won't work
-chimera

Pipes

cr173@saxon [~]$ cat ~/Sta523/filesystem/haiku.txt | grep ^[AC]
All night with no sleep
Caffeine your only partner

Oroborus cleverness

Want to see the 2nd line of the file and nothing else?

cr173@saxon [~]$ head -n 2 ~/Sta523/filesystem/haiku.txt | tail -n 1
Caffeine your only partner

What about the penultimate (2nd to last) line?

cr173@saxon [~]$ tail -n 2 ~/Sta523/filesystem/haiku.txt | head -n 1
Damn thing still won't work

Putting it together

cr173@saxon [filesystem]$ wc haiku.txt
 4 15 88 haiku.txt

cr173@saxon [filesystem]$ grep A haiku.txt
All night with no sleep
cr173@saxon [filesystem]$ grep A haiku.txt | wc
      1       5      24

Exercise

There is a text copy of Shakespeare’s Hamlet in the data directory of the file system repo, using the tools we just discussed answer the following questions (in order of increasing difficulty)

• How many lines in the play use the word Hamlet?

• How many times is the word Hamlet used? (Hint: a single line may have more than one usage)

• How many times is the word the used? (Hint: be careful with capitalization)

Acknowledgments

Above materials are derived in part from the following sources:

• Mark Hansen - UCLA’s Stat 202a
• Karl Broman - Wisc’s Tools4RR
• Software Carpentry Project