Lab 8: Command-Line Tools

Introduction

In this lab, we’ll learn how to use basic command-line tools. If there’s time, we’ll also learn a bit about a source control system called git.

Note that the Documentation and Handouts page of the course website has a command-line tools introduction PDF that covers some of the same material. For today, we’ll be learning about the basic commands:

cd         ls
mv         cp
rm         rmdir
mkdir      man
ssh        scp

We’ll also learn about how to redirect output and some useful commands like:

less
wc
grep
xargs

Finally, we’ll be learning the basics of text editing with vim, and how to compile and run Java code using javac and java.

Setup

For this lab, we’ll be doing all of our work on little.cs.pomona.edu, which is one of the department’s common servers. Because the server isn’t hooked up to a monitor, we’ll be forced to use the command line to do everything. Start by connecting to little.cs.pomona.edu using the command:

ssh yourusername@little.cs.pomona.edu

If you are connecting from one of the lab machines, you can leave off the “yourusername” part, as the local username will be the same as the remote one. You will have to type in your password, and then you’ll have a terminal that’s running on the remote machine. Because your home directory (/home/yourusername, which can be abbreviated as ~) is actually mounted remotely from a common file server, the files in your home directory on little.cs.pomona.edu are the same as those on the lab machines. You can see this if you run the command:

touch ~/Desktop/hello.txt

Which will create a file called hello.txt on your desktop. The file should show up on the desktop of the lab computer you’re using within a few seconds. For this lab we’ll be working exclusively through the terminal though.

To start the lab, cd into your desktop (cd ~/Desktop) and then copy over the starter files using the cp command:

cp -r /common/cs/cs062/labs/lab08 lab08

Notice that like all shell commands, cp takes arguments separated by spaces. Some arguments start with - and are called “flags” or “switches”—they alter the behavior of the command in some way. Other arguments are just listed out in order, and they tell the program what to do or operate on. In the case of cp, the first argument(s) are the thing(s) to copy, while the last argument is the directory to put them in (or the new name to copy a single source file as).

For cp, the ‘-r’ flag forces it to copy entire directories ’r’ecursively, rather than just copying individual files. But what if you wanted to figure that out? The first and most useful program we’ll talk about today is called man, which stands for manual. man can be used to get information about most other programs, so you can type man cp to get the manual page for the cp program. This will show how the command should be used (in the “SYNOPSIS” section) and list all of the different possible arguments along with what they do). If you want to figure out how to do something with a command, try to use the manual to find what you’re looking for.

Notice that at the bottom of the manual display it says “press h for help or q to quit”. The program man looks up a manual page, but then uses another program, called less to display it. less is nice because it allows you to search for things: just type / followed by the text you want to search for. So to find out what ‘-r’ does, we can do man cp and then type /-r to see the part that describes the -r flag. You can view any file using less by just typing less <filename>.

Part 1: The Basics

Sorting Files

To help you get familiar with some basic shell commands, the starter files include some simple tasks. The first is to separate out some image and text files. Go ahead and cd into the images-and-text/ directory. Now is a good time to be reminded that you can use the TAB key to auto-complete the names of files when typing in a shell. So you can type cd im<TAB> instead of cd images-and-text. This both saves you keystrokes and ensures that you don’t accidentally misspell the filename. If you just hit TAB several times without typing anything, the shell should list out all of the possibilities. You can use this to navigate when using cd to go deep into a directory structure.

Now that we’re in the images-and-text/ directory, use ls to list the directory contents. Notice that there is a mix of .jpg and .txt files in this directory. Your first task is to sort these into an images/ directory and a text/ directory. Start by creating these directories using the mkdir command. Now you can use the mv command to move the .txt files into the text/ directory and the .jpg files into the images/ directory.

Moving each file individually is a bit of a chore. Instead, we can ask the shell to handle all files that match a pattern. The * character in a shell argument will match any number of characters, so we can type mv *.txt text/ to move all of the .txt files into the text directory. Now we can use a second wildcard command to move all the .jpg files into the images/ directory.

Counting Files

How many text files are there? We could count by hand, but instead we’ll use another shell trick to find out. There’s a handy program called wc which counts words (and/or lines or characters). Notice that if you run just wc nothing happens. In fact, it cuts off your terminal (try typing ls)! Luckily, there’s a universal shortcut for interrupting the current process: hit control-C and the rogue wc process will be killed, giving us back our normal prompt.

What happened? Well, if we look at man wc we can see that if no “FILE” is given, it will “read standard input.” This means that when we ran wc it was expecting us to type in its input. That’s all fine and good, but how do we tell it that we’re done? Well, the terminal has another shortcut for that: control-D will tell a program that you’re done giving it input (be careful: if you hit control-D in the outer shell, it’ll quit the ssh session, and you’ll have to log in again). So we can type wc and then a<enter>b<enter>c<enter><control-D> and it should print out 3 3 6 which is the line-count, word-count, and character-count of the input, respectively. Notice from man wc that we can give it the -l flag to count just lines.

But how can we get wc to count files? It can read what we type or something from a file, but we want it to read the output from the ls command. In the shell, there are two ways to do this. First, we can redirect output into a file, using the > operator. So we could say ls > list and then wc -l list to count the number of files in the current directory. However, this would leave around an extra list file every time we did it. Instead, we can redirect output directly from one program into another using the | operator. So to count how many text files there are, assuming we’ve moved them all into the text/ directory, we can do:

ls text | wc -l

This should print out “10.” Note in this case that we didn’t give wc any arguments (besides the flag), so it works in “read from standard input mode.” However, because of the | redirection (also called a “pipe”) the standard input came from the output of the ls command, instead of from us typing. This pattern of redirection is quite useful, and can be repeated multiple times, with several programs feeding their output into each other.

Searching for Patterns

Now that we’ve sorted our text and image files, let’s move over to the programs directory. To do this, use cd ../programs (assuming you’re still in the images-and-text/ directory). In this directory are a few random Java programs that were downloaded from Pastebin. However, the directory is messy: it also has some .class files in it from compiling the .java files.

We want to find out which files contain drawing code. To do this, we’ll use a program called grep, which searches for patterns in its input. To search for the pattern “draw” we can use:

grep draw *.java

Recall that *.java matches all files in the current directory that end in .java. The entry for man grep tells us that it expects a “PATTERN” followed by one or more “FILEs,” so this command will search all of the matching files for the pattern “draw.” By default, it prints out each line that matches. But in this case, we only want to know which files contain that pattern, not how many times or what their content is. Luckily, grep has a -l flag which does exactly that: ’l’ists files that match. So we’ll use:

grep -l draw *.java

Notice that grep doesn’t care about the order of the -l flag: you can put it first, second or third, and the other arguments will be interpreted the same way.

Filtering Files

Now that we know which files contain some kind of drawing code, let’s move them all to a new directory. Create a new directory called draws/ using mkdir. Now all we need to do is issue an mv command targeting all of our grep -l files and putting them into this directory. Unfortunately, our grep -l command produces output (which we could move around with a ‘|’ or put in a file with ‘>’) but what we want those things to be are arguments to the mv command. We could redirect the output to a file using ‘>’ and then edit that file to add “mv” before them and “draws/” after them and then run that file as a script, but there’s a better way: a program called xargs exists just for this purpose.

As man xargs says, xargs is designed to take input and turn it into arguments. Normally, these arguments are passed to the target program after any default arguments, but in our case, we want the variable arguments to come before the destination argument of mv, so we need to use xargs‘’-i’ flag to tell it where to put the variable arguments. So we’ll construct a command line that does the following:

1. Use grep -l draw *.java to generate output.
2. Pipe that output into xargs -iZZ
3. Give xargs additional arguments to tell it to execute the mv command with our variable arguments followed by ‘draws/’. Use the ‘ZZ’ pattern that we gave to the ‘-i’ command to tell xargs where to put the additional arguments.

If you execute the command properly, you should be able to move all of the .java files that contain the text “draw” into the draws/ folder. Note that there’s another way to do this: in the shell, the ‘`’ character (a.k.a. the “backtick”) can be used to do command substitution, where the output of a command is used as part of another command. With this method, we write our mv command, and just use `grep -l draw *.java` where we want the extra arguments to appear.

Part 2: Compiling Java Code

javac

Since we’ve got these Java programs sitting here, let’s see if we can get any to run. First, let’s create a new directory has_main/ and put all of the java files which declare a “public void main” method somewhere in them into that directory. We can use the same technique as we did with “draw”, with one caveat: we want to search for the pattern “public void main”, but that pattern has spaces in it. Spaces are also used to separate arguments, but only one argument can be the pattern. If we just say:

grep public static void main *.java

the first three lines will look like:

grep: static: No such file or directory
grep: void: No such file or directory
grep: main: No such file or directory

To tell grep that we really want one big argument including spaces, we’ll use double quotes, like this:

grep "public static void main" *.java

Use this command with the pattern above to isolate all of the .java files which have main methods into a has_main/ directory.

Now let’s try to compile them. To compile Java files, you use the program javac which produces .class files as output. Let’s start by trying to compile all of the .java files:

javac *.java

Unfortunately, one of our .java files has an error in it (well, they were randomly downloaded from Pastebin, so what can you expect). Let’s skip that file for now by just changing its name (using mv to have an extra ‘a’ in the .java part, so that our *.java won’t pick it up). Now the command above should work¹.

java

To run a Java program, we use the program java. However, java is a bit quirky: we need to tell it the name of a class to run, not just the name of a .class file. Luckily in Java, the name of a .class file should always be the name of the class it defines. Unluckily, Java requires that classes which are in packages be in a folder with the name of that package. To see which classes need packages, use grep to print out lines containing the pattern “package” from .java files in the current directory.

Now pick a non-packaged file and run the class, using java <classname>. Depending on which you ran, it should either print a few numbers or hang. If it hangs, use <control-C> to kill it. Try running a few different classes.

The Classpath

When compiling or running Java code, you often need to tell it where to find external code that your code depends on. For example, if your code depends on the structure5 library, you need to tell it where to find that code (at least the .class files). Both javac and java require this information whenever you use import to import something that isn’t part of the Java standard library. You can give Java a list of places to look by using the --classpath flag (short version: -cp) followed by a colon-separated list of directories and/or .jar files to search. So if we want to import structure5 along with other .java files in the src/ directory, we’d give it:

-cp /common/cs/cs062/bailey.jar:src/

If we also wanted to use JUnit, we’d get:

-cp /common/cs/cs062/bailey.jar:/common/cs/cs062/junit.jar:src/

As a final convenience, you can separate the .class files that javac produces from their .java files by giving the ‘-d’ flag followed by a directory to put output in. Using -d bin as an argument along with -cp, we can exactly duplicate what Eclipse does when we tell it to run our project. First, we’ll run javac with the ‘-cp’ and ‘-d’ arguments as above, and then we’ll run java with just the ‘-cp’ argument. If all goes well, you should be able to run your assignment 3 this way (it’s the only assignment so far that doesn’t require some kind of graphics, which are unavailable while ssh’d onto another machine).

Part 3: Editing Text

Now that you can move files around and even compile and run Java code from the terminal, what if you want to edit a file? As hinted at above, you can use less to view the contents of a file and even search within it, but it doesn’t allow you to make changes.

To edit files on the command line, there’s a simple editor called nano that lets you type text and do basic operations. It even displays available commands at the bottom of the screen. However, for this lab, you’ll be learning a tiny bit of how to use vim, a more powerful and efficient editor.

vim basics

To open a file with vim, type vim <filename>. In this case, cd into the edit directory from the starter materials and type vim ed<TAB>. Now that you’re in vim, the most important thing to remember is how to quit: type ‘:’ followed by ‘q’ and then the enter/return key to quit. This works because in vim, the ‘:’ key initiates a command, and ‘q’ is short for the ‘quit’ command.

vim is a modal editor, which means that most keys on the keyboard execute commands instead of just inserting characters. To insert characters, you use the ‘i’ key to get into insert mode where typing produces text in the file you’re editing. When you’re done typing, use the escape key to get back to normal mode. Although this is confusing at first, if you get used to it it’s considerably more efficient when you enter lots of commands (as opposed to just typing text). In particular, vim has a powerful set of commands for moving the cursor, which we can use to our advantage.

In the edit-me.txt file, you’ll see a line that says “Delete me.” Let’s move to this line by using the ‘/’ command, which starts a search. Type “/elete<enter>” and the cursor should move to the first ‘e’ of the word “Delete.”² Now you can type ‘dd’ to delete the current line. At any point, if you want to undo a command, just type ‘u’ (you can also use control-R to redo something you undid).

Next, move down to the line that says “Add a line after this”, either with a ‘/’ command, or using the arrow keys or the h/j/k/l keys, which move the cursor around in vim. Use the ‘o’ command to start inserting on a new line after the current line, and type something in. Then hit escape to get back to normal mode.

Next, go to the line that asks you to add to its end. Here, use the ‘A’ command (shift-a) to append to the end of the line (while ‘i’ and ‘a’ ‘i’nsert and ’a’ppend respectively, ’I’ and ‘A’ do so at the beginning or end of the current line instead of at the cursor). Again, type something and hit escape.

Finally, let’s replace all of those targets with some other text. To do this, start by using a ‘/’ command to find the first target. Then notice that you can use the ‘n’ and ‘N’ commands to move forwards and backwards among matches to the last pattern. Now, we’ll use the ‘c’ command (for ‘c’orrect). The ’c’ command is special, because it wants to be told what to correct. After ‘c’, you can give vim a “movement command” to correct text from the cursor to the end of the movement. So “cl” would correct just the character under the cursor (‘l’ moves the cursor right one character), while “cj” would correct two entire lines (‘j’ moves down a line). We’ll use a special movement command “iw” that means “the current word.” So our whole command will be “ciw”, after which we’ll type some replacement text and then hit escape.

Conveniently, this “ciw” command, up to the point we hit escape, counts as a single command. And vim happens to have a shortcut for “repeat last non-movement command:” the “.” command. So to replace all of the targets quickly, we can now just hit “n” followed by “.” several times. This “.” command makes vim’s other commands much more useful, because it’s easy to repeat them (vim also has a full-fledged macro-creation system).

Editing a Complex File

Now that we’ve got the basics down, let’s try a more complicated task. Type “:tabe ge<TAB>” to edit the “geology.c” file in a new tab (:tabe is short for :tabedit; notice you can tab-complete filenames; to move between tabs, you can use “gt” and “gT” to go forwards/backwards). This is a complicated C file that’s part of a larger program, and we’ll be using it to try out some vim commands.

Imagine that we’ve just learned that the construct_efd_int_node function needs to take a global variable called WORLD as its second argument instead of NULL. We want to replace NULL throughout the file with WORLD, but only when it’s an argument to the construct_efd_int_node function. First, however, we’re going to get a bird’s-eye view of our targets.

To see all of the instances of construct_efd_int_node in this file at once, we’ll use grep to filter the file. We could go back to the command line, run grep, and then re-open the file, but there’s a simpler way. First, type “0gg” to warp to the 0th line of the file. Then type “V” to enter “visual-line” mode, where you can select lines to operate on. Next, type “G” to warp to the last line of the file. Now that you’ve selected the whole file, type “!grep construct_efd_int<enter>” to filter those lines through grep. This gives us a picture of the lines we want to edit. Once we’re done looking at it, we can use “u” to undo our filter and get back to the original file.

To find those lines again, use a “/” command to look for “constru.” The first hit will be a construct_efd_obj_node call, which isn’t what we want, so hit “n” to get to a construct_efd_int_node call. Now that we’re on the exact word we want to find, hit “*" to search for the word under the cursor. You can hit “n” and/or “N” a few times to convince yourself we’re finding the right stuff.

Now that we can skip through the file to the right lines, we need to be able to jump from the ‘c’ of “construct_efd_int_node” to the ‘N’ of “NULL.” Luckily, vim has a command ‘f’ that skips ‘f’orward on the current line until the next one of a certain character. Type “fN” to skip to the next capital N on the current line, which will be the ’N’ of “NULL.” This skip command can be repeated in the forward direction with ‘;’ and in the backward direction with ‘,’. Since we’re now on the word we want to replace, use “ciw” again to replace the “NULL” with “WORLD.”

With everything set up, we can go through the file and replace our NULL’s by repeating the sequence “n;.” which first repeats the last earch, then repeats the last skip, and then repeats the last edit command. This is kind of like a customized find/replace, but we’re just doing it using basic vim commands. Note that just using find/replace on NULL won’t work, because we only want to change NULLs that are arguments to construct_efd_int_node.

Saving Your Work

One other important command in vim is the “:w” command, which ’w’rites out (saves) the current file. Once you’re done editing geology.c, you can save it and quit it: “:wq” in an abbreviation that does both save and quit at once. You can also save/quit the edit-me.txt file now.

What to Submit

First, copy your edited geology.c file into the submit/ folder. Then fill out the lab08.json and answers.txt files with your username(s) and the answers to the questions in that file respectively. Make sure you put your answers exactly the example answers are. It may be useful to know that the “D” command deletes text from the cursor position to the end of the current line.

When you’re done, rename the submit directory using mv to be called “Lab08_YourName_PartnersName” and submit directly on litle.cs.pomona.edu using the submit script:

/common/cs/cs062/bin/submit cs062 lab08 Lab08_Names