sh, bash, Perl, Python, Ruby etc are scripting languages in which the "program" is a regular file containing plain text that is interpreted into MACHINE CODE at the time you run it. Other languages like c++, java, haskell, rust have a separate COMPILATION step to turn their regular text source file into a BINARY EXECUTABLE.
A script from sh, bash, Perl, Python, Ruby etc become EXECUTABLE simply by turning on the executable permission.
#!/bin/bash Allows the script interpreter to recognize that the file should be executed with bash #!/usr/bin/env bash Same, but now the script is portable. I.e., different systems will recognize it as a bash script The env program will find the bash that is found in your environment #!/usr/bin/env python3 Use the python3 that is found by the environment
Everything in UNIX is case sensitive. So $NAME ne $name ne $Name Probably a good idea to use lower case variable names in scripts, so not to accidentally overwrite environment variables
Analogous to 'use warnings' in perl? Write 'set -u' on top of your shell script
program 2> file.e Redirect STDERR to file.e program 1> file.o Redirect STDOUT to file.o program 1> file.o 2> file.e Redirect STDOUT to file.o and STDERR to file.e program 2>/dev/null Redirect output to /dev/null , where data is gone forever
file Truncate a file to size 0 if it exists, create an empty file if it doesnt exist
IF LOOP SYNTAX if []; then ... else ... fi
NESTED FOR LOOPS for i in {}; do for j in {}; do cmd $i $j; done; done;
FOR LOOPS
for i in {,,}; do cmd1 $i; cmd2; done
sometimes
for i in *.txt; do cmd; done Loops over all files that match the *.txt pattern.
WHILE READ LOOP Read a file line by line and assign variables to the line's content.
Example: ls -la | while read PERM USR USRGRP SIZE; do echo $PERM $USR $USRGRP $SIZE; done
while read -r LINE; do echo "$LINE" done < file.txt
cat file.txt | while read LINE; do echo $LINE done
cat file.txt | while IFS=$'\t' read -r WORD; do echo $WORD done
Use the read '-r' flag, 'while read -r STRING' to prevent read from eating '' in $STRING
LOOP CONTROL break Breaks and leaves the loop continue Skip to the next iteration of the loop (definitely analogous to next of Perl)
LOOP OVER RANGE OF NUMBERS for i in $(seq 89 2 97); do echo $i done
loops from 89 to 97 with steps of 2
Will return 89 91 93 95 97
COMPARE TWO STRINGS if [ "$S" -ge "$Q" ] ...
SHELL BOOLEANS FOR INTEGERS -eq Equal -ne Not Equal -gt Greater Than -lt Lesser Than -ge Greater or Equal then -le Lesser or Equal then
STRING COMPARISON OPERATORS = Match != Does not match
Example: Checks if the string "BACL10" is part of $alphabin
if [[ "$alphabin" != "BACL10" ]]; then echo "blastp $alphabin ..." fi
OTHER COMPARISONS [[ -e FILE ]] True if FILE exists [[ -f FILE ]] True if FILE exists and is a regular file [[ -h FILE ]] True if FILE exists and is a symbolic link [[ -s FILE ]] True if FILE is non-empty [[ -d DIR ]] True if DIR exists [[ -z $STRING ]] True if $STRING is null (doesn't exist) [[ -L SYMLINK ]] True if SYMLINK exists
cmd1 && cmd2 && is the AND operator. execute cmd1 and only if it succeeds execute cmd2 cmd1 || cmd2 || is the OR operator. execute cmd2 only if cmd1 fails Note that these operators already have an 'if' functionality built in!
&& is used to chain commands together, such that the command after the && is only run when the command prior to the && is run without any errors && is also used to mean AND in comparisons that return True/False
Example [[ ! -d "$OUT_DIR" ]] && mkdir -p "$OUT_DIR" if $OUT_DIR does not exist is TRUE, create $OUT_DIR
Alternatively
if ls *.orig.tsv 1> /dev/null 2>&1; then ... ; fi
if ! grep -q $pattern file.list; then ... ; fi
POSITIONAL ARGUMENTS FOR BASH SCRIPT
$0 Name of the bash script
$1 First argument
$2 Second argument
If multiple words on the command line are flanked by quotes, whatever is within the quotes is one positional argument ./script.sh ONE TWO "THREE PIGS" $1 -> ONE $2 -> TWO $3 -> THREE PIGS
CASE Matches a variable against a range of options described in the case loop
Example: var=apple case $var in apple) echo "variable was apple";; pear) echo "variable was pear";; mango) echo "variable was mango";; esac Will print "variable was apple"
GETOPTS Stating options in a bash script. Only works with one letter flags.
while getopts ":s:m:z:n:t:g:" opt; do
case
The : after the single letter in the getopts line, states that it expects an argument after the flag. I.e. -s bladiebla.aln The first : in that line prevents 'verbose error handling' :) gets activated when an option/flag that needs an argument is called without an argument ?) gets activated when an option/flag that is not specified in the code is called.
USING DELIMITER OTHER THAN WHITESPACE: grep -c ">" foo.fasta | while IFS=':' do echo $FILE $SEQCOUNT; done
i=teststring
echo
i=teststringstring
echo
OUTFMT="6 std stitle" If you pass $OUTFMT to some command line tool, it will evaluate to 6 std stitle as separate arguments If you pass "$OUTFMT" instead, it will evalute '6 std stitle' as a single argument
string=${var:-default} $string gets value $var if $var is set previously. Otherwise,
sum=$((1+1)) $((1+1)) returns to 2, prior to assigning sum the value 2
let sum=1+1 $sum is assigned the value 2 through simple arithmetic evaluation
declare -A arr Declare a new array arr arr=() Empty/reset arr arr=( ["moo"]="cow" ["woof"]="dog") Create a dictionary/hash. Supported since bash v4. Make sure to use #!/bin/bash and not #!/bin/sh while read -r one two; do arr[$one]="$two" done < input_file Fill up an array/dictionary/hash by reading in a file. It seems critical to have the file read in like this: done < input_file echo "${!arr[@]}" Return the KEYS of the dictionary echo "${arr[@]}" Return the VALUES of the dictionary
join <(sort FILE1) <(sort FILE2) > FILE3 <(sort FILE) obsoletes an intermediate file, it creates like a temporary file within the command. Otherwise you would have done: sort FILE1 > FILE1-sort sort FILE2 > FILE2-sort join FILE1-sort FILE2-sort > FILE3 rm FILE1-sort rm FILE2-sort So it saves you a lot of work! Amazing!!
cmd > file.txt Write STDOUT to file.txt cmd >> file.txt Append file.txt with STDOUT cmd &> file.txt Write STDOUT and STDERR to file.txt
Every command in Unix returns an exit status. A successfull exit status returns 0, an un-succesfull one returns a non-zero value. The last command executed determines the exit status. Within a script, when exiting, you can pass on a custom exit status through exit , where nnn is an integer between 0 and 255
The special variable