Bash-Oneliner

I am glad that you are here! I was working on bioinformatics a few years ago and was amazed by those single-word bash commands which are much faster than my dull scripts, time saved through learning command-line shortcuts and scripting. Recent years I am working on cloud computing and I keep recording those useful commands here. Not all of them is oneliner, but i put effort on making them brief and swift. I am mainly using Ubuntu, Amazon Linux, RedHat, Linux Mint, Mac and CentOS, sorry if the commands don't work on your system.

This blog will focus on simple bash commands for parsing data and Linux system maintenance that i acquired from work and LPIC exam. I apologize that there are no detailed citation for all the commands, but they are probably from dear search engine and Stack Overflow.

English and bash are not my first language, please correct me anytime, thank you. If you know other cool commands, please teach me!

Here's a more stylish version of Bash-Oneliner~

Handy Bash one-liners

• Terminal Tricks
• Variable
• Math
• Grep
• Sed
• Awk
• Xargs
• Find
• Condition and Loop
• Time
• Download
• Random
• Xwindow
• System
• Hardware
• Networking
• Data Wrangling
• Others

Terminal Tricks

Using Ctrl keys

Ctrl + a : move to the beginning of line.
Ctrl + d : if you've type something, Ctrl + d deletes the character under the cursor, else, it escapes the current shell.
Ctrl + e : move to the end of line.
Ctrl + k : delete all text from the cursor to the end of line.
Ctrl + l : equivalent to clear.
Ctrl + n : same as Down arrow.
Ctrl + p : same as Up arrow.
Ctrl + q : to resume output to terminal after Ctrl + s.
Ctrl + r : begins a backward search through command history.(keep pressing Ctrl + r to move backward)
Ctrl + s : to stop output to terminal.
Ctrl + t : transpose the character before the cursor with the one under the cursor, press Esc + t to transposes the two words before the cursor.
Ctrl + u : cut the line before the cursor; then Ctrl + y paste it
Ctrl + w : cut the word before the cursor; then Ctrl + y paste it
Ctrl + x + backspace : delete all text from the beginning of line to the cursor.
Ctrl + x + Ctrl + e : launch editor defined by $EDITOR to input your command. Useful for multi-line commands.
Ctrl + z : stop current running process and keep it in background. You can use `fg` to continue the process in the foreground, or `bg` to continue the process in the background.
Ctrl + _ : undo typing.

Change case

Esc + u
# converts text from cursor to the end of the word to uppercase.
Esc + l
# converts text from cursor to the end of the word to lowercase.
Esc + c
# converts letter under the cursor to uppercase, rest of the word to lowercase.

Run history number (e.g. 53)

!53

Run last command

!!
# run the previous command using sudo
sudo !!

Run last command and change some parameter using caret substitution (e.g. last command: echo 'aaa' -> rerun as: echo 'bbb')

# last command: echo 'aaa'
^aaa^bbb

#echo 'bbb'
#bbb

# Notice that only the first aaa will be replaced, if you want to replace all 'aaa', use ':&' to repeat it:
^aaa^bbb^:&
# or
!!:gs/aaa/bbb/

Run past command that began with (e.g. cat filename)

!cat
# or
!c
# run cat filename again

Bash globbing

# '*' serves as a "wild card" for filename expansion.
/etc/pa*wd    #/etc/passwd

# '?' serves as a single-character "wild card" for filename expansion.
/b?n/?at      #/bin/cat

# '[]' serves to match the character from a range.
ls -l [a-z]*   #list all files with alphabet in its filename.

# '{}' can be used to match filenames with more than one patterns
ls *.{sh,py}   #list all .sh and .py files

Some handy environment variables

$0   :name of shell or shell script.
$1, $2, $3, ... :positional parameters.
$#   :number of positional parameters.
$?   :most recent foreground pipeline exit status.
$-   :current options set for the shell.
$$   :pid of the current shell (not subshell).
$!   :is the PID of the most recent background command.
$_   :last argument of the previously executed command, or the path of the bash script.

$DESKTOP_SESSION     current display manager
$EDITOR   preferred text editor.
$LANG   current language.
$PATH   list of directories to search for executable files (i.e. ready-to-run programs)
$PWD    current directory
$SHELL  current shell
$USER   current username
$HOSTNAME   current hostname

Using vi-mode in your shell

set -o vi
# change bash shell to vi mode
# then hit the Esc key to change to vi edit mode (when `set -o vi` is set)
k
# in vi edit mode - previous command
j
# in vi edit mode - next command
0
# in vi edit mode - beginning of the command
R
# in vi edit mode - replace current characters of command
2w
# in vi edit mode - next to 2nd word
b
# in vi edit mode - previous word
i
# in vi edit mode - go to insert mode
v
# in vi edit mode - edit current command in vi
man 3 readline
# man page for complete readline mapping

Variable

[back to top]

Variable substitution within quotes

# foo=bar
echo $foo
# bar
echo "$foo"
# bar
# single quotes cause variables to not be expanded
echo '$foo'
# $foo
# single quotes within double quotes will not cancel expansion and will be part of the output
echo "'$foo'"
# 'bar'
# doubled single quotes act as if there are no quotes at all
echo ''$foo''
# bar

Get the length of variable

var="some string"
echo ${#var}
# 11

Get the first character of the variable

var=string
echo "${var:0:1}"
#s

# or
echo ${var%%"${var#?}"}

Remove the first or last string from variable

var="some string"
echo ${var:2}
#me string

Replacement (e.g. remove the first leading 0 )

var="0050"
echo ${var[@]#0}
#050

Replacement (e.g. replace 'a' with ',')

{var/a/,}

Replace all (e.g. replace all 'a' with ',')

{var//a/,}

Substitute environment variables in a file

export NAME="Alice"
envsubst < template.txt > output.txt
# template.txt:       Hello ${NAME}, your order is shipped.
# output.txt:         Hello Alice, your order is shipped.
# or
export USER="Bob" && echo 'Hi $USER, welcome!' | envsubst
# Hi Bob, welcome!

Grep lines with strings from a file (e.g. lines with 'stringA or 'stringB' or 'stringC')

# with grep
test="stringA stringB stringC"
grep ${test// /\\\|} file.txt
# turning the space into 'or' (\|) in grep

To change the case of the string stored in the variable to lowercase (Parameter Expansion)

var=HelloWorld
echo ${var,,}
helloworld

Expand and then execute variable/argument

cmd="bar=foo"
eval "$cmd"
echo "$bar" # foo

Record a terminal session

# https://github.com/asciinema/asciinema
asciinema rec demo.cast

Math

[back to top]

Arithmetic Expansion in Bash (Operators: +, -, *, /, %, etc)

echo $(( 10 + 5 ))  #15
x=1
echo $(( x++ )) #1 , notice that it is still 1, since it's post-increment
echo $(( x++ )) #2
echo $(( ++x )) #4 , notice that it is not 3 since it's pre-increment
echo $(( x-- )) #4
echo $(( x-- )) #3
echo $(( --x )) #1
x=2
y=3
echo $(( x ** y )) #8

Print out the prime factors of a number (e.g. 50)

factor 50
# 50: 2 5 5

Sum up input list (e.g. seq 10)

seq 10|paste -sd+|bc

Sum up a file (each line in file contains only one number)

awk '{s+=$1} END {print s}' filename

Column subtraction

cat file| awk -F '\t' 'BEGIN {SUM=0}{SUM+=$3-$2}END{print SUM}'

Simple math with expr

expr 10+20 #30
expr 10\*20 #600
expr 30 \> 20 #1 (true)

More math with bc

# Number of decimal digit/ significant figure
echo "scale=2;2/3" | bc

# .66

# Exponent operator
echo "10^2" | bc
# 100

# Using variables
echo "var=5;--var"| bc
# 4

Grep

[back to top]

Type of grep

grep = grep -G # Basic Regular Expression (BRE)
fgrep = grep -F # fixed text, ignoring meta-characters
egrep = grep -E # Extended Regular Expression (ERE)
rgrep = grep -r # recursive
grep -P # Perl Compatible Regular Expressions (PCRE)

Grep and count number of empty lines

grep -c "^$"

Grep and return only integer

grep -o '[0-9]*'
# or
grep -oP '\d*'

Grep integer with certain number of digits (e.g. 3)

grep '[0-9]\{3\}'

# or
grep -E '[0-9]{3}'

# or
grep -P '\d{3}'

Grep only IP address

grep -Eo '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}'

# or
grep -Po '\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}'

Grep whole word (e.g. 'target')

grep -w 'target'

# or using RE
grep '\btarget\b'

Grep returning lines before and after match (e.g. 'bbo')

# return also 3 lines after match
grep -A 3 'bbo'

# return also 3 lines before match
grep -B 3 'bbo'

# return also 3 lines before and after match
grep -C 3 'bbo'

Grep string starting with (e.g. 'S')

grep -o 'S.*'

Extract text between words (e.g. w1,w2)

grep -o -P '(?<=w1).*(?=w2)'

Grep lines without word (e.g. 'bbo')

grep -v bbo filename

Grep lines not begin with string (e.g. #)

grep -v '^#' file.txt

Grep variables with space within it (e.g. myvar="some strings")

grep "$myvar" filename
# remember to quote the variable!

Grep only one/first match (e.g. 'bbo')

grep -m 1 bbo filename

Grep and return number of matching line(e.g. 'bbo')

grep -c bbo filename

Count occurrence (e.g. three times a line count three times)

grep -o bbo filename |wc -l

Case insensitive grep (e.g. 'bbo'/'BBO'/'Bbo')

grep -i "bbo" filename

COLOR the match (e.g. 'bbo')!

grep --color bbo filename

Grep search all files in a directory(e.g. 'bbo')

grep -R bbo /path/to/directory
# or
grep -r bbo /path/to/directory

Search all files in directory, do not ouput the filenames (e.g. 'bbo')

grep -rh bbo /path/to/directory

Search all files in directory, output ONLY the filenames with matches(e.g. 'bbo')

grep -rl bbo /path/to/directory

Grep OR (e.g. A or B or C or D)

grep 'A\|B\|C\|D'

Grep AND (e.g. A and B)

grep 'A.*B'

Regex any single character (e.g. ACB or AEB)

grep 'A.B'

Regex with or without a certain character (e.g. color or colour)

grep 'colou\?r'

Grep all content of a fileA from fileB

grep -f fileA fileB

Grep a tab

grep $'\t'

Grep variable from variable

$echo "$long_str"|grep -q "$short_str"
if [ $? -eq 0 ]; then echo 'found'; fi
# grep -q will output 0 if match found
# remember to add space between []!

Grep strings between a bracket()

grep -oP '\(\K[^\)]+'

Grep number of characters with known strings in between(e.g. AAEL000001-RA)

grep -o -w "\w\{10\}\-R\w\{1\}"
# \w word character [0-9a-zA-Z_] \W not word character

Skip directory (e.g. 'bbo')

grep -d skip 'bbo' /path/to/files/*

Sed

[back to top]

Remove the 1st line

sed 1d filename

Remove the first 100 lines (remove line 1-100)

sed 1,100d filename

Remove lines with string (e.g. 'bbo')

sed "/bbo/d" filename
# case insensitive:
sed "/bbo/Id" filename

Remove lines who