My Personal Spelboak 'n' Incantations

Breathe deep and think before you type! No disaster is so bad you can't make it a little worse :)

1. get a typescript running to record your terminal session

export PS1=$PS1'\d \t # '; script

• Ctrl-D to end the typescript

• give the outfile a descriptive name

2. check things

dmesg | tail - What's in syslog?
uptime - How long has the server been running? What does the cpu load look like?
df -h - Running low on disk space? Network mounts hanging?
df -i - Running out of inodes?
vmstat 1 5

• Is 'r' (waiting processes) greater than lscpu | grep "^CPU(s):"? Processor saturation.
• Is swap in, out (si,so) greater than zero? Out of memory.
• Is the CPU idle waiting for I/O (wa)? Disk bottleneck.

• Is the disk I/O time (await) high?
• What about percentage utilization (%util)

3. get help

apropos <topic>

• searches manual page names and descriptions
• useful for when you've forgotten the name of a command or want to see what is available

• read the manual page; some "modern" tool authors don't believe in manuals so also try:

4. other stuff to try

tail -f /var/log/... - What do the logs say?
lsof -c <process name> - Where even are the logs?
ps aux | grep <pname> - What args were used to start it?
namei -l `pwd` - Are the file permissions right?
swapoff -a; swapon -a - Delete everything in swap (DESTRUCTIVE)
: > /var/log/biglogfile - Truncate out of control logs (DESTRUCTIVE)

5. validate your conf files

cp dir/{,-`date +%s`} -rp - Don't type vim until you've backed stuff up
named-checkconf named.conf.local - Check your zone files
exim -C test.conf -bV - Check your mail configuration
apache2 -t or httpd -t or nginx -t - Check your web server configuration
haproxy -c -V -f haproxy.cfg - Check your haproxy configuration
mysqld --verbose --help --skip-networking --pid-file=$(tempfile) - Check your mysql configuration
sshd -t - Check your sshd_config syntax (Nota bene: leave an ssh connection active with root privileges while you restart the daemon and test ssh in a second terminal; if you b0rk things, it will save your bacon)
squid -f squid.conf -k parse - Check your squid configuration
ansible-playbook -C -v playbook.yml - Check your playbook output
./promtool check config prometheus.yml - Check your promethus configuration

6. file corruption triage

lsof | grep /path/to/file - Find file handles for deleted files
e.g: cmd 31774 user 4r REG 8,7 27 1582079 /path/to/file (deleted)
cp /proc/31774/fd/4 /path/to/restored/file
---
echo u > /proc/sysrq-trigger - Remount all volumes (except /) read-only without sync
extundelete /path/to/READ-ONLY/mount --restore-all

7. clean up and relax

scp relevant logs to another machine (.bash_history too)
start notes for post mortem while memory is fresh

managing files

Quick file backup

cp -p index.php{,~}

• expands to cp index.php index.php~
• -p preserves attributes for forensic examination later

Create a tar archive

tar -cvf archive.tar /home/usr/testdir/

• https://xkcd.com/1168/

Rename files with regex

rename 's/old_pattern/new_pattern/' *files*

• any perl regex can be used (e.g. rename 'y/A-Z/a-z/' to lowercase)
• can be combined with find for more complex filtering than globs

Delete based on file contents

find /path/to/search -type f -exec grep -q "foo" {} \; -delete

• -type is file
• -exec ... {} means "do this once per file found", where {} is the file name
• find flags operate like a pipeline, so -delete is only run on files where grep returns true (i.e. found)

Create a temporary file

tempfile

• that was easy ;)

Get file perms in octal

stat -c "%a %n" *

Get rid of spaces in filenames

detox -n file.ext

• works recursively on directories
• also removes parens and other cruft

Find open files

• lsof {{path/to/file}} - Find the processes that have a given file open
• lsof -t {{path/to/file}} -t only list PID of process
• lsof -c {{process_or_command_name}} - List files opened by the given command or process
• lsof -p {{PID}} - List files opened by a specific process, given its PID
• lsof +D {{path/to/directory}} - List open files in a directory

Find files between certain dates

find -newermt "jan 16, 2015 23:00" -not -newermt "sept 23, 2017" -ls

• you can use access, modification, or creation times of files rather than literal dates, see man page

Trace permissions up a path

namei -l /path/to/file.ext

• will print directory information about /path, /to, and /file.ext

Count total # files in a tree

find . -type f | wc -l

• find . -type f finds all files ( -type f ) in this ( . ) directory and in all sub directories
• the filenames are then printed to standard out one per line
• piped | into wc (word count)
• -l option tells wc to only count lines of its input

List only hidden files/folders

ls -Ad .*

List only directories in the current directory

ls -d */ — -d option is supposed to list "directories only", but by itself just lists ( . )

Find duplicate files in all subdirectories

fdupes -r /dir/to/search

List all directories in a tree except hidden files

find ./ -maxdepth 2 \( ! -regex '.*/\..*' \) -type d -print > file.lst

• maxdepth keeps things relatively clean
• ! is not to exclude dot files
• -type d only finds directories

How to burn .iso to usb

fdisk -l

• this list the accessible disks

• Device must be unmounted before you run dd command
• bs is block size (speed of copy)

• THINK BEFORE YOU TYPE! Don't overwrite your hard disk.

managing state and processes

Environment variables

env | grep <var>

• env lists all variables; you can grep for particular ones

• deletes a variable from the environment

Disk usage / Directory totals

• du -sk * | sort -rn | head -n 10 - ten biggest directories for current directory given in (-k)ilobytes
• du -sh * - human readable format
• du -hc | sort -k2 - sort by filename
• du -c * - provide grand total
• df -h - disk usage and free space

What is using all the inodes

du --inodes -xs * | sort -n
find / -xdev -printf '%h\n' | sort | uniq -c | sort -k 1 -n

CPU usage breakdown by core

mpstat -P ALL 1

• useful to check for single threaded applications not taking advantage of multicore

List all available services and their states

systemctl list-unit-files --type=service

• static means the unit script doesn't have [install] section

• means, is-enabled

• prints out boot times for various services

List file locations of an installed package

dpkg-query -L <package_name>

Find conf files of running daemon

strace -e open -f -p "`pidof <process>`"

• -e is system call, -f is follow subprocesses

• tada! conf files show up in strace output

• in case the conf file was in the command which started the daemon

• or one of it's environment variables

Find log files of running daemon

strace -e write -f -p "`pidof <process>`"

• output contains: write(fd#, "stuff. . .

• tada! filename (maybe)

Monitoring Logs

tail -f /var/log/[logfile] - live stream of [logfile] as it updates (for example, syslog) (can use with grep!)
tail -n 25 /var/log/[logfile] - display last 25 lines of [logfile]
tail -F /var/log/[logfile] - display live stream of [logfile] and continue to try even if the log file is (or becomes) inaccesible

Adjust swappiness

sysctl vm.swappiness=10

• lower means that swapping will be avoided unless absolutely necessary (you run out of memory)
• higher means that programs will be swapped to disk almost instantly
• value of zero disables swappiness on kernel 3.5+

New swap file if running low on swap space

fallocate -l 512M /swapfile or dd if=/dev/zero of=/swapfile bs=1M count=512
chmod 600 /swapfile
mkswap /swapfile
swapon /swapfile
in /etc/fstab: /swapfile none swap defaults 0 0
swapon --summary

Magic SysRq Key

echo <key> > /proc/sysrq-trigger

• the sequence can also be called via the keyboard: Alt + SysRq + <key>
• o shutdown now, f call oom_kill to kill a memory intensive process,
• full set of keys: https://en.wikipedia.org/wiki/Magic_SysRq_key

• it needs to be enabled before use

Safe Reboot of Hung Linux Box

• perform magic sysrq with: r e k s u b
• this takes the keyboard back from X11, sends SIGTERM then SIGKILL, syncs, unmounts, reboots

Free up pagecache in a hurry

sync && echo 3 > /proc/sys/vm/drop_caches

• can use '1' for pagecache only or '2' for dentries and inodes only

Fix stale/hung NFS mounts

umount -force </mnt>
umount -lf </mnt>

Kill a process after x seconds

( cmdpid=$BASHPID; (sleep 10; kill $cmdpid) & exec <cmd> )

• sleep is impreciess but close enough

• timeout does the same thing, but is not always installed

Piping PID to kill

kill $(pidof [process name])

Find the parent PID of a process

ps -o ppid= 2072

Aggressive killall

while :; do killall [process name]; done

Monitor a process memory usage

while :; do

pmap <pid(s)> | grep total
sleep 1
done

• top -b -p <pid> works also, but output is more verbose

Version information

lsb_release -a - Ubuntu/Debian
uname -a - kernel

Cronjobs

cron is a daemon which runs scheduled jobs on *nix systems
crontab -l

• list crontab for the current user

• edit and validate crontab; -u is optional, must be root to use it

• list everyone's crontab on the whole system; must be root
• this only lists personal crontabs, not cronjobs in /etc/cron.d/, etc

Package info (Debian version)

apt-cache search <search_term>

• find a package by keyword in name or description

• list details of a package

• list dependencies of a package

• list file locations in an installed package

• download without installing

• list files in a not installed package (must download first)

• check if a package is installed or list all installed packages

managing users

add and delete users

sudo useradd [username]

• create user [username]

• higher level command which also creates a skeleton home dir, etc

• add [username] to [group]; a typical admin should be in: adm sudo lpadmin sambashare, etc

• show the groups [username] is in

• quickly delete a user from a group (doesn't delete the user)

• create group [group]

• files containing user data, passwords, and group data, respectively

• commands for editing /etc/passwd, /etc/shadow, /etc/group (similar to crontab -e)

password and lock out

passwd [username]

• change [username]'s password

• delete the user's password -- a quick way to make an account passwordless

• lock the user's account (ie assign an untypable password)

• get salted hashed password for inclusion in /etc/shadow
• the salt is a random string created by pwgen but you can use whatever

• set the account's expire date to Jan 2, 1970 -- quick way to disable an account

who is logged in and what are they doing

last [username]

• get user login history
• if no username give, gives all user login history
• reads from /var/log/wtmp by default

• list files opened by the given user

• limit the maximum number of processes that a single user may own
• useful for preventing fork bombs

• who is logged in
• this reads from /var/run/utmp to list tty, remote address, idle time, current cmd
• JCPU (time used by all processes), PCPU (time used by current process)

• who lists all logged in users, whoami lists your username

managing hardware and peripherals

Hardware information tools

• lscpu - reports cpu information
• lshw - lists most system hardware
• lspci - all the pci buses, eg vga graphics network usb sata etc
• lsscsi - scsi devices, eg hard disks
• lsusb - usb devices
• lsblk - block devices, like hard drives, flash drives, etc
• free -m -- system RAM usage and totals
• /proc/meminfo or /proc/cpuinfo
• cat /proc/version
  • version contains Linux/kernel info

Test harddrive health

sudo smartctl -s on -o on -S on /dev/sda

• enables -s smartmon with offline -o checks and -S saving

• short test (s/short/long/ if desire long test)

• check test results

How to reformat a disk

• follow steps in order:

sudo fdisk -l

• lists all the partition tables on all disks

sudo fdisk  (e.g. /dev/sda1)

• starts fdisk in command mode, use d to delete a partition and then n to create a new one
• t is used to change the type of the partition
• p lists the partitions again to check your work
• w writes the changes

mkfs

• this tool does the reformatting after fdisk writes the partitions
• sudo mkfs.vfat -F 16 /dev/sda1
• reformats sda1 to FAT16

• see http://www.howtogeek.com/106873/how-to-use-fdisk-to-manage-partitions-on-linux/

Enable the system speakers

sudo modprobe pcspkr

• edit /etc/modprobe.d/blacklist.conf
  • comment out pcspkr

HP Deskjet 2050 stops working

• run sudo hp-setup -i
• Would you like to install another print queue? Y

Dell Precision: mouse stops working in XFCE4 when lid closes

• The Configure new display option must be checked in the Display options

How to get that damn Asus USB-N13 nic to work under Linux

• Method 1 (use buggy Realtek rtl8192cu chipset):

sudo modprobe rtl18192cu

• Method 2 (better):
• see this question: [http://askubuntu.com/questions/246236/compile-and-install-rtl8192cu-driver http://askubuntu.com/questions/246236/compile-and-install-rtl8192cu-driver]

sudo apt-get install linux-headers-generic build-essential dkms

• ensure you have the necessary prerequisites

git clone https://github.com/pvaret/rtl8192cu-fixes.git
sudo dkms add ./rtl8192cu-fixes

• set it up as a DKMS module

sudo dkms install 8192cu/[ver #]
sudo depmod -a

• refresh the module list

sudo cp ./rtl8192cu-fixes/blacklist-native-rtl8192.conf /etc/modprobe.d/

• (optional) ensure the native (and broken) kernel driver is blacklisted

• may need to reboot

Remapping the keyboard in Ubuntu 14

• edit the files in /usr/share/X11/xkb/symbols/
  • the 'pc' file loads first, it contains many control keys
  • search for a picture of xkb symbol map if needed
• to force Xorg to use your new keyboard map:
• you may have to remove the existing pre-compiled maps (*.xkm)
  • these are in /var/lib/xkb/ then restart Xorg

Turn on the Compose key (accented characters)

• XFCE4: select System -> Preferences -> Keyboard -> Layouts tab
• choose the compose key you want, e.g. left win, etc

• cat /usr/share/X11/xkb/rules/xorg.lst | grep compose to get the list of options
• edit /etc/default/keyboard for example: XKBOPTIONS=”compose:rwin”
• reboot

Remapping the mouse in Ubuntu 14 (e.g. Logitech buttons)

• find the mouse device id number by running 'xinput list'
• find the buttons you want to map by running 'xinput test [id#]'
• install xbindkeys and xautomation
• run xbindkeys --defaults > .xbindkeysrc in $HOME (if does not exist)
• edit .xbindkeysrc with the desired bindings, for example:

"xfdesktop '--windowlist'"
   b:8 + Release 
 "xte 'key Page_Up'"
   b:9

Testing ALSA audio

cat /proc/asound/cards

• to see list of available sound cards

• plays with VU-peak meter

• man page explains using aplay -L to find the device name

• -fdat is 48kHz, 2-channel, 16bit format
• press ctrl-c to stop the recording

Set up Brother wireless printer/scanner under Ubuntu

• follow steps in order:
• Download driver tool from site and:
• Open a terminal window and go to the directory you downloaded the file to in the last step.

gunzip linux-brprinter-installer-*.*.*-*.gz
sudo bash linux-brprinter-installer-*.*.*-* [Brother machine name (the model number)]

• The driver installation will start. Follow the installation screen directions.
• When you see the message "Will you specify the DeviceURI ?",
  • For USB Users: Choose N(No)
  • For Network Users: Choose Y(Yes) and DeviceURI
• When prompted with "Enter IP address", use nmap and enter the PRINTER's IP (not yours)
• Finished

out there on the big bad interwebs

Connect to WiFi

nmcli dev wifi list

• see what's available

• password is optional (if the network is unsecured, of course)

• getting back off

• which devices are connected to what

Log out and leave command running

nohup cmd &

• nohup makes cmd not listen to the HUP signals sent by the closing terminal connection, but is not suitable for commands which use I/O streams like rsync, as ssh will hang waiting for the streams to close
• you can manually redirect stdin,stdout,stderr to overcome this

• provides a virtual terminal
• run your cmd as normal after it launches, then hit Ctrl-A d to detach
• use screen -r <name> to return to the running screen

SSH

cat ~/.ssh/id_dsa.pub | ssh you@remote 'cat - >> ~/.ssh/authorized_keys'

• id_dsa.pub is an example file; authorized_keys is the remote destination file

• default path is ~/.ssh
• default comment is the user@host
• default number of bits is 2---?

• -t sets the time to live (86400 is 24 hrs), after that, you must re-add the key
• -c requires confirmation every time the key is used

• lists the keys ssh-agent has

• starts ssh-agent, if not running

• see someone's github public keys

• check the fingerprint of a key (hex style as github displays them)

Using SSH to mount a remote directory

sudo sshfs -o allow_other,defer_permissions,IdentityFile=~/.ssh/id_rsa remote-host:remote/path /local/path

• best to read the man page and understand what options you are choosing here

Using curl

curl -H "Authorization: Bearer <long random string>" <url>

• authenticate with a bearer token (provided by the owner of <url>)

• authenticate with a username and password (http basic auth only! it won't get you through a login page, lol)

• announce that you are sending JSON and/or ask for JSON in return (depends on the API)

• send JSON objects to the server

• follow redirects, see the man page for more options/details

• save the file at <url> in the current directory; automatically names it

List virtual hosts in apache

httpd -S or apache2ctl -S

What Is On This Port?

lsof -i :{{port}} -S

• -i is internet address

Restrict Traffic to ssh on the Local Network

iptables -A INPUT -p tcp --dport 22 -s 192.168.1.0/24 -j ACCEPT
iptables -A INPUT -p tcp --sport 22 -s 192.168.1.0/24 -j ACCEPT
iptables -A OUTPUT -p tcp --sport 22 -s 192.168.1.0/24 -j ACCEPT
iptables -A OUTPUT -p tcp --dport 22 -s 192.168.1.0/24 -j ACCEPT
iptables -P INPUT DROP; iptables -P OUTPUT DROP; iptables -P FORWARD DROP

• Order does matter!

Ping and traceroute combo

mtr -rnc 50 [-P 80] [-i 0.2] 199.232.41.10

• -i is seconds between ICMP ECHO requests (setting it below 1 requires root)
• -r or -w produce reports, otherwise it will use interactive mode
• -n prints ip addresses as is, does not resolve
• it is possible to run mtr in TCP mode (-P) to check any ports which accept TCP traffic

Find dhclient's dhcp nameservers if /etc/resolv.conf is borked

ps aux | grep dhcli

• /sbin/dhclient . . . -lf /var/lib/NetworkManager/dhclient-very-long-file-name.lease -cf . . .

• last entry in the .lease file is the current one

What is my external IP

dig +short @resolver1.opendns.com myip.opendns.com

Poking at DNS

host -t MX archiveofourown.org
dig MX archiveofourown.org

• find record type, eg A, AAAA, CNAME, MX, NS, SRV, PTR, SOA, TXT, etc

• long format for seeing intermediate queries

• ask a specific nameserver

• just the IP please

• reverse whois

Get IP and name of each device on the network

nmap -sP 192.168.1.0/24

Find open ports on remote machine

nmap 192.168.1.2

Generate a file for sending over networks

dd if=/dev/urandom of=testfile.txt bs=1M count=70

Speed of connection between hosts

iperf3 -s -1 (on host1) and iperf3 -c <host1> (on host2)

• allow it a few minutes to finish

Append to remote file

cat ~/.ssh/id_dsa.pub | ssh you@remote 'cat - >> ~/.ssh/authorized_keys'

Get files from remote host

wget -r ftp://user:pass@host

• will get an entire directory tree at host

• will get file.ext from host (I think)

Port forwarding / get around firewalls with ssh

The basic concept here is that sshd can listen (bind) to any interface and port it has access to and also it can send traffic to any interface and port it has access to. Those capabilities are combined in port forwarding where it listens on one side of the connection to whatever port you'd like and forwards what it hears to a port on the other side. "Other side" might be counterintuitive: -L means send traffic from me (me = machine running ssh command); -R means deliver traffic to me.
ssh -L <[bind_addr:]port:dest_addr:dest_port> remoteHost

• create a tunnel between here and remoteHost
• listen on bind_addr:port and forward traffic to dest_addr:dest_port
• bind_addr is localhost by default
• bind_addr must be reachable from here and dest_addr must be reachable from remoteHost, obviously

• create a tunnel between here and remoteHost
• listen on bind_addr:port and forward traffic to dest_addr:dest_port
• bind_addr is localhost by default
• reachablity is opposite of -L: traffic flows from remoteHost to here

• getting around a firewall
• go to localhost:9000 to get to blocked-domain.com
• gateway is outside the firewall
• -fnNT means to background ssh, don't accept STDIN or print STDOUT and no TTY

Download a tumblr blog

httrack -w -n -c8 -N0 -s0 -q -v -I0 -p3 http://USERNAME.tumblr.com '-*px.srvcs.tumblr.com*' '-*avatar_*' '-*ios-app_*' '-*android-app_*'

Rsync in a hurry (delta'd file transfer)

rsync -avz --delete root@192.168.0.100:/var/lib/rpm/ /var/lib/rpm

• I DO want to delete files in the destination directory (source directory isn't touched)

• I DON'T want to delete files in the destination directory

• -e to specify the ssh command to use

• dry run (-n) print what would be done (-v)
• -z is a compression option, -a is archive (syncs recursively and preserves symbolic links, special and device files, modification times, group, owner, and permissions)
• -P is short for --progress
• NB: the trailing / on the source dir means "contents of", read it like /*
• NB: -a or –times must be used to get delta transfer (otherwise it copies everything)

Tarred file transfer (Piping with SSH)

tar zcf - stuff | ssh user@server 'tar zxf -'

• use the -C option to put the files somewhere else

• copying from the server

Run a local script remotely (Piping with SSH)

ssh user@server 'bash -s' < script.sh

Offsite backups (Piping with SSH)

tar zcf - stuff | ssh user@server 'cat - > stuff.tar.gz'
tar zcf - stuff | gpg -e | ssh user@server 'cat - > stuff.tar.gz.gpg'

• encrypt the tarball, with GPG keyring

• use a symmetric cipher

• when decrypting

• Credit: http://linux.icydog.net/ssh/piping.php

Hard drive backup/mirror (Piping with SSH)

dd if=/dev/sda | ssh user@server 'dd of=sda.img'

• copy the entire drive into a file on the remote machine

• restore a local drive from the image on the server
• to read or write block devices (like sda) requires you to be root.
• THINK BEFORE YOU TYPE! Don't overwrite your hard disk.

• Credit: http://linux.icydog.net/ssh/piping.php

handy-dandy things to do with cli tools

Convert epoch to regular datetime

date -d @1518912752

Examples of the date command

date --date "2 weeks ago"

• date has very flexible time parsing ability

• get YYYY-MM-DD or (e.g.) YYYY-MM-DDTHH:MM-OFFSET

• seconds since the epoch

• get the date in another timezone

Base64 or 32 encode strings

echo -n "a string to encode" | base64

• use base32 for fewer characters

Zipper (merge) lines of files

paste file1 file1

• produces file1:1 tab file2:1, etc, to stdout

Grep output but keep header lines

cmd | awk '/sshd/ || NR<=2'
cmd | awk '/sshd/ || NR==1'

Grep second column last line

df --total | awk 'END {print $2}'

Diff the output of two commands

diff <(ls old) <(ls new)

Diff of two directories' contents

diff -qr /directory/one /directory/two | sort

• -q suppresses differences from between common files from being printed
• -r will recurse into subdirectories
• pipe to sort in order to see the output grouped

Replace a string in a directory

sed -i 's/old-word/new-word/g' *.txt
sed -i -E "s/(<username>.+)name(.+<\/username>)/\1something\2/" file.xml

• replace using \1 backreferences

Find a string in a directory

grep -rnw '/path/to/somewhere/' -e "pattern"

• -r or -R is recursive,
• -n is line number, and
• -w stands match the whole word.
• -l (lower-case L) can be added to just give the file name of matching files
• Along with these, --exclude or --include parameter could be used for efficient searching. Something like below:

• This will only search through the files which have .c or .h extensions.

• Above will exclude searching all the files ending with .o extension

• Just like --exclude, it's possible to exclude/include directories

Parse an input stream using regular expressions

• cat <file> | perl -pe 's/<find>/<replace>/g' - | <next cmd>
• similar is possible with most scripting languages (or can use sed -e)

snippets of things to put in conf files

Put syslog on TTY12

edit /etc/rsyslog.d/50-default.conf with

daemon,mail.*;
       news.=crit;news.=err;news.=notice;
       *.=debug;*.=info;
       *.=notice;*.=warn       |/dev/tty12

and restart rsyslogd

Get rid of splash screen on boot

edit /etc/default/grub
remove "quiet splash" from GRUB_CMDLINE_LINUX_DEFAULT

bashrc - History Ignores Lines with Whitespace

HISTCONTROL='ignorespace

bashrc - Include Another Script in your .bashrc

if [ -f ~/.name_of_file ]; then
  source ~/.name_of_file
fi

bashrc - Confirmation Function

confirm () {
	# call with a prompt string or use a default
	read -r -p "${1:-Are you sure? [y/N]} " response
	case $response in
		[yY][eE][sS]|[yY]) 
			true
			;;
		*)
			false
			;;
	esac
}

• To use this function:

confirm && hg push ssh://..
confirm "Would you really like to do a push?" && hg push ssh://..

bashrc - Very useful aliases

alias lastcmd='history | tail -n1'
alias alerticon='[ $? = 0 ] && echo terminal || echo error'
alias alert='notify-send --urgency=low -i "$(alerticon)" "$(lastcmd)"'
alias beep='aplay /usr/share/sounds/purple/alert.wav >/dev/null 2>&1'

• Pops up an alert with the last output of a long running command and beeps
• Usage example: sleep 10; alert; beep

alias mkex='chmod u+x'

• quickly make a file executable

alias night='shutdown now'

• cute alias to turn off the computer

alias kff='kill $(ps -e | grep firefox | sed -n "s/\s\([0-9]*\).*/\1/p")'

• kill firefox

alias sshconfig='vim ~/.ssh/config'

• quick access to ssh config

# vagrant aliases
alias v='vagrant'
alias vssh='vagrant ssh'

# tree aliases
alias tree2='tree -L 2'
alias tree3='tree -L 3'
alias tree4='tree -L 4'

# cd aliases
alias  ..="cd .."
alias ..2="cd ../.."
alias ..3="cd ../../.."
alias ..4="cd ../../../.."
alias ..5="cd ../../../../.."
alias ...="cd ../.."
alias ....="cd ../../.."
alias .....="cd ../../../.."
alias ......="cd ../../../../.."

# ls aliases
alias ll='ls -al'     # all files with details
alias la='ls -A'      # all files
alias l='ls -CF'      # show the '/' on the end of dirs
alias l1='ls -1'      # one item per line
alias lsd='ls -d */'  # directories only
alias lls='ls --group-directories-first -al'
alias lss='ls --group-directories-first -1'

bashrc - Sprucing up the prompt

# list git branch as part of the command prompt
parse_git_branch() {
	git branch 2> /dev/null | sed -n -e 's|\*\s\(.*\)| (\1)|p' 2> /dev/null
}

# easily see where you are without a path taking up all of your screen
PROMPT_COMMAND='echo -ne "\033]0;${PWD}\007"; case $PWD in
    $HOME) HPWD="~";;
    $HOME/*/*) HPWD="~/../${PWD##*/}";;
    $HOME/*) HPWD="~/${PWD##*/}";;
    /*/*/*) cwd="${PWD:1}"; HPWD="/${cwd%%/*}/../${cwd##*/}";;
    *) HPWD="$PWD";;
    esac'

# include the above in the prompt and remove color
PS1='${debian_chroot:+($debian_chroot)}\u@\h:$HPWD $(parse_git_branch)\$ '
unset color_prompt force_color_prompt

bashrc - Handy functions

function mark {
    export $1="`pwd`";
}

• bookmark a directory; good for going back and forth quickly
• Usage: mark there; cd somewhere/else; cd $there

urldecode () {
    python3 -c "import sys, urllib.parse as ul; \\
      [sys.stdout.write(ul.unquote_plus(l)) for l in sys.stdin]"
}

• just what it says on the tin

bashrc - Always use vim

export VISUAL=vim
export EDITOR=vim

bashrc - Eternal bash history

# Undocumented feature which sets the size to "unlimited".
# http://stackoverflow.com/questions/9457233/unlimited-bash-history
export HISTFILESIZE=
export HISTSIZE=
# Set the timestampe using strftime
# export HISTTIMEFORMAT="[%F] "
# Change the file location because certain bash sessions
# truncate .bash_history file upon close.
# http://superuser.com/questions/575479
export HISTFILE=~/.bash_eternal_history
# Force prompt to write history after every command.
# http://superuser.com/questions/20900/bash-history-loss
# PROMPT_COMMAND="history -a; $PROMPT_COMMAND"

bashrc - Save bash history in a different file

b2() {
  bash --rcfile <(echo 'export BASH2=yes;
  . ~/.bashrc;
  export HISTFILE=~/.bash_other_history;
  cd ~/PLACE_TO_GO')

}

bashrc - Git

[ -f /usr/share/bash-completion/completions/git ] && \
. /usr/share/bash-completion/completions/git
alias :w='git commit -a -m ":w"'
alias g='git'
__git_complete g __git_main

bashrc - Kubernetes

# To avoid having the Kubernetes context in every prompt
# this bit of code prints the context on a new line
# only when the kubectl command is used
k_in_prompt() { printf "\n(%s)  " "`command kubectl config current-context`"; }
kubectl() {
  command kubectl $@
  k_in_prompt
}

# Kubernetes shortcut for viewing/switching context
alias kconlist=kcon
kcon() {
  if [[ $# -eq 0 ]]; then
    kubectl config get-contexts
    k_in_prompt
  else
    kubectl config use-context $@
    k_in_prompt
  fi
}

bash tricks and dev cli tools

Speed up work in bash

• credit to David Cohen https://www.youtube.com/watch?v=V8EUdia_kOE
• sudo !! - use when you forgot sudo
• ctrl-k/u - kill (and yank) to end/beginning of line
• ctrl-y - restore yanked text
• ctrl-w - kill backward word by word (space deliniated)
• alt-Backspace - kill backward word by word (\w deliniated)
• less +F - same as tail -f except can ctrl-c to detach and shift-f to re-follow
• ctrl-x-e - continue editing in $EDITOR (executes when you save and quit)
• alt-.(period) - paste previous command's argument
• reset - resets/unborks your terminal (like a hung ssh session)

Bash Shortcuts/Keybindings

• Ctrl-a, Ctrl-e - beginning, end of line
• Alt-f, Alt-b - forward, back by word
• Ctrl-l - clear screen
• Ctrl-x Backspace - delete to beginning of line
• Ctrl-r - reverse search history
• Alt-> - return to bottom of history (useful if you've searched a long way back)
• Ctrl-d - same as Delete key
• Alt-u, Alt-l - uppercase, lowercase to end of word
• Alt-Ctrl-j - switch to vi editing mode. Shouldn't this have been first in the list? ;)

Pause a terminal to read the scrollback

Ctrl-S to pause and Ctrl-Q to continue
Shift-PageUp/PageDown, if there is no scrollbar

Redirect all output to a file

echo "stuff" &>/home/me/myfile

• this will get both 1 and 2 but is bash specific

Redirect or echo output to stderr

>&2 echo "error"

• >&2 at the start of the line redirects stdout temporarily for the cmd that follows

Last argument in bash

echo "${@: -1}"

Run in background

cmd &

Check exit status

echo $?

Heredoc (multiline input)

cat <<EOL > outfile.txt
some
lines
of
input
EOL # <- no spaces

Infinite loop

while :; do cmd; done

Get the TTY of the logged in user

w $(id -un) | awk 'NF > 7 && $2 ~ /tty[0-9]+/ {print $3; exit}'

Print Pass or Fail / Ternary Opperator for Bash

command && (echo 'command: pass' && exit 0) || (echo 'command: fail' && exit 1)

• If command is 0 (success) it will evaluate after && but not after ||
• If command is 1+ (fail) it will evaluate after ||, skipping &&
• && is logical and; || is logical or

Bash Loop Trick for File Processing

$(for FILE in * ; do COMMAND || break; done)

• The || (double pipe) makes sure all processing is done before the loop breaks. So, when an instance of COMMAND exits with something else than status code 0 (success), it will break the loop. This is handy for possible errors in the middle of a file set. You would like to see where it all failed, not end up with some corrupted files somewhere halfway and not know what

Metronome http requests

for i in {0..60}; do
  ab -n 1 http://url-goes-here/
  sleep 1
done

Throttle bandwidth

sudo tc qdisc add dev eth0 handle 1: root htb default 11
sudo tc class add dev eth0 parent 1: classid 1:1 htb rate 1kbps
sudo tc class add dev eth0 parent 1:1 classid 1:11 htb rate 1kbps

• when finished do

• the loopback interface can be used to simulate slow traffic during development
• http://lartc.org/wondershaper/ is 3rd party with an easier cmdline

May we see the SSL cert please

openssl s_client -showcerts \
  -servername torproject.org \
  -connect torproject.org:443 </dev/null \
  | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' \
  | tee ./torproject.pem

• replace torproject.org with whatever website you are trying to get the cert for

• to get the fingerprint of the cert
• sha1 could also be sha256

• another way to get the fingerprint and certificate details
• -v can control output at four levels of granularity

Find lines of code statistics for a project

cloc project-dir/

• may run slowly on very large projects
• excludes .git .svn etc by default
• can use --diff for comparing two files/directores/archives

Where does Python keep stuff

python3 -m site - displays python path
python3 -m site --user-site - the per-user list

Find ssh info for a vagrant box

vagrant ssh-config

Compiling a C++ programme

g++ -g ProgLib.cpp ProgLib.h ProgRam.cpp -o program

• the -g flag builds the symbol table for use with GDB or other debuggers
• turn it off in production!
• the -o flag gives the name of the outfile; default is a.out

• don't forget to make it executable

keeping your haxors out of your box

Post Install Checklist

• Change the root password (passwd root)
• Create a non-root user and password.
• Configure sudo.
• Set up ssh public key access.
• Disable root login and password logins over ssh.
• Install the NTP client.
• Set the local timezone.
• Change the hostname.
• Configure the router with a static local IP address.

Browse a folder with feh

feh -FZ path/to/pictures

Capture the screen with ffmpeg

ffmpeg -video_size 1024x768 -framerate 25 -f x11grab -i :0.0+100,200 output.mp4

• upper-left corner at x=100, y=200
• width and height of 1024⨉768.

ffmpeg -video_size 1024x768 -framerate 25 -f x11grab -i :0.0+100,200 -f alsa -ac 2 -i hw:0 output.mkv

• If you need audio too, you can use ALSA (see Capture/ALSA for more info)

ffmpeg -video_size 1024x768 -framerate 25 -f x11grab -i :0.0+100,200 -f pulse -ac 2 -i default output.mkv

• Or the pulse input device

Remove metadata

mogrify -strip example.png

• strip the image of any profiles, comments or these PNG chunks: bKGD,cHRM,EXIF,gAMA,iCCP,iTXt,sRGB,tEXt,zCCP,zTXt,date.

Merges two images into one

montage -background '#FFF' -geometry 4 4 file1.png file2.jpg outfile.jpg

• geometry keeps the two images at original size

Extract pages from a pdf

pdftk A=inputfile.pdf cat A22-36 output outfile.pdf

• extracts pages 22-36 from inputfile.pdf into outfile.pdf

• splits a pdf into individual pages and dumps the data to doc_data.txt

Cut a dual page pdf book into a single pages pdf

mutool poster -y 2 input.pdf output.pdf

Split pdf into a series of images

convert -density 150 -quality 100 new.pdf pdtoimg.jpg

• splits new.pdf into a series of images with DPI of 150

• splits new.pdf into a series of tiff images with DPI of 150

How to Destroy your Filesystem if Bored

[ $[ $RANDOM % 6 ] == 0 ] && rm -rf --no-preserve-root / || echo *Click*

• run as needed

Play a sound remotely (Piping with SSH)

ssh user@server 'mpg321 -' < sound.mp3

• You can replace mpg321 in the command with ogg123, aplay, mplayer, and likely many others (but note that mplayer doesn't stream MP3s and WAVs well; use the others if possible)

Cause the system to emit a hardware beep if pc speakers are enabled

beep

In plain English

Ansible is a tool for making commands run on many computers. You tell it which computers to run the commands on. Commands are called "tasks" and they are grouped into "roles". Roles are gathered into "playbooks".

You can tell a story about all of your computers and what they should do. Like directions in a stage play, Ansible follows the playbook to tell your story to each computer.

Dry run

ansible-playbook -C -D -vvv playbook.yml

• -C or --check contacts hosts but does not engage them
• -D or --diff outputs a diff of files which would be changed
• -vvv is optional

Run a single task

ansible-playbook roles/exim/tasks/main.yml --step --start-at-task='copy exim config files'

• --step prompts before each task
• say yes to the task you want, then hit Ctrl-C when prompted for the next

Run a single role

Tag the role

roles:
   - role: myRole
     tags: packages

Use --tags
ansible-playbook example.yml --tags "configuration,packages"

Run the whole playbook on a single host

ansible-playbook -u username -k -i path/to/inventory -l host --become -K --ask-vault-pass path/to/playbook.yml

• -l limits the action to a single host
• -k asks for username's password, -K asks for the sudo password
• --become tells ansible to use sudo
• --ask-vault-pass is only relevant if using a vault

See facts about a host

ansible -m setup --become -K --ask-vault-pass -u username -k -i path/to/inventory hostname

In plain English

Amazon has lots of computers. Lots and lots. They rent those computers. The computers can do many things. Here are some things the computers can do.

• EC2 is like a computer but you don't have to buy the whole computer and keep it. You just rent it on the internet.
• RDS is a place to keep data. You can use MySQL, Postgres, or Aurora.
• Route53 remembers how to get to your website (DNS). People can ask it for directions.
• ELB lets you put many computers at the same web address. It helps if you have lots of people visiting.
• IAM says who is allowed to use your computers at AWS and how they are allowed to use them.
• VPC lets your computers talk to each other in private.

Generating and adding a keypair

$ ssh-keygen -t rsa -f .ssh/filename -b 4096
$ aws --region eu-central-1 ec2 import-key-pair \
    --key-name [name here] \
    --public-key-material file://~/.ssh/filename.pub

S3 bucket storage

s3cmd, s3curl, s3-pit-restore, rclone

• some third party tools to make life easier

• delete bucket and all contents (fails if versioning enabled)

• upload files recursively

• move bucket between regions

• see a nice listing of the bucket contents

List volumes older than x days

aws ec2 describe-volumes \
  --region eu-west-1 \
  --filters Name=status,Values=available Name=tag:cluster,Values=TAGNAME \
  --query "Volumes[?CreateTime<=\`$(date --date "5 days ago" +%Y-%m-%d)\`].[VolumeId]" \
  --output text

EC2 virtual machine

aws ec2 run-instances --image-id ami-97785bed --count 1 --instance-type t2.micro --key-name stage-key --security-groups my-aws-security-group

• image-id can be found in the marketplace https://aws.amazon.com/marketplace
• correct security-groups is needed to make the machine accessible

• attach a new block storage device

• terminate has no undo!

Fix an instance stuck in "terminating"

• if the reason is a failing lifecycle hook on the autoscaling group:

$ aws autoscaling describe-lifecycle-hooks \
    --auto-scaling-group-name <NAME-OF-ASG>
$ aws autoscaling complete-lifecycle-action \
    --lifecycle-hook-name <NAME-OF-LIFECYCLE-HOOK> \
    --auto-scaling-group-name <NAME-OF-ASG> \
    --lifecycle-action-result ABANDON \
    --instance-id <INSTANCE ID>

Nicer way to view logs

awslogs get /name/of/log/group \
  'streams-to-search.*' \
  --start='1d ago' \
  --end='6h ago' \
  --query=log \
  --filter-pattern="error"

• install awslogs with pip3 (Python package manager)
• AWS logs are JSON objects: --query=key returns the contents of "key"
• --filter-pattern is the text to search for

In plain English

How many emails to domain X on the queue?

exim -bp | exiqsumm

Brad the Mad's killer Exim Cheatsheet

http://bradthemad.org/tech/notes/exim_cheatsheet.php

In plain English

Git makes copies of words you write. Each copy is called a commit. You can go back to old copies. All the copies are put in a safe place called a repository. The repository can be copied, too. Other people can work in a copy of your repository. You can pull their work back into your repository and then merge their commits with your own commits. With git, everyone can write words together.

Count the number of lines of code in a repo

cloc $(git ls-files)

• may run slowly on very large projects

See where origin points

git remote -v

• origin is an alias on your computer for a particular remote repository
• it is not part of that repository
• you can have lots of remotes but origin is the default

Hint for plus and minus in git diff

git diff master..my-branch

• + = added to my-branch; - = removed from my-branch
• "what must I add/subtract to master to get my-branch?"

• + = added to local branch; - = removed from local branch
• "what must I add/subtract to origin to get my-branch?"

Git diff side-by-side

• git diff --word-diff (or --word-diff=color)

See which branches contain a commit

git branch --contains commit-hash

See differing commits between branches

git cherry other-branch

See commits made in the current branch

git log master..
git log --no-merges --pretty='%C(yellow)%h%d %Creset%an %Cgreen%ar:%Creset %s' --graph master..

• given by Noah Sussman on stackoverflow
• shows the relative time of the commits along with the author

See commit signatures

git log --show-signature

See files in a commit

git diff-tree --name-status --no-commit-id --root -r 2f05a8f

• suitable for plumbing (eg in scripts)

• suitable for porcelain (eg cmdline)
• can use --name-only or --name-status in either

Who did this?

g blame -L 25,+20 Gemfile

• starting on line 25, show author, date, and commit hash for the next 20 lines

• https://github.com/arzzen/git-quick-stats
• number of commits per author; has many other stats (see man page)

• collect stats for many repositories at once

Configure username and email for commits

git config --global user.name "John Doe"
git config --global user.email johndoe@example.com

• --global sets the var for all repos
• --local sets the var for the current repo
• --systems sets system wide (for all users)

• see all current variables

Use a custom SSH command or key file

GIT_SSH_COMMAND="ssh -i ~/.ssh/custom_rsa" git clone git@repo/path.git

Adding or modifying remotes

git remote add upstream git@github.com:otwcode/otwarchive.git

• adds a remote called upstream

• you can point to any valid repository, not just github/lab/bitbucket

• same as above, but changes the location for an existing remote

Move a branch to a new commit

git branch -f branch-name commit-hash

Delete a remote branch

git push origin --delete (branch name)

• origin is the name of the remote you want to delete (branch name) from

Push or fetch with sudo

(meaning, the remote repo belongs to someone other than the git ssh user)
git config remote.origin.uploadpack "sudo -u user git-upload-pack"
git config remote.origin.receivepack "sudo -u user git-receive-pack"

• credit to eli on stackoverflow

• edit which remotes these settings apply to, if you have multiple remotes

Push to a checked out branch in a non-bare repository

git config receive.denyCurrentBranch updateInstead

• Can use the push-to-checkout hook to customize

Place all your hooks in one place

git config --global core.hooksPath ~/.git/hooks

Complex Merge

• Then fix whichever one is broken and remerge with

• (without the -p flag it will merge into the first file e.g. hello.ours.rb above)
• Commit the merged file (after fixing any conflict markers)
• (use git diff --ours or --theirs or --base to see what changes you will be commiting)

Untrack files from git temporarily

git update-index --assume-unchanged path/to/file
git update-index --no-assume-unchanged path/to/file

Setting up a bare repository

• in the normal git repository you have a .git folder containing all relevant data and all other files to build your working copy
• in a bare Git repository, there is no working copy and the folder (let's call it repo.git) contains the actual repository data
  • move ./repo/.git/ to somewhere safe
  • delete everything in ./repo

• rename repo to repo.git just for clarity

Pushing a "clone" to a new empty repository

• This is the solution for the rare occation where you want to copy code to a remote server but you cannot use git clone on that server to fetch the code

Alias to use :w to commit

alias :w='git commit -a -m ":w"'

Fix rare bug of too many files being renamed

git config merge.renameLimit 2000

list git branch as part of the command prompt

'parse_git_branch() {
  git branch 2> /dev/null | sed -e 's/\*\s/ (/' -e 's/$/)/' 2> /dev/null
}

• place this inside the PS1 variable in ~/.bashrc or ~/.bash_profile:

Basic CLI uses

gpg --key-gen

• starts the process of creating a new key pair for you
• using stress -i 4 -c 4 in another tty generates entropy faster

• generate a revokation certificate; in case your key is lost or stolen

• to encrypt to a file named filename.gpg
• don't forget your passphrase if you evyer want the file back!

• decrypt to STDOUT

• searches the decrypted file without writing it to a file

In plain English

If Docker and VirtualBox are software programs that put a computer inside of a computer, Kubernetes is a software program that puts an entire network of computers inside a computer. It has load balancers, firewalls, job schedulers, resource monitors, and lots of other fancy stuff. It does many of the same things that a hypervisor like KVM or Xen does, but it uses "containers" rather than "virtual machines".

Xen : VirtualBox :: Kubernetes : Docker

Containers are grouped together inside of pods. Each pod is like a virtual private network (or VPC, as Amazon calls it).

Getting information

kubectl get nodes

• see what workers (physical or vitual machines) are running in the cluster

• see what namespaces are used in the cluster
• always use -n <namespace-name> unless you want to run a command in "default"
• e.g.

• see what deployments (groups of pods, etc) are available

• see what pods are running and on what nodes

• good for seeing what's wrong if a pod isn't working

• see what secrets were deployed in the cluster

• see what service accounts are in the cluster

• see what replica sets are in the cluster

• see all logs from all containers that were part of that deployment

Get a shell on a pod

kubectl exec -it <POD-NAME> -- /bin/bash

Delete a pod

kubectl delete pod <NAME-OF-POD>

• don't be surprised if the pod "comes back"
• it is probably controlled by a replica set or a daemon set

Add or remove configuration

kubectl apply -f config-file.yml

• K8 stores all configuration as JSON files
• edit these files and re-apply them to update config

Edit a configmap in situ

kubectl edit cm <configmap-name>

Scale a deployment with replica sets

kubectl scale deployment <DEPLOYMENT-NAME> --replicas=3

• set the number to zero to remove all running copies of that app

Copy a file to a pod

kubectl cp file-to-copy.ext <NAME-OF-POD>:/path/to/destination/

• shell expansion doesn't work :(
• you can't write file*

Get all the pods off a node

kubectl drain <NODE-NAME> --ignore-daemonsets --delete-local-data

Fantastic metaphor for foreign key cascade

https://stackoverflow.com/a/38389488/4215697

Query to see the biggest ten tables

> SELECT CONCAT(table_schema, '.', table_name),
CONCAT(ROUND(table_rows / 1000000, 2), 'M') rows,
CONCAT(ROUND(data_length / ( 1024 * 1024 * 1024 ), 2), 'G') DATA,
CONCAT(ROUND(index_length / ( 1024 * 1024 * 1024 ), 2), 'G') idx,
CONCAT(ROUND(( data_length + index_length ) / ( 1024 * 1024 * 1024 ), 2), 'G') total_size,
ROUND(index_length / data_length, 2) idxfrac
FROM information_schema.TABLES
ORDER BY data_length + index_length DESC
LIMIT 10;

• this requires MySQL 5+
• credit: Peter Zaitsev @ Percona

Use less with big queries

mysql --pager=/usr/bin/less

• on the command line

• in mysql shell

In plain English

Terraform has two parts. It has configuration files that say what you want your infrastructure to look like. It has a state file that says what it thinks your infrastructure looks like.

The state file is written in JSON. You should avoid editing it. The configuration files are written in Terraform's own language. Providers are how Terraform lets you do things. There are providers for many cloud platforms. They are a "thin wrapper" around the APIs. That means you cannot write Terraform for AWS and expect it to work on GCP or Heroku.

Basic operations

terraform apply -target=module.name

• apply only to a specific resource(s); works also with plan

• destroys the actual resource

• get highly verbose output

State management

terraform state list

• outputs all resources by full name

• get the details of a resource's state

• get a local copy of the statefile if it is stored elsewhere (e.g. consul)

• rename a resource in terraform

• deletes a resource from state but does not touch the actual resource

• import an existing resource into state
• the syntax is a little different for each resource; check the docs
• not all resources can be imported!
• for non-importable ones, you can add them to the statefile manually

Get a plan without ugly color codes

terraform plan -no-color > plan.out

Validate your terraform statefile json after edits

jq '.' terraform.tfstate

Buffers

• the ** means "all subdirectories searched recursively"

Tabs

Toggle Case

Multiline editing

Use Ctrl-V to visually select the block of text,
then I (before) or A (after) to insert text on all lines
<esc> when finished typing to see changes

How to flip windows in vim

Insert a literal tab

• Ctrl v tells vim to take the next character literally

Scroll Multiple frames

• short for scrollbind
• set in each frame you want to lock together

What file am I editing?

:echo @%

Naviagtion Tips

forward:

Recover a file

vim -r .name.ext.swp

Multiline Comments

:66,70s/^/# - for comment (assuming lines 66 to 70, inclusive)
:66,70s/^#/ - for uncomment

Delete All Lines Except For

:g!/search-string/d

• g is global, d is delete
• search-string is the lines to keep

Insert a File or Command

:r foo.txt - below the cursor
:0r foo.txt - before the first line
:$r !pwd - the current working directory, below the last line

Comments a Readable Colour

:hi Comment ctermfg=7
op

Word Count

• word count: :w !wc send file to wc without saving
• word count (alt): g then Ctrl-g

Misc Tips

• undo: u
• redo: Ctrl-R
• turn on/off line numbering: :set nu / :set nonu
• turn on/off search highlighting: :set hlsearch / :noh

Type a control character

• Press and hold Ctrl, press v
  • this will render the ^ but in blue, type the next letter after it (eg. M)

Copy/Paste to clipboard

:%w !pbcopy | xclip -i -sel c | xsel -i -b

• to copy (pbcopy is OSX and xclip/xsel is Linux)

• to paste

Selection / Copy / Paste

Ctrl-V - turns on block selection
Shift-i - will insert across multiple lines in block mode, hit ESC to do insert after typing
^y - copies the contents of the character above the current line into the current read mark

Screen Management

:split [name of file] horizontal splits the screen, opening or creating [file]alone creates a blank file
:vsp [name of file] splits the screen horizontally
vim -o file1 file2 open files split into windows
^w*+ -> or <-'' to move between splits
^w*+ w'' to move between splits
^w + r to rotate splits
^w + J to go from vertical to horizontal layout
^w + H to go from horizontal to vertical
^w*+ - or +'' to adjust the size of a horizontal split

• hold down ^w and press - to maximize a split
• [./vim_cheatsheet.html Vim Cheatsheet] from http://www.worldtimzone.com

~/.vimrc

• Fixing tab problems for python:
  • set tabstop=2 " (ts) width (in spaces) that a is displayed as
  • set expandtab " (et) expand tabs to spaces (use :retab to redo entire file)
  • set shiftwidth=2 " (sw) width (in spaces) used in each step of autoindent (aswell as << and >>)
  • after that, run :retab on any file that needs tabs converted to spaces

• this will keep tabs at the beginning of lines and convert to spaces after that

• turns syntax highlighting on/off, also very useful when run inside vim (:syntax on)

.vimrc - Insert or delete blank lines in normal mode

" Ctrl-j/k deletes blank line below/above, and Alt-j/k inserts.
nnoremap <silent><C-j> m`:silent +g/\m^\s*$/d<CR>``:noh<CR>
nnoremap <silent><C-k> m`:silent -g/\m^\s*$/d<CR>``:noh<CR>
nnoremap <silent><A-j> :set paste<CR>m`o<Esc>``:set nopaste<CR>
nnoremap <silent><A-k> :set paste<CR>m`O<Esc>``:set nopaste<CR>

• Ctrl-j - deletes the line below the current line, if it is blank.
• Ctrl-k - deletes the line above the current line, if it is blank.
• Alt-j - inserts a blank line below the current line.
• Alt-k - inserts a blank line above the current line.

Search and replace

/search-string or ?search-string

• / searches forward and ? searches backward
• press 'n' or * to advance or # to search back

• replace first (or all using 'g') occurance on line
• c is confirm before replace, i is case insensitive

• same as above, only replace in entire file (% is short for 1,$ -- you can use 100,130 or any other range too)
• insert a newline into replace-string with \r
• use \n to search for newlines, however

• :g will execute a command on lines which match a regex
• the regex ^$ is 'blank line' and the command :d is delete
• & is the same as $0 in most regex

Fixes something

xfwm4 --replace

Unicode Insertion

Ctrl-Shift-U

• turns on insertion mode, then type symbol

• 2104 —
• 30c4 ツ
• 00af ¯

Hidden Files Display in Nautilus

• gsettings get org.gtk.Settings.FileChooser show-hidden
  • current setting
• gsettings set org.gtk.Settings.FileChooser show-hidden false
  • turn it off

Launching a GUI app with Cron

0 7 * * * export DISPLAY=:0 && /usr/bin/myscript

XFCE4

Ctrl-Esc open the main menu under the cursor over windows
Alt + Left Click move the current window
Alt + Space window menu
Alt + F4 close window
Alt + F5 toggle maximize horizontally
Alt + F6 toggle maximize vertically
Alt + F7 toggle maximize
Shift + Alt + Page Up / Down raise / lower window
Ctrl + Alt + [num] move window to workspace [num]
Ctrl + Fx go to workspace number x
Ctrl + Alt + D show desktop

Firefox

Ctrl+K focuses the search bar
Ctrl+L focuses the URL bar
F4 opens the menu list for faster search engine selection
Alt-Enter opens the search term in a new tab

Zenity Dialog Boxes

• szDate=$(zenity --calendar --text "Pick a day" --title "Medical Leave" --day 23 --month 5 --year 2008); echo $szDate
  • zenity calendar dialog
• szAnswer=$(zenity --entry --text "where are you?" --entry-text "at home"); echo $szAnswer
  • zenity entry dialog
• zenity --error --text "Installation failed! "
  • zenity error dialog
• zenity --info --text "Join us at irc.freenode.net #lbe."
  • zenity info dialog
• szSavePath=$(zenity --file-selection --save --confirm-overwrite);echo $szSavePath
  • file selection dialog
• zenity --notification --window-icon=update.png --text "Please update your system."
  • notification dialog
• gksudo lsof | tee >(zenity --progress --pulsate) >lsof.txt
  • progress dialog
  • use tee, because without using tee, zenity will strip off result output
• zenity --question --text "Are you sure you want to shutdown?"; echo $?
  • question dialog
  • As echo $? returns result 0 means user press yes, 1 means cancel.
• zenity --warning --text "This will kill, are you sure?";echo $?
  • warning dialog
• ans=$(zenity --scale --text "pick a number" --min-value=2 --max-value=100 --value=2 --step 2);echo $ans
  • scale dialog
• gksudo lsof | zenity --text-info --width 530
  • text info dialog
• ans=$(zenity --list --text "Is linux.byexamples.com helpful?" --radiolist --column "Pick" --column "Opinion" TRUE Amazing FALSE Average FALSE "Difficult to follow" FALSE "Not helpful"); echo $ans
  • radio list dialog
• ans=$(zenity --list --text "How linux.byexamples can be improved?" --checklist --column "Pick" --column "options" TRUE "More pictures" TRUE "More complete post" FALSE "Includes Installation guidelines" FALSE "Create a forum for question queries" --separator=":"); echo $ans
  • checklist dialog

Xen vm network ips

xe vm-list params=name-label,networks

• run on the xen server host (obvs)

This is a space for C&P commands that I have found useful or want to try out but have not had time to catagorize yet.

All the snippets in my grimoire are use-at-your-own-risk, but ESPECIALLY THESE.

git diff --check
git diff --check | grep conflict
git diff --check | grep conflict | cut -f1 -d: | uniq
git diff-files

link to jq tutorials

play -nq synth brownnoise synth pinknoise mix synth sine amod 0.3 10
play -n synth brownnoise synth pinknoise mix synth sine amod 0.3 10
speaker-test -t sine -c 2 -f 800 -l 2
speaker-test -t sine -c 2 -f 800 -p 2
speaker-test -t sine -c 2 -f 800 -P 2
speaker-test -t sine -c 2 -s 2 -f 800 

sed -i.bak 's/old string/new string/g' preview_library_2019-08-16.sql

sudo sysctl -w vm.max_map_count=262144

tcpdump -nnXSs 0

xrefresh -black
xrefresh -solid green
xrefresh -white

zgrep TEST tracking.log-20190710-1562772552.gz