# Shell recipes
The last time I organized a workshop at the local hackerspace, I made copious use of the shell. The audience ranged from ambitious youngsters to greybeards who ate their teeth on decades of Linux. And yet, I managed to surprise them with a tiny piece of bash.
Here's the piece:
```
mv /home/dcz/file.md{,.bak}
```
Notice the brackets and the comma? This syntax turns the path into two paths: `/home/dcz/file.md` and `/home/dcz/file.md.bak`. How could a shell-hater like me know of it, while true hackers didn't? I think it's a testament to shell's discoverability of features. (Hint: it's awful.)
## The tricks file
It turns out that other people I know have a similar problem with the shell: they remember that they did something, but they can't easily find the way after some time. They create a file containing useful commands or incantations. I do as well, and I was asked to share it.
Most of those tricks will not directly apply to you. But they can serve as a base to whatever use case you have in mind, after slight adjustments. And of course I'm not responsible for any loss of data resulting from using my recipes.
So here goes!
### ARM emulation wth binfmt
This one is useful before you want to run a binary compiled for the ARM architecture. It requires QEMU to be installed, and a chroot containing an ARM file system. I usually use podman to pull an arm64 container, so I don't have to create the file system myself. I use it to build binaries for the Librem 5 without having to set up the cross-compiler.
```
echo ':aarch64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7:\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-aarch64-static:F' > /proc/sys/fs/binfmt_misc/register
echo ':arm:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-arm-static:F' > /proc/sys/fs/binfmt_misc/register
```
The podman command for armhf is:
```
podman run -ti -v /containers/build:/mnt/build:z --name=debian_armhf multiarch/debian-debootstrap:armhf-buster /bin/bash
```
### Copy a hard drive with bad sectors (untested)
When a hard drive has bad sectors, the reading process will eventually hit an area that cannot be read immediately. Most copying programs will retry several times, or abort outright. This is wasting lots of time if you have thousands of sectors. Because I am diligent with backups, when my hard drive failed, I wanted to read it back to save the 2 hours needed to recreate some workspaces. A couple of missing files would not have hurt. So I came up with this command to read back everything, but without trying to hard.
Note that I specify disks by ID which contains the serial number. I **never** use `/dev/sdx` because it's too easy to make a mistake and overwrite something important.
CAUTION: I think I actually wrote this and never tested it.
```
dd iflag=fullblock conv=sync,noerror bs=1M if=/dev/disk/by-id/ata-Hitachi_xxx of=/dev/disk/by-id/ata-Hitachi_yyy
```
### Trace file access
Applications break in thousand ways. Sometimes they crash outright, or don't pick up the correct configuration, but they won't tell you what they were actually expecting. For those cases, I take them on the *strace* dissection table and observe their calls *in vivo*.
Here's the command that will tell you everything about how an application uses files. Most useful for figuring out where it attempts to look for configuration files, or dynamic libraries (the `LD_LIBRARY_PATH` env var), or pkgconfig instructions (`PKG_CONFIG_DIR`). One of my most used recipes.
```
strace -fe trace=access,creat,open,openat,unlink,unlinkat --decode-fds=all
```
### Socat
*Socat* is a great tool to translate streams from one form to another. It can listen or connect, convert between Unix pipes or TCP sockets. Here's a basic forwarder from localhost to another:
```
socat TCP-LISTEN:8000,fork TCP:192.168.4.4:8000
```
### Git change authors
There are reasons you might forget to set your git identity. A new computer, or a computer where you want to work under multiple identities. You might realize that you had used the wrong email for this repository. Here's a recipe which alters the history of any repository by replacing your name on all commits:
```
git filter-repo --name-callback 'return b"dcz"' --email-callback 'return b"dcz@example.com"'
```
### Git change identity
Here's a related one. If you have different identities, you might have different ssh keypairs for each. Here's how you set a custom keypair for the current repository. Don't forget to ssh-add.
**WARNING**: Opsec is hard. I think SSH will still disclose your other identities to the server when connecting.
```
git config --local core.sshcommand "ssh -i ~/.ssh/id_rsa.dcz -F /dev/null"
```
### Git serve
When you want to share your local git repository to the wide world, you need git-daemon installed, and the following entry in the .git/config file:
```
[alias]
serve = !git daemon --reuseaddr --verbose --base-path=. --export-all ./.git
```
### GPT partitioning tool
This is the simplest recipe I have, but not the least useful.
```
sgdisk
```
### Search for text files
The *grep* program will by default search for the given string in all files, including binary files. This is almost never what I want: when I look for text, I want to find text, and including binaries slows the search for nothing. Even worse, even when my string exists in the binary file, all *grep* says is "binary file matched", dropping any context. Completely useless. I use this to search only in text files:
```
grep -r -I text dir
```
### Warm-plug of SATA drives
You can plug in SATA hard drives when the computer is running. But they will not always get detected. This is how you poke the bus to try to find newcomers (try host0 through host3):
```
echo '- - -' > /sys/class/scsi_host/host0/scan
```
Then you can kick out the drive again by doing:
```
udisks --detach /dev/disk/by-id/ata-xxxx
```
and unplug it.
### Show most recent messages in Gajim
```
sqlite3 ./.local/share/gajim/logs.db
select * from logs where message != '' order by time desc limit 10;
```
### Show network connections in real time
It works also on OpenWRT. Needs tcpdump on the remote host and etherape on the host which inspects the traffic.
```
ssh root@192.168.5.1 tcpdump -n -i br-lan -w - not port 22 | etherape -m ip -r -
```
### Nmap ping scan
Check who's up on the local network. A regular scan with nmap takes too much time.
```
nmap -sn -PE 192.168.5.0/24
```
### Graphical podman
If you want to sandbox games in your podman, it's possible with some tricks:
Wayland:
```
podman run -e XDG_RUNTIME_DIR=/tmp -e WAYLAND_DISPLAY=wayland-0 --security-opt label=disable -v $XDG_RUNTIME_DIR/wayland-0:/tmp/wayland-0:rw -ti debian:buster /bin/bash
```
X11:
```
podman run -ti -v /tmp/.X11-unix:/tmp/.X11-unix:rw --security-opt label=disable game_container
```
### Ripping a CD
I never actually tried this one. It's supposed to read all the data, and produce a more complete file than just ISO. ISO does not contain audio tracks, for example:
```
cdrdao read-cd --read-raw --read-subchan
```
### Touchpad
Ever played a shooter game on the touchpad? Well, I was once bored enough on the train… But I needed to to make it possible to aim and move first. I did it by not disabling the touchpad while the keyboard is in use. First, check the device number in `xinput list`, and then:
```
xinput set-prop 11 298 0
```
### Unrar
RAR files have an Libre decompressor, but it doesn't work on all files. I refuse to pollute my system with such tools, so I keep them contained in a container.
```
podman run --rm -v $PWD/unrar:/files:z maxcnunes/unrar:latest unrar e -or -r myfile.rar
```
### Listening to WOL packets
Nothing tricky in this one.
```
tcpdump -i enp9s0 'ether proto 0x0842'
```
### Widen
I don't use fixed width fonts in editors. Proportional fonts are mostly superior, but they have a downside. Sometimes "ASCII" diagrams are useful to quickly sketch an idea visually, and they are typically drawn assuming a fixed font on the reader end, so they appear as misaligned garbage when I see them.
Instead of assuming a fixed font, the authors could *ensure* fixed font. This is what this Python function does: it converts ASCII to fixed width, on the *character* level.
```
WIDE_MAP = {i: i + 0xFEE0 for i in range(0x21, 0x7F)}
WIDE_MAP[0x20] = 0x3000
def widen(s):
"""
Convert all ASCII characters to their full-width counterpart.
>>> print widen('test, Foo!')
ｔｅｓｔ， Ｆｏｏ！
>>>
"""
return s.translate(WIDE_MAP)
```
## Video tricks
Ffmpeg is so versatile that it warrants an entire section.
### Keyframes
Dropping all frames from the video except for keyframes results in a movie at normal speed but a very low FPS and lower size.
For video streams encoded in h264:
```
ffmpeg -i test.mkv -c copy -map v -bsf:v "filter_units=remove_types=1" key.mkv
```
This one is h265, I think:
```
ffmpeg -i IN.MOV -c:v copy -c:a copy -bsf:v noise=drop='not(key)' out.mkv
```
### Metadata
To write all metadata (global, video, audio) to a file, use:
```
ffmpeg -i in.mp4 -c copy -map_metadata 0 -map_metadata:s:v 0:s:v -map_metadata:s:a 0:s:a -f ffmetadata in.txt
```
To add all metadata from a file, use:
```
ffmpeg -i in.mp4 -f ffmetadata -i in.txt -c copy -map_metadata 1 out.mp4
```
### Reencode as h265
Those settings worked for me to get good quality:
```
ffmpeg -i ./XXX.MP4 -acodec copy -vcodec libx265 -bufsize 2M -maxrate 1M ./20.mkv
```
### Extract the music
Losslessly. This commands works for music encoded using the AAC codec, but other codecs mean you have to use different output formats.
```
ffmpeg -i ../Music/movie -vn -acodec copy -bsf:a aac_adtstoasc out.m4a
```
### Replace the sound track
When your camera can't record the sound in a good enough quality and you have to use a separate sound recorder, you will end up with a video file and a music file. This is how to synchronize the times and merge them:
```
ffmpeg -itsoffset -00:00:59.8 -i an.wav -i XXX.MOV -ss 00:00:00 -c:a flac -c:v copy -map 0 -map 1 -disposition:a:1 none -map_metadata 1 out.mkv
```