Miscellaneous short utilities #############################

:tags: python, shell :author: Roland Smith

.. Last modified: 2024-02-24T15:45:09+0100

Introduction

This is a collection of small utilities that I've written over the years. Some of them are simple front-ends for a utility with some standard options, to save me from having to recall the options every time I need them.

Given that there are a number of different programs in this repository, releases or packages for this repo should not be expected. Just clone the repo or download the zipfile from the main branch.

Another portion are basically Python_ front-ends to run a utility in parallel on different files.

.. _Python: http://www.python.org/

All the functions in the python scripts come with documentation strings to explain what they do. The shell scripts have comments where necessary. They use basic sh syntax and to not use bash extensions.

All these programs are tested and in use on the FreeBSD operating system. The shell-scripts use the plain old sh that comes with FreeBSD, but should work with bash. Bug reports and patches welcome. Most of it should work on other BSD systems, Linux or OS-X without major problems. Scripts that require FreeBSD are noted as such.

All Python scripts use Python 3 specific features. Most recently I converted str.format calls to f-strings, meaning that you'll need Python 3.6 or later for the Python scripts.

Note that these scripts assume that Python 3 is installed as python.

For the Python code:

.. image:: https://img.shields.io/badge/code%20style-black-000000.svg :target: https://github.com/psf/black

Running tests

Tests for some of the functions used in the Python scripts are contained in scripts-tests.py. Running the tests requires pytest. Running the tests is done as follows::

pytest scripts-tests.py

Tests for security issues of the Python scripts can be done with bandit. I've run the tests as follows::

bandit -s B404 -x scripts-tests.py *.py | less

One might consider adding B603 and B607 to the exclusion list when the use of subprocess calls has been audited. These are:

• B603 subprocess_without_shell_equals_true
• B607 start_process_with_partial_path

When run with::

bandit -s B404,B603,B607 -x scripts-tests.py *.py

The file scripts-tests.py is excluded because it contains lots of assert calls as part of the testing mechanism. In this case, these are no cause for alarm.

The result should be: No issues identified.

Further general testing of Python scripts is done with pylama. The aim is to have no warnings or errors.

The programs

all-git.py

This script traverses all the directories under the current directory. If it finds a directory that is managed by git_, it runs git log to get the time and hash of the latest commit. This is then printed followed by the directory name.

.. _git: http://git-scm.com/

blocklist.py

This script generates configuration data for dnsmasq_ and unbound_ to block facebook and ad networks.

.. _dnsmasq: http://www.thekelleys.org.uk/dnsmasq/doc.html .. _unbound: https://nlnetlabs.nl/projects/unbound/about/

clean.sh

This script removes several types of generated files from the directory it is called from.

csv2tbl.py

Convert a CSV file to a LaTeX table.

default_options.py

This script generates a list of names of installed packages for which the options on the related port are equal to the default options.

denylog.py

This script reads /var/log/security or any other file that contains ipfw_ log messages, and makes an overview of incoming packages that have been logged.

.. _ipfw: https://www.freebsd.org/doc/en/books/handbook/firewalls-ipfw.html

This of course requires that blocked packets are logged!

If you are writing your own firewall script, make sure to use deny log instead of just deny.

dicom2jpg.py

A modification of the (pre 2023.12.31) dicom2png program mentioned below to produce JPEG output. This is meant for situaties where lossy compression is acceptable. For ms-windows users, this version is recommended. Just make sure that the location where the ImageMagick programs are installed is part of your PATH environment variable.

dicom2png.py

Convert DICOM_ files from an x-ray machine to PNG format.

.. _DICOM: http://en.wikipedia.org/wiki/DICOM

As of version 2023.12.31, this script uses pydicom_. Older versions used the `convert`` program from ImageMagick_.

.. _pydicom: https://pydicom.github.io/

Multiple images are processed in parallel using a ProcessPoolExecutor from the concurrent.futures module to start subprocesses using as many worker processes as your CPU has cores. This number is determined by the os.cpu_count function, so this program requires at least Python 3.4.

This version is recommended for ms-windows users.

dicominfo.py

This program uses pydicom_ to extract and print information about DICOM files.

.. _pydicom: https://pydicom.github.io/

dvd2webm.py

(Note that I don't use this much anymore. These days I tend to use Handbrake for my encoding needs.)

When I buy DVDs, I generally transfer their contents to my computer for easier viewing. However, the video and audio format used on DVD is not very compact. So I tend to use ffmpeg_ to convert it to smaller formats without losing quality. As of 2016, my favorite storage format is a webm_ container with a VP9_ video stream and vorbis_ audio.

.. _VP9: https://en.wikipedia.org/wiki/VP9

Initially I used the simple webm.sh script mentioned below. This had some shortcomings. It does not crop the video and cannot incorporate subtitles. It does enable multiple quality setting, but I seldomly used those.

The dvd2webm.py script performs a 2-pass encoding in constrained quality_ mode. Optionally it also adds subtitles to the video, and starts from an offset.

.. _constrained quality: http://wiki.webmproject.org/ffmpeg/vp9-encoding-guide

eps2png.sh

This script uses ghostscript_ to render encapsulated PostScript files to PNG format. Using command-line arguments the resolution and the type of PNG file can be changed.

.. _ghostscript: http://www.ghostscript.com/

epubinfo

Extract and print the metadata from an epub_ file, a standardized_ form of book readable on a computer, tablet or other compatible device.

The metadata is contained in a file traditionally named content.opf, which might or might not be located in the root directory of the zip-file that is an epub.

.. _epub: https://en.wikipedia.org/wiki/EPUB .. _standardized: https://www.w3.org/publishing/epub32/

ffmutt.sh

Small helper script to start mutt_ in an urxvt_ terminal for a mailto link.

.. _mutt: http://www.mutt.org/ .. _urxvt: http://software.schmorp.de/pkg/rxvt-unicode.html

find-modified.sh

Front-end for find_ to locate all files under the current directory that have been modified up to a given number of days ago.

.. _find: https://www.freebsd.org/cgi/man.cgi?query=find

find-pkg-upgrades.py

Script for FreeBSD to compare the versions of locally installed packages to the versions available from the FreeBSD package repo mirror_. It will tell you which packages can be upgraded via pkg upgrade, and which have to be built from source.

.. _package repo mirror: http://pkg.freebsd.org/

fixbb.sh

Corrects the BoundingBox for single-page PostScript_ documents. It requires the ghostscript_ program.

.. _PostScript: http://en.wikipedia.org/wiki/PostScript

fixfn.py

Fix filenames by replacing whitespace, converting to lower case and removing trailing IDs.

foto4lb.py

Scales fotos for including them into LaTeX documents. The standard configuration sets the width to 886 pixels and sets the resolution to 300 dpi. This gives an image 75 mm (about 3 in) wide.

foto4lb-wand.py +++++++++++++++

This uses and requires the wand_ binding for ImageMagick.

.. _wand: https://docs.wand-py.org/en/0.6.7/

In my (limited) testing with Wand 0.6.7 it was slightly faster than using convert from Python with subprocess. It is definitely more Pythonic.

genbackup.sh

Generates a backup of the directory it is called from in the form of a tar-file. The name of the backup file generally consists of;

• the word backup,
• the date in the form YYYYMMDD,
• the short hash-tag if the directory is managed by git_.

These parts are separated by dashes, and the file gets the .tar extension. It requires the tar program. Tested with FreeBSD's tar. Should work with GNU tar as long as you don't use the -x option; the exclude syntax is different between BSD tar and GNU tar.

genotp.py

Generates an old-fashioned one-time pad; 65 lines of 12 groups of 5 random capital letters. Each pad has a header line containing a random identifier. It was inspired by reading Neal Stephenson's Cryptonomicon.

It uses random numbers from the operating system via Python's os.urandom function.

A partial example::

+++++ KWSNKYJLFF +++++
01  WAGGB HJVHQ TTQPD LQUMD KFRFS GGCKA SVLLA WEUCS HTXNI DITNW RBZKM SEGGW
02  GDSBB XECBL AUVLQ TUDPO DTXKW MWGAV DLRXT NRYAH HTGII YXEJJ JLNRC BIVDX
03  JDQUJ QPAUT CUEHN RHIHT QYBGV WOVAQ MKVZQ WPRGL QJAVA RPLRS AXIII FKLEP
04  WXYAD JNSAQ LBRXE QLCUX ZCLIE WPHSO OZBNH ZQLVN FAUEZ IDAJY VPQJN WVCAD
05  BEYRE WORKU CPEGE JKKWZ XUVYU WSZXQ NOULH QOFDQ PREMG YJBIT GMOAM USKLV
06  ZVATP YSRWH EEQDV LIPVQ FVYSY CIICG JKMOA RFJYE RUDJG HHJXI NNPNU VERMN
07  WAHFD WGGGN GHIUM BCJNN CVBCK QXYGZ PEYLW XOGMT SJFQJ NWEBE BFBPJ IDHDB
08  NPPEG HNONE YCJTG BFSFA NFYUR CMCGD XSKRO NSRBX WSDDX MEMLX BBMLC IMDJL
09  PZNAK OCOXA PEGNL UAWQW YCVDM WBNZZ YQICH MTLBG LDQTW TQMCS KUYBN RUNXT
...

Testing /dev/random on FreeBSD ++++++++++++++++++++++++++++++

My impression is that the random data device on FreeBSD is pretty good;

.. code-block:: console

> ./ent -u
ent --  Calculate entropy of file.  Call
        with ent [options] [input-file]

        Options:   -b   Treat input as a stream of bits
                -c   Print occurrence counts
                -f   Fold upper to lower case letters
                -t   Terse output in CSV format
                -u   Print this message

By John Walker
http://www.fourmilab.ch/
January 28th, 2008

> dd if=/dev/random of=rdata.bin bs=1K count=1K
1024+0 records in
1024+0 records out
1048576 bytes transferred in 0.086200 secs (12164455 bytes/sec)

> ./ent rdata.bin
Entropy = 7.999857 bits per byte.

Optimum compression would reduce the size
of this 1048576 byte file by 0 percent.

Chi square distribution for 1048576 samples is 208.12, and randomly
would exceed this value 98.57 percent of the times.

Arithmetic mean value of data bytes is 127.5057 (127.5 = random).
Monte Carlo value for Pi is 3.137043522 (error 0.14 percent).
Serial correlation coefficient is 0.000771 (totally uncorrelated = 0.0).

According to the manual_ page, Wikipedia_ and other_ sources I could find the FreeBSD random device is intended to provide cryptographically secure pseudorandom data.

.. _manual: https://www.freebsd.org/cgi/man.cgi?query=random&sektion=4 .. _Wikipedia: http://en.wikipedia.org/?title=/dev/random .. _other: http://www.2uo.de/myths-about-urandom/

Using the new secrets module in Python;

.. code-block:: console

> python
Python 3.9.9 (main, Dec 11 2021, 14:34:11) 
[Clang 12.0.1 (git@github.com:llvm/llvm-project.git llvmorg-12.0.1-0-gfed41342a on freebsd13
Type "help", "copyright", "credits" or "license" for more information.
>>> import secrets
>>> data = secrets.