Droplet Video Analyzer

A python program to implement the video analysis techniques described in the paper Low-cost measurement of facemask efficacy for filtering expelled droplets during speech by Emma P. Fischer, Martin C. Fischer, David Grass, Isaac Henrion, Warren S. Warren and Eric Westman of Duke University.

This is a first pass at a script to extract droplet counts and sizes from video files created with the laser testing device described in the paper. The implementation is based on the description in the paper. For testing, I've used the video files published by the Duke team.

Installing / Getting started

The project is still in the early stages, and is not set up as a python package. I've included a conda environment specification to bootstrap a working environment quickly.

All development so far has been on OSX Mojave 10.14; nothing here has been tested on Windows or Linux.

If you don't already have Conda installed, you can find an OSX installer package for the Miniconda package manager here: https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-x86_64.pkg

(Other builds are here: https://repo.anaconda.com/miniconda/)

With Miniconda installed, in a terminal window, go to the destination directory you've chosen for the code and clone the repository:

git clone https://github.com/rlevine/droplet_video_analyzer.git

Go to the top level directory of the cloned code:

cd droplet_video_analyzer

If you're running on OSX, I've provided a slightly easier way to get started.

OSX

You should see spec-file.txt in the top-level directory of the distribution. that directory. Use conda to download the set of packages needed by the virtual runtime environment for the script:

conda create --prefix ./envs --file spec-file.txt

It will ask you to confirm overwriting the empty envs directory. Do so, and then activate the conda virtual environment from the that directory:

conda activate ./envs

Note that this runtime environment is isolated from any other python environment you have on your computer, and will run packages and executables from the envs directory. Any other packages you install with conda or pip will land in the envs directory tree. At any time, when you're done working with this code, you can exit the conda virtual environment by typing

conda deactivate

Any other OS

If you're running on Windows or Linux, you'll have to install packages for your OS. (And again, this code has only been tested on OSX 10.14.)

First, create and activate a conda environment for the install. (This uses python 3.7, as I haven't found a widely-distributed build of opencv 3.4 for python 3.8.)

conda create --name maskvideo python=3.7.9
conda activate maskvideo

Then, install these packages:

conda install -c conda-forge numpy
conda install -c conda-forge opencv=3.4.2
conda install -c conda-forge ffmpeg=4.0.2
conda install -c conda-forge ffmpeg-python Pillow scipy scikit-image prettytable ansi2html

By default, the conda forge opencv build for OSX brings in ffmpeg 4.0, which doesn't include H.264 video encoder support, so I'm forcing an update to ffmpeg 4.0.2. An H.264 codec is needed for writing and converting video files. opencv 4 seems to have problems with positioning large video preview windows, so I'm using opencv 3.4.2.

As with the OSX instructions, above, running conda deactivate will "turn off" the selected conda virtual environment.

Basic usage

You can run the script from the top directory of the distribution. For instance, to analyze the test video file included in the distribution:

./dva sample_files/Fleece_04.mp4

The script will create the directory output in the same directory as the source file, containing an annotated video file and a .csv file with droplet data from the analysis.

./dva sample_files/Fleece_04_40_frame_test.mp4 --show-video

Adding the --show-video flag will turn on an interactive video window. (Fleece_04_40_frame_test.mp4 and Fleece_04_10_frame_test.mp4 are very short video files for testing and experimentation)

./dva sample_files/Fleece_04_40_frame_test.mp4 --show-video --not-interactive

Specifying --not-interactive will advance through frames in the video window without requiring a keypress to advance frames.

When running the above examples, an annotated video file and .csv data file will be created in the sample_files/output directory in the file tree; the -o/--output flag allows you to set an alternate destination directory for the output.

For convenience in running the script, until there's a proper package for it, add the distribution directory to your shell path, or add a symbolic link to a directory already in your path, such as /usr/local/bin.

Note that the samples files are drawn from the data published by the research team at Duke University, and are covered by a a Creative Commons license. More information can be found in the file accompanying README file

Interactive Video Window and Annotated Video

The video window displays all discovered droplets in a frame. In default interactive mode:

• Pressing one number key or two number keys in quick succession will advance that many frames.
• back arrow, < or , will move one frame backwards (Note: this is disabled when capturing video.)
• c will capture a .png image of the frame to disk.
• + or - will increase or decrease the detection threshold used to isolate droplets, to aid in finding the best threshold for a video. (This is also disabled when capturing video.)
• Pressing any other key will advance one frame.

Droplets are assigned unique numbers during the detection process. Their contours are shown in white, with a red bounding box, and pixel size is annotated. Droplets discovered to be repeat appearances from a prior frame show the prior "parent" droplet number and their unique ID is show in yellow. The history trail of repeat droplets is also shown in yellow.

Each frame includes a cumulative graph showing droplets per frame, cumulative droplets detected, and a synchronized audio graph. Note that the graph currently shows all droplets detected, before winnowing for duplicates.

Command Line Parameters

Input/Output

  <file name> [<file name> ...], --file <file name> [<file name> ...]
                        video files to analyze, either absolute or relative
                        paths, with `.` or  `..`` and wildcards.
  -o <output directory>, --output-dir <output directory>
                        directory for all file output (optional; default will
                        create "output" in video source dir)
  -i <input directory>, --input-dir <input directory>
                        video source directory (optional; default is user's
                        home directory)

Input and output directories and source file names are specified separately in the interface, with defaults. Typically, your shell will require file specs with wildcards to be enclosed in double quotes.

Droplet Detection

  -t <detection threshold>, --threshold <detection threshold>
                        droplet detection threshold; default=62

The analysis process converts each video frame to grayscale, and then thresholds each image, ignoring each pixel that isn't brighter than the supplied threshold. This allows us to ignore background light scatter and other light artifacts that might not be droplets. The threshold value is an 8-bit pixel brigtness, ie from 0 to 255. Any pixels dimmer than this value will be ignored.

  -b <border width>, --border <border width>
                        width of border region of frame to ignore, in pixels;
                        default=20

We can ignore detections at the edges if a video frame, if there are droplets or dust particles that remain in circulation because of airflow or other reasons.

  --droplet-similarity <similarity threshold>
                        droplet similarity threshold; smaller is more similar;
                        default=30
  --distance-threshold <distance threshold>
                        absolute distance threshold; greater than overrides
                        similarity; default=40
  --frame-history <frame history>
                        number of frames to consider for prior droplet
                        similarity; default=1

The script uses three factors to try and determine if a droplet detection in a given frame is a new droplet, or another sighting of an already-reported droplet in a prior frame.

--distance-threshold is the maximum distance between a pair of droplet centroids for them to be considered as duplicates.

In addition, I attempt to use Hu moment invariants calculated from the shape of each droplet to identify duplicate droplets. This technique becomes less effective as droplet size decreases.

Multiplying droplet distance by the calculated moment similarity yields a confidence factor. --droplet-similarity is the upper limit for that factor for droplets in a frame series to be considered duplicates.

--frame-history is the number of prior frames to be examined for matches to droplets in the current frame. I've found that looking at more than the most immediate prior frame (--frame-history of 1) yields too many false positives in many circumstances.

  --top-10              generate image files for the top 10 frames by droplet

--top-10 was implemented as way to capture .png images of the frames with the highest numbers of droplets.

Output Options

  --no-csv              don't create a .csv data file
  -c, --no-video-output
                        don't create a new video file with annotation
  -l, --capture-log     create an HTML log file
  -q, --quiet           suppress console window output

By default, the script will save an annotated video file with droplets identified and a graph of droplet detections over time, as well as a .csv file with data on all detected droplets for analysis. In addition, the script will write extensive progress information to the shell it's running in.

Progress output and video and .csv file generation can be turned off and creation of a detailed log file can be turned on.

  --show-video          show video preview image while processing file
  -n, --not-interactive
                        do not require keyboard interaction to advance to next
                        frame

The script can display an interactive video window showing the progress of droplet detection, frame by frame. By default, you'll need to press keyboard keys to advance th