Loading repository data…
prairie-guy / repository
A Python wrapper for GNU parallel that naturally embeds shell code into Python scripts or Jupyter notebooks (no delimiter hell). Features parameter substitution, environment substitution, and cross-product generation to eliminate shell loops. Perfect for integrating Unix programs into Python environments for bioinformatics and data science.
A transparent discovery signal based on current public GitHub metadata.
This score does not audit code, security, maintainers, documentation quality, or suitability. Verify the repository and its current documentation before adoption.
A Python wrapper for GNU parallel that naturally embeds shell code into Python scripts or Jupyter notebooks (no delimiter hell). Features parameter substitution, environment substitution, and cross-product generation to eliminate shell loops. Ideal for seamlessly integrating third-party Unix programs into Python environments for bioinformatics and data science pipelines.
Author: C. Bryan Daniels (quendor_at_nandor.net)
Before - Python script depending upon Juptyer magic !-command or Python subprocess function to execute shell or Unix code
%%time
in_paths = ['dedup_MT', 'dedup_2', 'dedup_human']
out_paths = ['conv_unconv3n_MT', 'conv_unconv3n_2', 'conv_unconv3n_human']
samples = ['E1', 'E2', 'E3', 'Z1', 'Z2', 'Z3', 'U1', 'U2', 'U3']
threads = 6
for in_path, out_path in zip(in_paths, out_paths):
ref = str(in_path).split('_')[-1]
for sample in samples:
!samtools view -e "rlen<100000" -h {fname(in_path,sample,'bam')} |\
hisat-3n-table -p {threads}--unique-only --alignments - --ref {get_ref(ref,'fa')} --output-name /dev/stdout --base-change C,T|\
bgzip -@ {nc} -c > {fname(out_path,sample,'tsv.gz')}
CPU times: user 1min 22s, sys: 20.5 s, total: 1min 43s
Wall time: 4h 33min 14s
After - parallel_zip using native shell code, portable to Jupyter or Python with parallelism built-in for speed
%%time
from parallel_zip import parallel_zip, Cross
parallel_zip(
"""
samtools view -e "rlen<100000" -h {in_path}/{sample}.bam | \
hisat-3n-table -p 6 --unique-only --alignments - --ref {get_ref(ref,'fa')} --output-name /dev/stdout --base-change C,T | \
bgzip -@ 6 -c > {out_path}/{sample}.tsv.gz
""",
in_paths = ['dedup_MT', 'dedup_2', 'dedup_human'] ,
out_paths= ['conv_unconv3n_MT', 'conv_unconv3n_2', 'conv_unconv3n_human']
cross=Cross(sample=['E1', 'E2', 'E3', 'Z1', 'Z2', 'Z3', 'U1', 'U2', 'U3']))
CPU times: user 103 ms, sys: 12.4 ms, total: 116 ms
Wall time: 41min 27s
What is executed - using dry_run=True
parallel_zip(
"""
samtools view -e "rlen<100000" -h {in_path}/{sample}.bam | \
hisat-3n-table -p 6 --unique-only --alignments - --ref {get_ref(ref,'fa')} --output-name /dev/stdout --base-change C,T | \
bgzip -@ 6 -c > {out_path}/{sample}.tsv.gz
""",
in_paths = ['dedup_MT', 'dedup_2', 'dedup_human'] ,
out_paths= ['conv_unconv3n_MT', 'conv_unconv3n_2', 'conv_unconv3n_human']
cross=Cross(sample=['E1', 'E2', 'E3', 'Z1', 'Z2', 'Z3', 'U1', 'U2', 'U3']),
dry_run=True)
Selected from shared topics, language and repository description—not editorial ratings.
Winterwind /
A wrapper for the CellNeighborEX python library (all credits go to Hyobin Kim) made as part of a greater project for analyzing MERFISH spatial transcriptomics data
8/100 healthsamtools view -e "rlen<100000" -h dedup_MT/E1.bam | hisat-3n-table -p 6 --unique-only --alignments - --ref ../../reference/fasta/human.fa --output-name /dev/stdout --base-change C,T | bgzip -@ 6 -c > conv_unconv3n_MT/E1.tsv.gz
samtools view -e "rlen<100000" -h dedup_MT/E2.bam | hisat-3n-table -p 6 --unique-only --alignments - --ref ../../reference/fasta/human.fa --output-name /dev/stdout --base-change C,T | bgzip -@ 6 -c > conv_unconv3n_MT/E2.tsv.gz
samtools view -e "rlen<100000" -h dedup_MT/E3.bam | hisat-3n-table -p 6 --unique-only --alignments - --ref ../../reference/fasta/human.fa --output-name /dev/stdout --base-change C,T | bgzip -@ 6 -c > conv_unconv3n_MT/E3.tsv.gz
.
. . . [27 Total] . . .
.
samtools view -e "rlen<100000" -h dedup_human/U3.bam | hisat-3n-table -p 6 --unique-only --alignments - --ref ../../reference/fasta/human.fa --output-name /dev/stdout --base-change C,T | bgzip -@ 6 -c > conv_unconv3n_human/U3.tsv.gz
Quick shell commands - with the pz() function:
from parallel_zip import pz
# Get file sizes
pz("ls -la *.txt")
# Returns: ['total 48', '-rw-r--r-- 1 user staff 156 Jan 15 data1.txt', ...]
# Quick data inspection
pz("head -3 data.csv")
# Returns: ['id,name,value', '1,apple,3.5', '2,banana,2.7']
# Complex shell command with natural shell syntax (no delimiting hell)
pz("""ls -l {os.getcwd()} | cut -f1 -d' ' | sed s/--// | sed s/^-// | awk '$1 ~ /x/ {split($1, parts, "-"); print parts[1]}' """)
# Returns: ['drwxrwxr', 'drwxrwxr', 'drwxrwxr', 'drwxrwxr']
pz()pz("command"){param} syntax instead of error-prone string concatenationpz() commands for one-off tasks and data exploration$, quotes, and special characters that are complicated within PythonPerfect for bioinformatics pipelines, data processing workflows, file operations, and any scenario where you need the power of shell tools with the convenience of Python.
GNU parallel is required and must be installed on your system:
Linux (Ubuntu/Debian)
sudo apt-get install parallel
Linux (CentOS/RHEL/Fedora)
sudo yum install parallel # CentOS/RHEL
sudo dnf install parallel # Fedora
macOS
brew install parallel
This is a single-file module. The simplest way to install:
# Clone and copy the module to your project
git clone https://github.com/prairie-guy/parallel_zip.git
cp parallel_zip/parallel_zip.py /path/to/your/project/
Optional: Install Locally with pip
git clone https://github.com/prairie-guy/parallel_zip.git
cd parallel_zip
pip install .
from parallel_zip import parallel_zip
# Process multiple files in parallel
parallel_zip("""wc -l sample_data/{file}""",
file=["data1.txt", "data2.txt", "data3.txt"],
verbose=True, lines=True)
# Returns:
['2 sample_data/data1.txt',
'14 sample_data/data2.txt',
'45 sample_data/data3.txt']
# Same files, different parameters - extract different numbers of lines
parallel_zip("""head -{num_lines} sample_data/{file}""",
file=["data1.txt", "data2.txt"],
num_lines=[2, 5],
verbose=True, lines=True)
# Returns:
['Sample data line one',
'This is line two',
'Line 1: Introduction',
'Line 2: Methods overview',
'Line 3: Data collection started',
'Line 4: Quality control passed',
'Line 5: Processing pipeline initialized ']
from parallel_zip import parallel_zip, Cross
# Test every file with every search pattern - 6 total combinations
parallel_zip("""grep -c '{pattern}' sample_data/{file}""",
file=["data1.txt", "data2.txt"],
cross=Cross(pattern=["line", "data", "Line"]),
verbose=True, lines=True)
# Returns: ['3', '1', '0', '1', '0', '15']
pz()from parallel_zip import pz
# One-liner for immediate results
pz("ls sample_data")
# Returns:
['data1.txt',
'data2.txt',
'data3.txt',
'data.csv',
'numbers.txt',
'sample1.txt',
'sample2.txt',
'server_logs.txt']
pz("wc -w sample_data/*.txt")
# Returns:
[' 11 sample_data/data1.txt',
' 70 sample_data/data2.txt',
' 148 sample_data/data3.txt',
' 8 sample_data/numbers.txt',
' 16 sample_data/sample1.txt',
' 16 sample_data/sample2.txt',
' 31 sample_data/server_logs.txt',
' 300 total']
pz("pwd")
# Returns: ['/home/quendor/stuff/parallel_zip']
The heart of parallel_zip is intuitive parameter substitution using {parameter} syntax in command templates.
Warning: The following parameter names are reserved and should not be used as named parameters to parallel_zip: command, cross, verbose, lines, dry_run, strict, java_memory. This is a known issue that will be fixed in a future version.
# Single parameter
parallel_zip("""cat sample_data/{filename}""", filename="data1.txt", verbose=True, lines=True)
# Returns: ['Sample data line one', 'This is line two', 'Final line three']
# Multiple parameters
parallel_zip("""head -{num} sample_data/{file} | tail -{last}""",
file="data2.txt", num=10, last=3, verbose=True, lines=True)
# Returns:
['Line 8: Statistical analysis begun',
'Line 9: Results compilation phase',
'Line 10: Visualization generated']
# Parameters are zipped together (like Python's zip function)
parallel_zip("""echo 'File {file} has {line_count} lines'""",
file=["data1.txt", "data2.txt", "data3.txt"],
line_count=[3, 15, 42], verbose=True, lines=True)
# Returns:
['File data1.txt has 3 lines',
'File data2.txt has 15 lines',
'File data3.txt has 42 lines']
# Real file processing - multi-step workflow
parallel_zip("""
wc -w sample_data/{input} > sample_data/{output}
cat sample_data/{output}
rm sample_data/{output}
""",
input=["data1.txt", "data2.txt"],
output=["count1.txt", "count2.txt"],
verbose=True, lines=True)
# Returns: ['11 sample_data/data1.txt', '70 sample_data/data2.txt']
# Single values automatically broadcast to match list length
parallel_zip("""grep '{pattern}' sample_data/{file}""",
file=["data1.txt", "data2.txt", "server_logs.txt"],
pattern="line", dry_run=True)
# Returns:
["grep 'line' sample_data/data1.txt",
"grep 'line' sample_data/data2.txt",
"grep 'line' sample_data/server_logs.txt"]
# Embed Python e