Scantron

Overview

Scantron is a distributed nmap and masscan scanner comprised of two components. The first is a console node that consists of a web front end used for scheduling scans and storing scan targets and results. The second component is an engine that pulls scan jobs from the console and conducts the actual scanning. A majority of the application's logic is purposely placed on the console to make the engine(s) as "dumb" as possible. All scan target files and scan results reside on the console and are shared through a network file share (NFS) leveraging SSH tunnels. The engines call back to the console periodically using a REST API to check for scan tasks and provide scan status updates. There is also an option to generate nmap scan diffs emailed to you using the pyndiff library.

Checkout the Python Scantron API client for interacting with the Scantron API and driving automated workflows.

Scantron is coded for Python3.6+ exclusively and leverages Django for the web front-end, Django REST Framework as the API endpoint, PostgreSQL as the database, a Redis job queue for tasks, Postfix for email scan alerts, and comes complete with Ubuntu-focused Ansible playbooks for smooth deployments. Scantron has been tested on Ubuntu 18.04 and may be compatible with other operating systems. Scantron's inspiration comes from:

• dnmap
• Minions
• rainmap svn / rainmap github
• rainmap-lite

Scantron relies heavily on utilizing SSH port forwards (-R / -L) as an umbilical cord to the engines. Either an SSH connection from console --> engine or engine --> console is acceptable and may be required depending on different firewall rules, but tweaking the port forwards and autossh commands will be necessary. If you are unfamiliar with these concepts, there are some great overviews and tutorials out there:

• https://help.ubuntu.com/community/SSH/OpenSSH/PortForwarding
• https://www.systutorials.com/39648/port-forwarding-using-ssh-tunnel/
• https://www.everythingcli.org/ssh-tunnelling-for-fun-and-profit-autossh/

Use cases

Scantron is not engineered to be quickly deployed to a server to scan for a few minutes, then torn down and destroyed.
It's better suited for having a set of static scanners (e.g., "internal-scanner", "external-scanner") with a relatively static set of assets to scan. A Scantron API client is also available for creating, retrieving, updating, or deleting sites, scan commands, scans, etc.

Architecture Diagram

Hardware Requirements

• Engine: If you plan on compiling masscan on an engine, you'll need at least 1024 MB of memory. It fails to build with only 512 MB. If you do not want to build masscan, set install_masscan_on_engine to False in ansible-playbooks/group_vars/all.yml

• Console: 512 MB of memory was the smallest amount successfully tested, however, if you plan on processing large scan files (using the scripts found in console/scan_results: masscan_json_to_csv.py, nmap_to_csv.py or xml_to_json_nmap_results.py), you'll need more memory.

Ansible Deployment Server and Initial Setup

This is your local box, preferably Linux. Ansible >= 2.4.0.0 is the minimum version required for utilizing ufw comments.

Clone the project and execute initial_setup.sh.

# Clone scantron project.
git clone https://github.com/rackerlabs/scantron.git
cd scantron
./initial_setup.sh  # Run as non-root user.

Installation

Installation requires a general knowledge of Python, pip, and Ansible. Every attempt to make the deployment as simple as possible has been made.

Cloud Provider Caveats

NAT'd instances

If the console server is actually a RFC1918 IP and not the public IP (because of NAT), the NAT'd RFC1918 IP (e.g., 10.1.1.2) will have to be added to the ALLOWED_HOSTS in ansible-playbooks/roles/console/templates/production.py.j2

This is common in AWS and GCP environments.

IBM Cloud

Per https://github.com/0xtavian: For the Ansible workload to work on IBM Cloud, edit the file /boot/grub/menu.lst by changing

# groot=LABEL...

to

# groot=(hd0)

Update hosts

Edit the hosts in this file:

• ansible-playbooks/hosts.ini

Console Installation

The recommendation is to deploy the console first.

Update Console Ansible Variables

Edit any variables in ansible-playbooks/group_vars/all.yml before running playbook. Note the time zone variables:

• timezone_server - Set this to be the timezone you want the server to be in, usually UTC.
• timezone_django - Set this to be your local timezone. It makes dealing with dates, times, and scheduling easier.

If you plan on utilizing the same API key across all engines (not recommended, but easier for automated deployments), change utilize_static_api_token_across_engines to True. This prevents you from having to log into each engine and update engine_config.json with the corresponding API key. The group_vars/static_api_key will be created by the the console ansible playbook. The Ansible engine playbook will autofill the engine_config.json.j2 template with the API key found in group_vars/static_api_key.

WARNING: The engine_config.json.j2 will generate a random scan_engine (e.g., engine-847623), so if you deploy more than 1 engine, you won't run into complications with engine name collisions. You will, however, need to add create the user on the console, since the console returns scheduled jobs to the engine based off the engine's name!

Update Console Secrets Variables

Rename console/scantron_secrets.json.empty to console/scantron_secrets.json (should be done for you by initial_setup.sh)

Update all the values console/scantron_secrets.json if you do not like ones generated using initial_setup.sh. Only the production values are used.

• All Scantron Django passwords have a minimum password length of 12.

• For the "SECRET_KEY", per Django's documentation: The secret key must be a large random value and it must be kept secret.

Change scantron user password (optional)

The scantron operating system user password is not really leveraged and is populated by providing a salted hash of a random password generated using Python's passlib library. If you want to change the password, you will have to generate a hash for the desired password and update the temp_user_pass variable in scantron/ansible-playbooks/roles/add_users/vars/main.yml.

pip3 install passlib

python3 -c "from passlib.hash import sha512_crypt; import getpass; print(sha512_crypt.encrypt(getpass.getpass()))"

Execute Console Ansible Playbook

Ensure you have a SSH key (or username/password) to access the console box, specified by --private-key in the Ansible command. User must also have password-less sudo privileges.

cd ansible-playbooks

# non-root user with password-less sudo capabilities.
ansible-playbook console.yml -u ubuntu --become --private-key=<engine SSH key>

# root user.
ansible-playbook console.yml -u root --private-key=<engine SSH key>

Change Django user passwords with manage.py (optional)

cd into the console directory scantron/console and run the following to change the admin (or whatever user needs their password changed) user password.

python3 manage.py changepassword admin

Engine Installation

Update Engine Ansible Variables

Edit any variables in these files before running playbook:

• ansible-playbooks/group_vars/all.yml
• ansible-playbooks/roles/engine/vars/main.yml

Ensure proper user permissions

Ensure you have a SSH key (or username/password) to access the engine box, specified by --private-key in the Ansible command. The user must also have password-less sudo privileges. If you are creating the boxes on AWS, then the user is ubuntu for Ubuntu distros and the user already has password-less sudo capabilities. If you need to add password-less sudo capability to a user, create a /etc/sudoder.d/<USERNAME> file, where <USERNAME> is the actual user, and populate it with:

<USERNAME> ALL=(ALL) NOPASSWD: ALL

SSH-ing in as root will also work for the Ansible deployment, but is not generally recommended.

Execute Engine Ansible Playbook

cd ansible-playbooks

# non-root user with password-less sudo capabilities.
ansible-playbook engine.yml -u ubuntu --become --private-key=<engine SSH key>

# root user.
ansible-playbook engine.yml -u root --private-key=<engine SSH key>

Adding additional engines

A Scantron engine is synonymous with a user.

engines <--> users

Users / engines are added through the webapp, so once a user / engine is added, an API token is automatically generated for that user / engine. The user's / engine's password is not necessary for Scantron to function since all user / engine authentication is done using the API token. The username and password can be used to login to the webapp to test API functionality. More API testing information can be found in the Test Engine API section of this README.

Update /etc/rc.local with engine IPs for autossh

This is done automatically for one engine through Ansible. You may have to add additional lines and update SSH keys for each engine if they are different. These commands are for the console connecting to the engines.

In this example:

• console - 192.168.1.99
• engine1 - 192.168.1.100
• engine2 - 192.168.1.101

# console --> engine 1
su - autossh -s /bin/bash -c 'autossh -M 0 -f -N -o "StrictHostKeyChecking no" -o "ServerAliveInterval 60" \
    -o "ServerAliveCountMax 3" -p 22 -R 4430:127.0.0.1:443 -R 2049:127.0.0.1:2049 \
    -i /home/scantron/console/autossh.key autossh@192.168.1.100'

# console --> engine 2
su - autossh -s /bin/bash -c 'autossh -M 0 -f -N -o "StrictHostKeyChecking no" -o "ServerAliveInterval 60" \
    -o "ServerAliveCountMax 3" -p 22 -R 4430:127.0.0.1:443 -R 2049:127.0.0.1:2049 \
    -i /home/scantron/console/autossh.key autossh@192.168.1.101'

If the console cannot SSH to an engine, then the autossh command will be run on the engine and the port forwards will be local (-L) instead of remote (-R).

# console <-- engine 1
su - autossh -s /bin/bash -c 'autossh -M 0 -f -N -o "StrictHostKeyChecking no" -o "ServerAliveInterval 60" \
    -o "ServerAliveCountMax 3" -p 22 -L 4430:127.0.0.1:443 -L 2049:127.0.0.1:2049 \
    -i /home/scantron/console/autossh.key autossh@192.168.1.99'

Engines

Engine's engine_config.json

engine_config.json is a configuration file used by engines to provide basic settings and bootstrap communication with the console. Each engine can have a different configuration file.

The "api_token" will have to be modified on all the engines after deploying the co