Loading repository data…
Loading repository data…
radiantone / repository
A lightweight (serverless) native python parallel processing framework based on simple decorators and call graphs.
A transparent discovery signal based on current public GitHub metadata.
This score does not audit code, security, maintainers, documentation quality, or suitability. Verify the repository and its current documentation before adoption.
This version: 0.2.3
Current development version is here: 0.2.4
A lightweight (serverless) native python parallel processing framework based on simple decorators and call graphs, supporting both control flow and dataflow execution paradigms as well as de-centralized CPU & GPU scheduling.
For a quick look at what makes Entangle special, take a look at Design Goals.
With Entangle you can run simple, hardware parallelized code with conditional logic that looks like this.
result = add(
add(
num(6),
two() if False else one()
),
subtract(
five(),
two()
)
)
print(result())
or train two AI models in parallel using tensorflow container utilizing dedicated CPU and GPU usage.
@process
@docker(image="tensorflow/tensorflow:latest-gpu", packages=['tensorflow_datasets'])
def train_modelA():
# train it
return
@process
@docker(image="tensorflow/tensorflow:latest-gpu", packages=['tensorflow_datasets'])
def train_modelB():
# train it
return
@workflow
def train_models(*args):
# I'm training a bunch of models in parallel!
return
workflow = train_models(
train_modelA(),
train_modelB()
)
result = workflow()
To quickly get started with Entangle, build and run a docker container from the included Dockerfile.
$ docker build -t entangle .
$ docker run -it entangle:latest
root@9579336b3e34:/# python -m entangle.examples.example
Or if you have the NVIDIA Docker Environment setup you can test the numba GPU vector example.
$ docker run -it --gpus all entangle
root@13428af4a37b:/# python -m entangle.examples.example3
(0.2957176749914652, 0.41134210501331836)
Entangle is a different kind of parallel compute framework for multi-CPU/GPU environments. It allows for simple workflow design using plain old python and special decorators that control the type of parallel compute and infrastructure needed.
One key feature of entangle is fine-grained control over individual functions in a workflow. You could easily describe multiple functions running across multiple compute environments all interacting as if they were simple local python functions. No central scheduler or workflow manager is needed allowing you to choose where and how functions operate with declarative infrastructure.
Another unique quality is the use of composition to build parallel workflows dynamically.
The term is derived from a quantum physics phenomena called quantum entanglement which involves the state of a pair or group of particles affecting one another at a distance instantaneously.
In this context, it is a metaphor for how tasks send data (particles) to one another in the context of a connected microflow.
Please keep in mind that Entangle is in development and is classified as Pre-Alpha. Some of the functionality shown here is incomplete. If you clone this repo and want to experiment be sure to update often as things break, improve, get fixed etc. quite frequently. The main branch will always contain the most current release. All development for the next version is done on the development branch for the next released listed at the top of this document.
NOTE: At the moment entangle only works with python 3.8 due to how coroutines work there and also shared memory features.
From PyPi
$ pip install --upgrade py-entangle
$ python -m entangle.examples.example
From repo root
python3.8
$ virtualenv --python=python3.8 venv
$ source venv/bin/activate
(venv) $ python setup.py install
(venv) $ python -m entangle.examples.example
miniconda3
$ conda init
$ python setup.py install
$ python -m entangle.examples.example
NOTE: Numba package is disabled by default in
setup.py. If you want this package, just uncomment it; however some OS specific steps might be required.
On some systems you might encounter the following error when trying to install numba.
RuntimeError: Could not find a `llvm-config` binary.
Try the following remedy (for ubuntu systems)
$ sudo apt-get install -y --no-install-recommends llvm-10 llvm-10-dev
$ export LLVM_CONFIG=/usr/bin/llvm-config-10
$ pip3 install numba
$ pytest --verbose --color=yes --disable-pytest-warnings --no-summary --pyargs entangle.tests
or if you don't have GPU
$ pytest --verbose --color=yes --pyargs entangle.tests.test_entangle
or just do this
$ python setup.py test
Clean all build files, directories, temp files and any files created by examples and tests.
$ python setup.py clean
If you are planning to run or use GPU enabled code it is recommended to set up a miniconda3 virtualenv.
Entangle is designed without a central scheduler or workflow manager. Rather, each function is decorated with special descriptors that turn them into their own workflow managers. These decorators implement logic to parallelize and gather values from its dependent arguments, which are executed as separate processes. As each function is assigned a dedicated CPU, the workflow is thus an ensemble of parallel, independent micro-flows that resolve themselves and pass their values into queues until the workflow completes.
This offers an extreme shared nothing design that maximizes CPU usage in a multi-CPU environment.
Each function (or task) is given a process and scheduled to a CPU by the operating system. Since python Processes are native bound OS processes, this inherits the benefit of the operating system scheduler which is optimized for the underlying hardware. Arguments that satisfy the function are run in parallel in the same fashion. The parent function then uses asyncio coroutines to monitor queues for the results from the processes. This keeps the CPU usage down while the dependent functions produce their results and eliminates the need for monitor threads.
As a workflow executes, it fans out over CPUs. Each process acting as it's own scheduler to spawn new processes and resolve arguments, while also monitoring queues for incoming results asynchronously. This makes the workflow a truly emergent, dynamic computing construct vs a monolithic service managing all the pieces. Of course, this is not to say one approach is better, just that entangle takes a different approach based on its preferred tradeoffs.
Every design approach is a balance of tradeoffs. Entangle favors CPU utilization and true parallelism over resource managers, centralized (which is to say network centric) schedulers or other shared services. It favors simplicity over behavior - leaving specific extensions to you, attempting to be minimal and un-opinionated. It tries to be invisible to the end user as much as possible. It strives for the basic principle that, "if it looks like it should work, it should work."
Entangle leans on the OS scheduler to prioritize processes based on the behavior of those processes and underlying resource utilizations. It therefore does not provide its own redundant (which is to say centralized) scheduler or task manager. Because of this, top-down visibility or control of workflow processes is not as easy as with centralized task managers.
Entangle prefers the non-API approach, where it looks like regular python expressions, over strict API's or invocation idioms. This makes it easier to pick up and use and plays well with 3rd party frameworks too.
Because of these tradeoffs, there are certain use cases that align with entangle and others that probably do not.
If you want top-down visibility & control of workflows and tasks, Entangle is probably not ready for you.
If you have lots of CPUs, entangle could be for you! If you want easy python workflows that span local and remote cloud resources, entangle could be for you. If you want to write custom handlers that enrich or execute code in custom ways for your needs, entangle makes this easy for you.
One focused use case for entangle is when you want to orchestrate across different compute nodes, remote APIs and other disparate endpoints in a single workflow, with inherent parallelism.
Each step of the workflow has different parameters, needs and protocols used to communicate with it. Such a workflow might simply look like:
data = data_refinement(
get_source_data()
)
result = measure_vectors(
vector1(
data("vector1")
),
vector2(
data("vector2")
),
vector3(
data("vector3")
)
)
Another use case is the need to run multiple parallel tasks that operate on matrix data using a GPU. Entangle makes this quite easy as seen in [GPU Example](#gpu-exam