thread_chunks

A python package for performing memory intensive computations in parallel using chunks and checkpointing.

Functions

chunk

Calls func in parallel, a chunk at a time, for each set of *args in parameters and returns the outputs in an ordered list.

Classes

Checkpoint	A data structure to store the progress of the parallel execution of a function func.
Chunker	A collection of RemoteLabelledActor s pre-loaded with a function to execute that can be reused.
LabelledActor	A ray actor that can execute the function func along with a label to allow an ordering of calls to be re-established.
RemoteLabelledActor	A remote ray actor that can execute the function func along with a label to allow an ordering of calls to be re-established.

Exceptions

CheckpointFailedWarning

Warns that a checkpoint failed to save.

Modules

config

The default configuration for the thread-chunks python package.