CheckpointingΒΆ
Typically, we are interested in parallelising computationally intensive tasks. For this reason it is important to checkpoint the progress of the task in case the program crashes or your computer unexpectedly shuts down. To achieve this we employ the saveable-objects package.
Specifically, if we parallelise the function
def f(x,y):
return x+y
using
from thread_chunks import chunk
outputs = chunk(f, [[1, 4], [2, 5], [3, 6]], path="checkpoint.pkl")
then as soon as each call to f completes chunk() saves the result to the pickle file checkpoint.pkl. If our computation crashes partway through we can simply call
from thread_chunks import chunk
outputs = chunk(f, [[1, 4], [2, 5], [3, 6]], path="checkpoint.pkl")
again and the computation will pick up where it left off. That is no call already executed will be repeated.
However, the checkpoint also tracks the function name and the parameters. Thus if our computation crashed halfway through and we then execute
from thread_chunks import chunk
outputs = chunk(g, [[1, 4], [2, 5], [3, 6]], path="checkpoint.pkl")
for some different function g or
from thread_chunks import chunk
outputs = chunk(f, [[0, 4], [2, 5], [3, 6]], path="checkpoint.pkl") # different parameters
or
from thread_chunks import chunk
outputs = chunk(f, [[1, 4], [2, 5], [3, 6], [3, 7]], path="checkpoint.pkl") # more parameters
or
from thread_chunks import chunk
outputs = chunk(f, [[1, 4], [2, 5]], path="checkpoint.pkl") # less parameters
or
from thread_chunks import chunk
outputs = chunk(f, [[2, 5], [1, 4], [3, 6]], path="checkpoint.pkl") # different order of parameters
then checkpoint.pkl will be overwritten and all function executions redone.
The checkpoint is an instance of Checkpoint and can be loaded to inspect the progress as follows:
from thread_chunks import Checkpoint
checkpoint = Checkpoint.load("checkpoint.pkl")
The output of chunk() is stored as a list in checkpoint.output. Uncompleted executions have the value None.
In the next chapter we will delve deeper into the actors that underlie thread-chunks.