The function took 5.04 s to compute.
│ │ │ │ +The function took 5.05 s to compute.
│ │ │ │
│ │ │ │ The transformed data are:
│ │ │ │ [ 0.49671415 -0.1382643 0.64768854 1.52302986 -0.23415337 -0.23413696
│ │ │ │ 1.57921282 0.76743473 -0.46947439 0.54256004]
│ │ │ │
│ │ │ │
│ │ │ │ At the first call, the results will be cached. Therefore, the computation
│ │ │ │ @@ -125,15 +125,15 @@
│ │ │ │ data_trans = costly_compute_cached(data)
│ │ │ │ end = time.time()
│ │ │ │
│ │ │ │ print('\nThe function took {:.2f} s to compute.'.format(end - start))
│ │ │ │ print('\nThe transformed data are:\n {}'.format(data_trans))
│ │ │ │
The function took 0.01 s to compute.
│ │ │ │ +The function took 0.02 s to compute.
│ │ │ │
│ │ │ │ The transformed data are:
│ │ │ │ [ 0.49671415 -0.1382643 0.64768854 1.52302986 -0.23415337 -0.23413696
│ │ │ │ 1.57921282 0.76743473 -0.46947439 0.54256004]
│ │ │ │
│ │ │ │
│ │ │ │ At the second call, the computation time is largely reduced since the results
│ │ │ │ @@ -168,45 +168,45 @@
│ │ │ │ data_trans = transformer.transform(data)
│ │ │ │ end = time.time()
│ │ │ │
│ │ │ │ print('\nThe function took {:.2f} s to compute.'.format(end - start))
│ │ │ │ print('\nThe transformed data are:\n {}'.format(data_trans))
│ │ │ │
The function took 5.08 s to compute.
│ │ │ │ +The function took 5.05 s to compute.
│ │ │ │
│ │ │ │ The transformed data are:
│ │ │ │ [ 0.49671415 -0.1382643 0.64768854 1.52302986 -0.23415337 -0.23413696
│ │ │ │ 1.57921282 0.76743473 -0.46947439 0.54256004]
│ │ │ │
│ │ │ │
│ │ │ │ start = time.time()
│ │ │ │ data_trans = transformer.transform(data)
│ │ │ │ end = time.time()
│ │ │ │
│ │ │ │ print('\nThe function took {:.2f} s to compute.'.format(end - start))
│ │ │ │ print('\nThe transformed data are:\n {}'.format(data_trans))
│ │ │ │
│ │ │ │
│ │ │ │ -The function took 0.04 s to compute.
│ │ │ │ +The function took 0.02 s to compute.
│ │ │ │
│ │ │ │ The transformed data are:
│ │ │ │ [ 0.49671415 -0.1382643 0.64768854 1.52302986 -0.23415337 -0.23413696
│ │ │ │ 1.57921282 0.76743473 -0.46947439 0.54256004]
│ │ │ │
│ │ │ │
│ │ │ │ As expected, the second call to the transform method load the results
│ │ │ │ which have been cached.
│ │ │ │
│ │ │ │
│ │ │ │ Clean up cache directory¶
│ │ │ │ memory.clear(warn=False)
│ │ │ │
│ │ │ │
│ │ │ │ -Total running time of the script: (0 minutes 15.272 seconds)
│ │ │ │ +Total running time of the script: (0 minutes 15.174 seconds)
│ │ │ │
│ │ │ │
│ │ │ │ First round - caching the data
│ │ │ │ -Elapsed time for the entire processing: 4.52 s
│ │ │ │ +Elapsed time for the entire processing: 4.33 s
│ │ │ │
│ │ │ │
│ │ │ │ By using 2 workers, the parallel processing gives a x2 speed-up compared to
│ │ │ │ the sequential case. By executing again the same process, the intermediate
│ │ │ │ results obtained by calling costly_compute_cached will be loaded from the
│ │ │ │ cache instead of executing the function.
│ │ │ │
│ │ │ │
│ │ │ │ Second round - reloading from the cache
│ │ │ │ -Elapsed time for the entire processing: 0.02 s
│ │ │ │ +Elapsed time for the entire processing: 0.01 s
│ │ │ │
│ │ │ │
│ │ │ │
│ │ │ │
│ │ │ │ Reuse intermediate checkpoints¶
│ │ │ │
│ │ │ │
│ │ │ │
│ │ │ │ Reusing intermediate checkpoints
│ │ │ │ -Elapsed time for the entire processing: 0.03 s
│ │ │ │ +Elapsed time for the entire processing: 0.01 s
│ │ │ │
│ │ │ │
│ │ │ │ The processing time only corresponds to the execution of the max
│ │ │ │ function. The internal call to costly_compute_cached is reloading the
│ │ │ │ results from the cache.
│ │ │ │
│ │ │ │
│ │ │ │ Clean-up the cache folder¶
│ │ │ │ memory.clear(warn=False)
│ │ │ │
│ │ │ │
│ │ │ │ -Total running time of the script: (0 minutes 12.596 seconds)
│ │ │ │ +Total running time of the script: (0 minutes 12.360 seconds)
│ │ │ │
│ │ │ │ Running tasks with return_as='list'...
│ │ │ │ Accumulate results:......................................................................................................................................................
│ │ │ │ All tasks completed and reduced successfully.
│ │ │ │ -Peak memory usage: 2.44GB
│ │ │ │ +Peak memory usage: 2.49GB
│ │ │ │ If we use return_as="generator", res is simply a generator on the
│ │ │ │ results that are ready. Here we consume the results as soon as they arrive
│ │ │ │ with the accumulator_sum and once they have been used, they are collected
│ │ │ │ by the gc. The memory footprint is thus reduced, typically around 300MB.
monitor_gen = MemoryMonitor()
│ │ │ │ @@ -169,15 +169,15 @@
│ │ │ │ peak = max(monitor_gen.memory_buffer) / 1e6
│ │ │ │ print(f"Peak memory usage: {peak:.2f}MB")
│ │ │ │ Create result generator with return_as='generator'...
│ │ │ │ Accumulate results:......................................................................................................................................................
│ │ │ │ All tasks completed and reduced successfully.
│ │ │ │ -Peak memory usage: 80.59MB
│ │ │ │ +Peak memory usage: 269.88MB
│ │ │ │ We can then report the memory usage accross time of the two runs using the │ │ │ │ MemoryMonitor.
│ │ │ │In the first case, as the results accumulate in res, the memory grows
│ │ │ │ linearly and it is freed once the accumulator_sum function finishes.
In the second case, the results are processed by the accumulator as soon as │ │ │ │ @@ -200,15 +200,15 @@ │ │ │ │ plt.show() │ │ │ │
It is important to note that with return_as="generator", the results are
│ │ │ │ still accumulated in RAM after computation. But as we asynchronously process
│ │ │ │ them, they can be freed sooner. However, if the generator is not consumed
│ │ │ │ the memory still grows linearly.
Total running time of the script: (0 minutes 54.398 seconds)
│ │ │ │ +Total running time of the script: (0 minutes 23.428 seconds)
│ │ │ │