The function took 5.09 s to compute.
│ │ │ │ +The function took 5.10 s to compute.
│ │ │ │
│ │ │ │ The transformed data are:
│ │ │ │ [ 0.49671415 -0.1382643 0.64768854 1.52302986 -0.23415337 -0.23413696
│ │ │ │ 1.57921282 0.76743473 -0.46947439 0.54256004]
│ │ │ │
│ │ │ │
│ │ │ │ At the first call, the results will be cached. Therefore, the computation
│ │ │ │ @@ -168,45 +168,45 @@
│ │ │ │ data_trans = transformer.transform(data)
│ │ │ │ end = time.time()
│ │ │ │
│ │ │ │ print('\nThe function took {:.2f} s to compute.'.format(end - start))
│ │ │ │ print('\nThe transformed data are:\n {}'.format(data_trans))
│ │ │ │
The function took 5.10 s to compute.
│ │ │ │ +The function took 5.11 s to compute.
│ │ │ │
│ │ │ │ The transformed data are:
│ │ │ │ [ 0.49671415 -0.1382643 0.64768854 1.52302986 -0.23415337 -0.23413696
│ │ │ │ 1.57921282 0.76743473 -0.46947439 0.54256004]
│ │ │ │
│ │ │ │
│ │ │ │ start = time.time()
│ │ │ │ data_trans = transformer.transform(data)
│ │ │ │ end = time.time()
│ │ │ │
│ │ │ │ print('\nThe function took {:.2f} s to compute.'.format(end - start))
│ │ │ │ print('\nThe transformed data are:\n {}'.format(data_trans))
│ │ │ │
│ │ │ │
│ │ │ │ -The function took 0.03 s to compute.
│ │ │ │ +The function took 0.05 s to compute.
│ │ │ │
│ │ │ │ The transformed data are:
│ │ │ │ [ 0.49671415 -0.1382643 0.64768854 1.52302986 -0.23415337 -0.23413696
│ │ │ │ 1.57921282 0.76743473 -0.46947439 0.54256004]
│ │ │ │
│ │ │ │
│ │ │ │ As expected, the second call to the transform method load the results
│ │ │ │ which have been cached.
│ │ │ │
│ │ │ │
│ │ │ │ Clean up cache directory¶
│ │ │ │ memory.clear(warn=False)
│ │ │ │
│ │ │ │
│ │ │ │ -Total running time of the script: (0 minutes 15.402 seconds)
│ │ │ │ +Total running time of the script: (0 minutes 15.461 seconds)
│ │ │ │
│ │ │ │
│ │ │ │ Sequential processing
│ │ │ │ -Elapsed time for the entire processing: 8.00 s
│ │ │ │ +Elapsed time for the entire processing: 8.02 s
│ │ │ │
│ │ │ │
│ │ │ │ costly_compute is expensive to compute and it is used as an intermediate
│ │ │ │ step in data_processing_mean. Therefore, it is interesting to store the
│ │ │ │ intermediate results from costly_compute using joblib.Memory.
│ │ │ │
│ │ │ │
│ │ │ │ First round - caching the data
│ │ │ │ -Elapsed time for the entire processing: 4.80 s
│ │ │ │ +Elapsed time for the entire processing: 5.21 s
│ │ │ │
│ │ │ │
│ │ │ │ By using 2 workers, the parallel processing gives a x2 speed-up compared to
│ │ │ │ the sequential case. By executing again the same process, the intermediate
│ │ │ │ results obtained by calling costly_compute_cached will be loaded from the
│ │ │ │ cache instead of executing the function.
│ │ │ │
│ │ │ │
│ │ │ │ Second round - reloading from the cache
│ │ │ │ -Elapsed time for the entire processing: 0.04 s
│ │ │ │ +Elapsed time for the entire processing: 0.11 s
│ │ │ │
│ │ │ │
│ │ │ │
│ │ │ │
│ │ │ │ Reuse intermediate checkpoints¶
│ │ │ │
│ │ │ │
│ │ │ │
│ │ │ │ Reusing intermediate checkpoints
│ │ │ │ -Elapsed time for the entire processing: 0.04 s
│ │ │ │ +Elapsed time for the entire processing: 0.10 s
│ │ │ │
│ │ │ │
│ │ │ │ The processing time only corresponds to the execution of the max
│ │ │ │ function. The internal call to costly_compute_cached is reloading the
│ │ │ │ results from the cache.
│ │ │ │
│ │ │ │
│ │ │ │ Clean-up the cache folder¶
│ │ │ │ memory.clear(warn=False)
│ │ │ │
│ │ │ │
│ │ │ │ -Total running time of the script: (0 minutes 12.922 seconds)
│ │ │ │ +Total running time of the script: (0 minutes 13.469 seconds)
│ │ │ │
│ │ │ │ Create result generator with return_as='generator'...
│ │ │ │ Accumulate results:......................................................................................................................................................
│ │ │ │ All tasks completed and reduced successfully.
│ │ │ │ -Peak memory usage: 126.90MB
│ │ │ │ +Peak memory usage: 144.18MB
│ │ │ │ We can then report the memory usage accross time of the two runs using the │ │ │ │ MemoryMonitor.
│ │ │ │In the first case, as the results accumulate in res, the memory grows
│ │ │ │ linearly and it is freed once the accumulator_sum function finishes.
In the second case, the results are processed by the accumulator as soon as │ │ │ │ @@ -200,15 +200,15 @@ │ │ │ │ plt.show() │ │ │ │
It is important to note that with return_as="generator", the results are
│ │ │ │ still accumulated in RAM after computation. But as we asynchronously process
│ │ │ │ them, they can be freed sooner. However, if the generator is not consumed
│ │ │ │ the memory still grows linearly.
Total running time of the script: (1 minutes 15.933 seconds)
│ │ │ │ +Total running time of the script: (1 minutes 10.622 seconds)
│ │ │ │