The function took 5.05 s to compute.\n+The function took 5.04 s to compute.\n \n The transformed data are:\n [ 0.49671415 -0.1382643 0.64768854 1.52302986 -0.23415337 -0.23413696\n 1.57921282 0.76743473 -0.46947439 0.54256004]\n
\n \n start = time.time()\n data_trans = transformer.transform(data)\n end = time.time()\n \n print('\\nThe function took {:.2f} s to compute.'.format(end - start))\n print('\\nThe transformed data are:\\n {}'.format(data_trans))\n
\n \n-The function took 0.02 s to compute.\n+The function took 0.01 s to compute.\n \n The transformed data are:\n [ 0.49671415 -0.1382643 0.64768854 1.52302986 -0.23415337 -0.23413696\n 1.57921282 0.76743473 -0.46947439 0.54256004]\n
\n \n As expected, the second call to the transform method load the results\n which have been cached.
\n \n \n Clean up cache directory\u00b6
\n memory.clear(warn=False)\n
\n \n-Total running time of the script: (0 minutes 15.175 seconds)
\n+Total running time of the script: (0 minutes 15.142 seconds)
\n \n
\n First round - caching the data\n-Elapsed time for the entire processing: 4.25 s\n+Elapsed time for the entire processing: 4.19 s\n
\n \n By using 2 workers, the parallel processing gives a x2 speed-up compared to\n the sequential case. By executing again the same process, the intermediate\n results obtained by calling costly_compute_cached will be loaded from the\n cache instead of executing the function.
\n start = time.time()\n@@ -178,15 +178,15 @@\n results from the cache.\n \n \n Clean-up the cache folder\u00b6
\n memory.clear(warn=False)\n
\n \n-Total running time of the script: (0 minutes 12.284 seconds)
\n+Total running time of the script: (0 minutes 12.222 seconds)
\n \n
\n Running tasks with return_as='list'...\n Accumulate results:......................................................................................................................................................\n All tasks completed and reduced successfully.\n-Peak memory usage: 2.48GB\n+Peak memory usage: 2.53GB\n
\n \n If we use return_as="generator", res is simply a generator on the\n results that are ready. Here we consume the results as soon as they arrive\n with the accumulator_sum and once they have been used, they are collected\n by the gc. The memory footprint is thus reduced, typically around 300MB.
\n monitor_gen = MemoryMonitor()\n@@ -169,15 +169,15 @@\n peak = max(monitor_gen.memory_buffer) / 1e6\n print(f"Peak memory usage: {peak:.2f}MB")\n
\n \n Create result generator with return_as='generator'...\n Accumulate results:......................................................................................................................................................\n All tasks completed and reduced successfully.\n-Peak memory usage: 173.25MB\n+Peak memory usage: 163.07MB\n
\n \n We can then report the memory usage accross time of the two runs using the\n MemoryMonitor.
\n In the first case, as the results accumulate in res, the memory grows\n linearly and it is freed once the accumulator_sum function finishes.
\n In the second case, the results are processed by the accumulator as soon as\n@@ -200,15 +200,15 @@\n plt.show()\n
\n \n ![\"parallel](\"../_images/sphx_glr_parallel_generator_001.png\")
It is important to note that with return_as="generator", the results are\n still accumulated in RAM after computation. But as we asynchronously process\n them, they can be freed sooner. However, if the generator is not consumed\n the memory still grows linearly.
\n-Total running time of the script: (0 minutes 20.912 seconds)
\n+Total running time of the script: (0 minutes 19.145 seconds)
\n \n Elapsed time computing the average of couple of slices 0.73 s\n+Elapsed time computing the average of couple of slices 0.66 s\n
\n \n Parallel processing is already faster than the sequential processing. It is\n also possible to remove a bit of overhead by dumping the data array to a\n memmap and pass the memmap to joblib.Parallel.
\n \n \n-Elapsed time computing the average of couple of slices 0.53 s\n+Elapsed time computing the average of couple of slices 0.52 s\n
\n \n Therefore, dumping large data array ahead of calling\n joblib.Parallel can speed up the processing by removing some\n overhead.
\n \n \n \n Clean-up the memmap\u00b6
\n \n Remove the different memmap that we created. It might fail in Windows due\n@@ -225,15 +225,15 @@\n \n try:\n shutil.rmtree(folder)\n except: # noqa\n print('Could not clean-up automatically.')\n
\n
\n-Total running time of the script: (0 minutes 2.719 seconds)
\n+Total running time of the script: (0 minutes 2.553 seconds)
\n \n \n The different generated vectors in a sequential manner are:\n- [[7 8 5 9 0]\n- [8 5 3 3 0]\n- [1 9 8 4 8]\n- [0 5 4 4 8]\n- [1 1 4 9 5]]\n+ [[5 4 4 5 3]\n+ [2 5 1 9 8]\n+ [1 9 8 3 2]\n+ [6 5 2 1 7]\n+ [5 0 2 8 3]]\n
\n \n \n \n Parallel behavior\u00b6
\n \n Joblib provides three different backends: loky (default), threading, and\n@@ -98,33 +98,33 @@\n
backend = 'loky'\n random_vector = Parallel(n_jobs=2, backend=backend)(delayed(\n stochastic_function)(10) for _ in range(n_vectors))\n print_vector(random_vector, backend)\n
\n \n The different generated vectors using the loky backend are:\n- [[3 6 3 4 0]\n- [1 9 4 7 4]\n- [9 2 2 8 4]\n- [0 4 8 0 3]\n- [3 9 4 2 5]]\n+ [[4 0 9 3 9]\n+ [6 4 8 2 6]\n+ [4 1 5 9 8]\n+ [7 9 9 2 6]\n+ [6 4 0 4 5]]\n
\n \n backend = 'threading'\n random_vector = Parallel(n_jobs=2, backend=backend)(delayed(\n stochastic_function)(10) for _ in range(n_vectors))\n print_vector(random_vector, backend)\n
\n \n The different generated vectors using the threading backend are:\n- [[0 6 4 6 6]\n- [0 8 4 3 0]\n- [9 1 7 7 3]\n- [8 2 7 1 4]\n- [2 1 8 7 1]]\n+ [[5 5 1 8 6]\n+ [4 4 8 8 3]\n+ [4 1 0 0 2]\n+ [7 5 8 8 2]\n+ [3 9 5 9 0]]\n
\n \n Loky and the threading backends behave exactly as in the sequential case and\n do not require more care. However, this is not the case regarding the\n multiprocessing backend with the \u201cfork\u201d or \u201cforkserver\u201d start method because\n the state of the global numpy random stated will be exactly duplicated\n in all the workers
\n@@ -137,19 +137,19 @@\n backend = 'multiprocessing'\n random_vector = Parallel(n_jobs=2, backend=backend)(delayed(\n stochastic_function)(10) for _ in range(n_vectors))\n print_vector(random_vector, backend)\n \n
The different generated vectors using the multiprocessing backend are:\n- [[7 0 2 4 1]\n- [7 0 2 4 1]\n- [6 6 2 5 6]\n- [6 6 2 5 6]\n- [2 4 8 8 3]]\n+ [[2 3 7 2 2]\n+ [2 3 7 2 2]\n+ [3 5 4 0 4]\n+ [3 5 4 0 4]\n+ [6 8 4 3 1]]\n Some of the generated vectors are exactly the same, which can be a\n problem for the application.
\nTechnically, the reason is that all forked Python processes share the\n same exact random seed. As a result, we obtain twice the same randomly\n generated vectors because we are using n_jobs=2. A solution is to\n@@ -166,19 +166,19 @@\n
if mp.get_start_method() != "spawn":\n random_vector = Parallel(n_jobs=2, backend=backend)(delayed(\n stochastic_function_seeded)(10, None) for _ in range(n_vectors))\n print_vector(random_vector, backend)\n The different generated vectors using the multiprocessing backend are:\n- [[6 5 8 2 9]\n- [1 3 5 1 8]\n- [2 3 5 8 6]\n- [2 8 1 3 2]\n- [2 5 2 5 1]]\n+ [[2 3 1 1 9]\n+ [5 2 7 5 6]\n+ [0 8 7 4 0]\n+ [9 7 4 8 7]\n+ [9 8 5 9 4]]\n Fixing the random state to obtain deterministic results\u00b6
\n\n\nThe pattern of
stochastic_function_seededhas another advantage: it\n@@ -195,29 +195,29 @@\n \n random_vector = Parallel(n_jobs=2, backend=backend)(delayed(\n stochastic_function_seeded)(10, rng) for rng in random_state)\n print_vector(random_vector, backend)\n
The different generated vectors using the multiprocessing backend are:\n- [[0 7 3 5 4]\n- [1 0 3 2 0]\n- [5 6 9 4 6]\n- [0 2 0 2 8]\n- [2 2 4 6 9]]\n+ [[7 4 6 0 0]\n+ [1 9 1 0 5]\n+ [5 2 1 4 9]\n+ [1 6 1 3 4]\n+ [9 6 2 8 8]]\n \n The different generated vectors using the multiprocessing backend are:\n- [[0 7 3 5 4]\n- [1 0 3 2 0]\n- [5 6 9 4 6]\n- [0 2 0 2 8]\n- [2 2 4 6 9]]\n+ [[7 4 6 0 0]\n+ [1 9 1 0 5]\n+ [5 2 1 4 9]\n+ [1 6 1 3 4]\n+ [9 6 2 8 8]]\n Total running time of the script: (0 minutes 0.378 seconds)
\n+Total running time of the script: (0 minutes 0.320 seconds)
\n \n