In [144]: ser = pd.Series(arr[:, 0])\n \n In [145]: %timeit ser.iloc[indexer]\n .....: %timeit ser.take(indexer)\n .....: \n-356 us +- 34.6 us per loop (mean +- std. dev. of 7 runs, 1,000 loops each)\n-345 us +- 77.4 us per loop (mean +- std. dev. of 7 runs, 1,000 loops each)\n+138 us +- 1.35 us per loop (mean +- std. dev. of 7 runs, 10,000 loops each)\n+147 us +- 15.1 us per loop (mean +- std. dev. of 7 runs, 10,000 loops each)\n We have discussed MultiIndex in the previous sections pretty extensively.\n Documentation about DatetimeIndex and PeriodIndex are shown here,\n", "details": [{"source1": "html2text {}", "source2": "html2text {}", "unified_diff": "@@ -1241,23 +1241,23 @@\n In [141]: indexer = np.arange(10000)\n \n In [142]: random.shuffle(indexer)\n \n In [143]: %timeit arr[indexer]\n .....: %timeit arr.take(indexer, axis=0)\n .....:\n-621 us +- 68.9 us per loop (mean +- std. dev. of 7 runs, 1,000 loops each)\n-288 us +- 26.1 us per loop (mean +- std. dev. of 7 runs, 1,000 loops each)\n+308 us +- 1.63 us per loop (mean +- std. dev. of 7 runs, 1,000 loops each)\n+123 us +- 88.3 ns per loop (mean +- std. dev. of 7 runs, 10,000 loops each)\n In [144]: ser = pd.Series(arr[:, 0])\n \n In [145]: %timeit ser.iloc[indexer]\n .....: %timeit ser.take(indexer)\n .....:\n-356 us +- 34.6 us per loop (mean +- std. dev. of 7 runs, 1,000 loops each)\n-345 us +- 77.4 us per loop (mean +- std. dev. of 7 runs, 1,000 loops each)\n+138 us +- 1.35 us per loop (mean +- std. dev. of 7 runs, 10,000 loops each)\n+147 us +- 15.1 us per loop (mean +- std. dev. of 7 runs, 10,000 loops each)\n *\b**\b**\b**\b**\b* I\bIn\bnd\bde\bex\bx t\bty\byp\bpe\bes\bs_\b#\b# *\b**\b**\b**\b**\b*\n We have discussed MultiIndex in the previous sections pretty extensively.\n Documentation about DatetimeIndex and PeriodIndex are shown _\bh_\be_\br_\be, and\n documentation about TimedeltaIndex is found _\bh_\be_\br_\be.\n In the following sub-sections we will highlight some other index types.\n *\b**\b**\b**\b* C\bCa\bat\bte\beg\bgo\bor\bri\bic\bca\bal\blI\bIn\bnd\bde\bex\bx_\b#\b# *\b**\b**\b**\b*\n _\bC_\ba_\bt_\be_\bg_\bo_\br_\bi_\bc_\ba_\bl_\bI_\bn_\bd_\be_\bx is a type of index that is useful for supporting indexing with\n"}]}, {"source1": "./usr/share/doc/python-pandas-doc/html/user_guide/enhancingperf.html", "source2": "./usr/share/doc/python-pandas-doc/html/user_guide/enhancingperf.html", "unified_diff": "@@ -601,31 +601,31 @@\n ...: s += f(a + i * dx)\n ...: return s * dx\n ...: \n \n \n
We achieve our result by using DataFrame.apply() (row-wise):
In [5]: %timeit df.apply(lambda x: integrate_f(x["a"], x["b"], x["N"]), axis=1)\n-210 ms +- 8.6 ms per loop (mean +- std. dev. of 7 runs, 1 loop each)\n+102 ms +- 196 us per loop (mean +- std. dev. of 7 runs, 10 loops each)\n Let\u2019s take a look and see where the time is spent during this operation\n using the prun ipython magic function:
\n# most time consuming 4 calls\n In [6]: %prun -l 4 df.apply(lambda x: integrate_f(x["a"], x["b"], x["N"]), axis=1) # noqa E999\n- 605951 function calls (605933 primitive calls) in 1.613 seconds\n+ 605951 function calls (605933 primitive calls) in 0.283 seconds\n \n Ordered by: internal time\n List reduced from 159 to 4 due to restriction <4>\n \n ncalls tottime percall cumtime percall filename:lineno(function)\n- 1000 0.834 0.001 1.425 0.001 <ipython-input-4-c2a74e076cf0>:1(integrate_f)\n- 552423 0.591 0.000 0.591 0.000 <ipython-input-3-c138bdd570e3>:1(f)\n- 3000 0.029 0.000 0.129 0.000 series.py:1095(__getitem__)\n- 16098 0.023 0.000 0.030 0.000 {built-in method builtins.isinstance}\n+ 1000 0.165 0.000 0.245 0.000 <ipython-input-4-c2a74e076cf0>:1(integrate_f)\n+ 552423 0.080 0.000 0.080 0.000 <ipython-input-3-c138bdd570e3>:1(f)\n+ 3000 0.006 0.000 0.025 0.000 series.py:1095(__getitem__)\n+ 3000 0.005 0.000 0.005 0.000 base.py:3777(get_loc)\n By far the majority of time is spend inside either integrate_f or f,\n hence we\u2019ll concentrate our efforts cythonizing these two functions.
In [9]: %timeit df.apply(lambda x: integrate_f_plain(x["a"], x["b"], x["N"]), axis=1)\n-175 ms +- 16.2 ms per loop (mean +- std. dev. of 7 runs, 10 loops each)\n+84.8 ms +- 69.5 us per loop (mean +- std. dev. of 7 runs, 10 loops each)\n This has improved the performance compared to the pure Python approach by one-third.
\nWe can annotate the function variables and return types as well as use cdef\n@@ -667,36 +667,36 @@\n ....: for i in range(N):\n ....: s += f_typed(a + i * dx)\n ....: return s * dx\n ....: \n \n \n
In [11]: %timeit df.apply(lambda x: integrate_f_typed(x["a"], x["b"], x["N"]), axis=1)\n-29 ms +- 478 us per loop (mean +- std. dev. of 7 runs, 10 loops each)\n+14.3 ms +- 17.1 us per loop (mean +- std. dev. of 7 runs, 100 loops each)\n Annotating the functions with C types yields an over ten times performance improvement compared to\n the original Python implementation.
\nWhen re-profiling, time is spent creating a Series from each row, and calling __getitem__ from both\n the index and the series (three times for each row). These Python function calls are expensive and\n can be improved by passing an np.ndarray.
In [12]: %prun -l 4 df.apply(lambda x: integrate_f_typed(x["a"], x["b"], x["N"]), axis=1)\n- 52528 function calls (52510 primitive calls) in 0.158 seconds\n+ 52528 function calls (52510 primitive calls) in 0.037 seconds\n \n Ordered by: internal time\n List reduced from 157 to 4 due to restriction <4>\n \n ncalls tottime percall cumtime percall filename:lineno(function)\n- 3000 0.024 0.000 0.110 0.000 series.py:1095(__getitem__)\n- 16098 0.020 0.000 0.025 0.000 {built-in method builtins.isinstance}\n- 3000 0.017 0.000 0.045 0.000 series.py:1220(_get_value)\n- 3000 0.015 0.000 0.028 0.000 indexing.py:2765(check_dict_or_set_indexers)\n+ 3000 0.006 0.000 0.024 0.000 series.py:1095(__getitem__)\n+ 3000 0.004 0.000 0.005 0.000 base.py:3777(get_loc)\n+ 3000 0.004 0.000 0.012 0.000 series.py:1220(_get_value)\n+ 16098 0.003 0.000 0.004 0.000 {built-in method builtins.isinstance}\n In [13]: %%cython\n ....: cimport numpy as np\n ....: import numpy as np\n ....: cdef double f_typed(double x) except? -2:\n ....: return x * (x - 1)\n@@ -731,33 +731,33 @@\n This implementation creates an array of zeros and inserts the result\n of integrate_f_typed applied over each row. Looping over an ndarray is faster\n in Cython than looping over a Series object.
Since apply_integrate_f is typed to accept an np.ndarray, Series.to_numpy()\n calls are needed to utilize this function.
In [14]: %timeit apply_integrate_f(df["a"].to_numpy(), df["b"].to_numpy(), df["N"].to_numpy())\n-2.6 ms +- 44.3 us per loop (mean +- std. dev. of 7 runs, 100 loops each)\n+1.91 ms +- 4.7 us per loop (mean +- std. dev. of 7 runs, 1,000 loops each)\n Performance has improved from the prior implementation by almost ten times.
\n \nThe majority of the time is now spent in apply_integrate_f. Disabling Cython\u2019s boundscheck\n and wraparound checks can yield more performance.
In [15]: %prun -l 4 apply_integrate_f(df["a"].to_numpy(), df["b"].to_numpy(), df["N"].to_numpy())\n- 78 function calls in 0.003 seconds\n+ 78 function calls in 0.002 seconds\n \n Ordered by: internal time\n List reduced from 21 to 4 due to restriction <4>\n \n ncalls tottime percall cumtime percall filename:lineno(function)\n- 1 0.003 0.003 0.003 0.003 <string>:1(<module>)\n- 1 0.000 0.000 0.003 0.003 {built-in method builtins.exec}\n+ 1 0.002 0.002 0.002 0.002 <string>:1(<module>)\n 1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}\n+ 1 0.000 0.000 0.002 0.002 {built-in method builtins.exec}\n 3 0.000 0.000 0.000 0.000 frame.py:4062(__getitem__)\n In [16]: %%cython\n ....: cimport cython\n ....: cimport numpy as np\n ....: import numpy as np\n@@ -1189,19 +1189,19 @@\n compared to standard Python syntax for large DataFrame. This engine requires the\n optional dependency numexpr to be installed.\n The 'python' engine is generally not useful except for testing\n other evaluation engines against it. You will achieve no performance\n benefits using eval() with engine='python' and may\n incur a performance hit.
\n In [40]: %timeit df1 + df2 + df3 + df4\n-768 ms +- 17.3 ms per loop (mean +- std. dev. of 7 runs, 1 loop each)\n+458 ms +- 54.2 ms per loop (mean +- std. dev. of 7 runs, 1 loop each)\n
\n \n In [41]: %timeit pd.eval("df1 + df2 + df3 + df4", engine="python")\n-753 ms +- 56.8 ms per loop (mean +- std. dev. of 7 runs, 1 loop each)\n+486 ms +- 82.9 ms per loop (mean +- std. dev. of 7 runs, 1 loop each)\n
\n \n \n \n The DataFrame.eval() method#
\n In addition to the top level pandas.eval() function you can also\n evaluate an expression in the \u201ccontext\u201d of a DataFrame.
\n@@ -1316,39 +1316,39 @@\n In [58]: nrows, ncols = 20000, 100\n \n In [59]: df1, df2, df3, df4 = [pd.DataFrame(np.random.randn(nrows, ncols)) for _ in range(4)]\n
\n \n DataFrame arithmetic:
\n In [60]: %timeit df1 + df2 + df3 + df4\n-808 ms +- 61.1 ms per loop (mean +- std. dev. of 7 runs, 1 loop each)\n+506 ms +- 135 ms per loop (mean +- std. dev. of 7 runs, 1 loop each)\n
\n \n In [61]: %timeit pd.eval("df1 + df2 + df3 + df4")\n-240 ms +- 36 ms per loop (mean +- std. dev. of 7 runs, 1 loop each)\n+143 ms +- 9.47 ms per loop (mean +- std. dev. of 7 runs, 10 loops each)\n
\n \n DataFrame comparison:
\n In [62]: %timeit (df1 > 0) & (df2 > 0) & (df3 > 0) & (df4 > 0)\n-53.6 ms +- 2.66 ms per loop (mean +- std. dev. of 7 runs, 10 loops each)\n+15.8 ms +- 29.1 us per loop (mean +- std. dev. of 7 runs, 100 loops each)\n
\n \n In [63]: %timeit pd.eval("(df1 > 0) & (df2 > 0) & (df3 > 0) & (df4 > 0)")\n-11.3 ms +- 224 us per loop (mean +- std. dev. of 7 runs, 100 loops each)\n+5.07 ms +- 49.4 us per loop (mean +- std. dev. of 7 runs, 100 loops each)\n
\n \n DataFrame arithmetic with unaligned axes.
\n In [64]: s = pd.Series(np.random.randn(50))\n \n In [65]: %timeit df1 + df2 + df3 + df4 + s\n-1.23 s +- 63.9 ms per loop (mean +- std. dev. of 7 runs, 1 loop each)\n+762 ms +- 43.3 ms per loop (mean +- std. dev. of 7 runs, 1 loop each)\n
\n \n In [66]: %timeit pd.eval("df1 + df2 + df3 + df4 + s")\n-252 ms +- 13.4 ms per loop (mean +- std. dev. of 7 runs, 1 loop each)\n+161 ms +- 4.23 ms per loop (mean +- std. dev. of 7 runs, 10 loops each)\n
\n \n \n Note
\n Operations such as
\n 1 and 2 # would parse to 1 & 2, but should evaluate to 2\n 3 or 4 # would parse to 3 | 4, but should evaluate to 3\n", "details": [{"source1": "html2text {}", "source2": "html2text {}", "unified_diff": "@@ -106,33 +106,32 @@\n ...: dx = (b - a) / N\n ...: for i in range(N):\n ...: s += f(a + i * dx)\n ...: return s * dx\n ...:\n We achieve our result by using _\bD_\ba_\bt_\ba_\bF_\br_\ba_\bm_\be_\b._\ba_\bp_\bp_\bl_\by_\b(_\b) (row-wise):\n In [5]: %timeit df.apply(lambda x: integrate_f(x[\"a\"], x[\"b\"], x[\"N\"]), axis=1)\n-210 ms +- 8.6 ms per loop (mean +- std. dev. of 7 runs, 1 loop each)\n+102 ms +- 196 us per loop (mean +- std. dev. of 7 runs, 10 loops each)\n Let\u2019s take a look and see where the time is spent during this operation using\n the _\bp_\br_\bu_\bn_\b _\bi_\bp_\by_\bt_\bh_\bo_\bn_\b _\bm_\ba_\bg_\bi_\bc_\b _\bf_\bu_\bn_\bc_\bt_\bi_\bo_\bn:\n # most time consuming 4 calls\n In [6]: %prun -l 4 df.apply(lambda x: integrate_f(x[\"a\"], x[\"b\"], x[\"N\"]),\n axis=1) # noqa E999\n- 605951 function calls (605933 primitive calls) in 1.613 seconds\n+ 605951 function calls (605933 primitive calls) in 0.283 seconds\n \n Ordered by: internal time\n List reduced from 159 to 4 due to restriction <4>\n \n ncalls tottime percall cumtime percall filename:lineno(function)\n- 1000 0.834 0.001 1.425 0.001 :1\n+ 1000 0.165 0.000 0.245 0.000 :1\n (integrate_f)\n- 552423 0.591 0.000 0.591 0.000 :1\n+ 552423 0.080 0.000 0.080 0.000 :1\n (f)\n- 3000 0.029 0.000 0.129 0.000 series.py:1095(__getitem__)\n- 16098 0.023 0.000 0.030 0.000 {built-in method\n-builtins.isinstance}\n+ 3000 0.006 0.000 0.025 0.000 series.py:1095(__getitem__)\n+ 3000 0.005 0.000 0.005 0.000 base.py:3777(get_loc)\n By far the majority of time is spend inside either integrate_f or f, hence\n we\u2019ll concentrate our efforts cythonizing these two functions.\n *\b**\b**\b**\b* P\bPl\bla\bai\bin\bn C\bCy\byt\bth\bho\bon\bn_\b#\b# *\b**\b**\b**\b*\n First we\u2019re going to need to import the Cython magic function to IPython:\n In [7]: %load_ext Cython\n Now, let\u2019s simply copy our functions over to Cython:\n In [8]: %%cython\n@@ -143,15 +142,15 @@\n ...: dx = (b - a) / N\n ...: for i in range(N):\n ...: s += f_plain(a + i * dx)\n ...: return s * dx\n ...:\n In [9]: %timeit df.apply(lambda x: integrate_f_plain(x[\"a\"], x[\"b\"], x[\"N\"]),\n axis=1)\n-175 ms +- 16.2 ms per loop (mean +- std. dev. of 7 runs, 10 loops each)\n+84.8 ms +- 69.5 us per loop (mean +- std. dev. of 7 runs, 10 loops each)\n This has improved the performance compared to the pure Python approach by one-\n third.\n *\b**\b**\b**\b* D\bDe\bec\bcl\bla\bar\bri\bin\bng\bg C\bC t\bty\byp\bpe\bes\bs_\b#\b# *\b**\b**\b**\b*\n We can annotate the function variables and return types as well as use cdef and\n cpdef to improve performance:\n In [10]: %%cython\n ....: cdef double f_typed(double x) except? -2:\n@@ -163,36 +162,35 @@\n ....: dx = (b - a) / N\n ....: for i in range(N):\n ....: s += f_typed(a + i * dx)\n ....: return s * dx\n ....:\n In [11]: %timeit df.apply(lambda x: integrate_f_typed(x[\"a\"], x[\"b\"], x[\"N\"]),\n axis=1)\n-29 ms +- 478 us per loop (mean +- std. dev. of 7 runs, 10 loops each)\n+14.3 ms +- 17.1 us per loop (mean +- std. dev. of 7 runs, 100 loops each)\n Annotating the functions with C types yields an over ten times performance\n improvement compared to the original Python implementation.\n *\b**\b**\b**\b* U\bUs\bsi\bin\bng\bg n\bnd\bda\bar\brr\bra\bay\by_\b#\b# *\b**\b**\b**\b*\n When re-profiling, time is spent creating a _\bS_\be_\br_\bi_\be_\bs from each row, and calling\n __getitem__ from both the index and the series (three times for each row).\n These Python function calls are expensive and can be improved by passing an\n np.ndarray.\n In [12]: %prun -l 4 df.apply(lambda x: integrate_f_typed(x[\"a\"], x[\"b\"], x\n [\"N\"]), axis=1)\n- 52528 function calls (52510 primitive calls) in 0.158 seconds\n+ 52528 function calls (52510 primitive calls) in 0.037 seconds\n \n Ordered by: internal time\n List reduced from 157 to 4 due to restriction <4>\n \n ncalls tottime percall cumtime percall filename:lineno(function)\n- 3000 0.024 0.000 0.110 0.000 series.py:1095(__getitem__)\n- 16098 0.020 0.000 0.025 0.000 {built-in method\n+ 3000 0.006 0.000 0.024 0.000 series.py:1095(__getitem__)\n+ 3000 0.004 0.000 0.005 0.000 base.py:3777(get_loc)\n+ 3000 0.004 0.000 0.012 0.000 series.py:1220(_get_value)\n+ 16098 0.003 0.000 0.004 0.000 {built-in method\n builtins.isinstance}\n- 3000 0.017 0.000 0.045 0.000 series.py:1220(_get_value)\n- 3000 0.015 0.000 0.028 0.000 indexing.py:2765\n-(check_dict_or_set_indexers)\n In [13]: %%cython\n ....: cimport numpy as np\n ....: import numpy as np\n ....: cdef double f_typed(double x) except? -2:\n ....: return x * (x - 1)\n ....: cpdef double integrate_f_typed(double a, double b, int N):\n ....: cdef int i\n@@ -233,31 +231,31 @@\n This implementation creates an array of zeros and inserts the result of\n integrate_f_typed applied over each row. Looping over an ndarray is faster in\n Cython than looping over a _\bS_\be_\br_\bi_\be_\bs object.\n Since apply_integrate_f is typed to accept an np.ndarray, _\bS_\be_\br_\bi_\be_\bs_\b._\bt_\bo_\b__\bn_\bu_\bm_\bp_\by_\b(_\b)\n calls are needed to utilize this function.\n In [14]: %timeit apply_integrate_f(df[\"a\"].to_numpy(), df[\"b\"].to_numpy(), df\n [\"N\"].to_numpy())\n-2.6 ms +- 44.3 us per loop (mean +- std. dev. of 7 runs, 100 loops each)\n+1.91 ms +- 4.7 us per loop (mean +- std. dev. of 7 runs, 1,000 loops each)\n Performance has improved from the prior implementation by almost ten times.\n *\b**\b**\b**\b* D\bDi\bis\bsa\bab\bbl\bli\bin\bng\bg c\bco\bom\bmp\bpi\bil\ble\ber\br d\bdi\bir\bre\bec\bct\bti\biv\bve\bes\bs_\b#\b# *\b**\b**\b**\b*\n The majority of the time is now spent in apply_integrate_f. Disabling Cython\u2019s\n boundscheck and wraparound checks can yield more performance.\n In [15]: %prun -l 4 apply_integrate_f(df[\"a\"].to_numpy(), df[\"b\"].to_numpy(),\n df[\"N\"].to_numpy())\n- 78 function calls in 0.003 seconds\n+ 78 function calls in 0.002 seconds\n \n Ordered by: internal time\n List reduced from 21 to 4 due to restriction <4>\n \n ncalls tottime percall cumtime percall filename:lineno(function)\n- 1 0.003 0.003 0.003 0.003 :1()\n- 1 0.000 0.000 0.003 0.003 {built-in method builtins.exec}\n+ 1 0.002 0.002 0.002 0.002 :1()\n 1 0.000 0.000 0.000 0.000 {method 'disable' of\n '_lsprof.Profiler' objects}\n+ 1 0.000 0.000 0.002 0.002 {built-in method builtins.exec}\n 3 0.000 0.000 0.000 0.000 frame.py:4062(__getitem__)\n In [16]: %%cython\n ....: cimport cython\n ....: cimport numpy as np\n ....: import numpy as np\n ....: cdef np.float64_t f_typed(np.float64_t x) except? -2:\n ....: return x * (x - 1)\n@@ -644,17 +642,17 @@\n The 'numexpr' engine is the more performant engine that can yield performance\n improvements compared to standard Python syntax for large _\bD_\ba_\bt_\ba_\bF_\br_\ba_\bm_\be. This\n engine requires the optional dependency numexpr to be installed.\n The 'python' engine is generally n\bno\bot\bt useful except for testing other evaluation\n engines against it. You will achieve n\bno\bo performance benefits using _\be_\bv_\ba_\bl_\b(_\b) with\n engine='python' and may incur a performance hit.\n In [40]: %timeit df1 + df2 + df3 + df4\n-768 ms +- 17.3 ms per loop (mean +- std. dev. of 7 runs, 1 loop each)\n+458 ms +- 54.2 ms per loop (mean +- std. dev. of 7 runs, 1 loop each)\n In [41]: %timeit pd.eval(\"df1 + df2 + df3 + df4\", engine=\"python\")\n-753 ms +- 56.8 ms per loop (mean +- std. dev. of 7 runs, 1 loop each)\n+486 ms +- 82.9 ms per loop (mean +- std. dev. of 7 runs, 1 loop each)\n *\b**\b**\b**\b* T\bTh\bhe\be _\bD\bD_\ba\ba_\bt\bt_\ba\ba_\bF\bF_\br\br_\ba\ba_\bm\bm_\be\be_\b.\b._\be\be_\bv\bv_\ba\ba_\bl\bl_\b(\b(_\b)\b) m\bme\bet\bth\bho\bod\bd_\b#\b# *\b**\b**\b**\b*\n In addition to the top level _\bp_\ba_\bn_\bd_\ba_\bs_\b._\be_\bv_\ba_\bl_\b(_\b) function you can also evaluate an\n expression in the \u201ccontext\u201d of a _\bD_\ba_\bt_\ba_\bF_\br_\ba_\bm_\be.\n In [42]: df = pd.DataFrame(np.random.randn(5, 2), columns=[\"a\", \"b\"])\n \n In [43]: df.eval(\"a + b\")\n Out[43]:\n@@ -751,29 +749,29 @@\n _\bp_\ba_\bn_\bd_\ba_\bs_\b._\be_\bv_\ba_\bl_\b(_\b) works well with expressions containing large arrays.\n In [58]: nrows, ncols = 20000, 100\n \n In [59]: df1, df2, df3, df4 = [pd.DataFrame(np.random.randn(nrows, ncols)) for\n _ in range(4)]\n _\bD_\ba_\bt_\ba_\bF_\br_\ba_\bm_\be arithmetic:\n In [60]: %timeit df1 + df2 + df3 + df4\n-808 ms +- 61.1 ms per loop (mean +- std. dev. of 7 runs, 1 loop each)\n+506 ms +- 135 ms per loop (mean +- std. dev. of 7 runs, 1 loop each)\n In [61]: %timeit pd.eval(\"df1 + df2 + df3 + df4\")\n-240 ms +- 36 ms per loop (mean +- std. dev. of 7 runs, 1 loop each)\n+143 ms +- 9.47 ms per loop (mean +- std. dev. of 7 runs, 10 loops each)\n _\bD_\ba_\bt_\ba_\bF_\br_\ba_\bm_\be comparison:\n In [62]: %timeit (df1 > 0) & (df2 > 0) & (df3 > 0) & (df4 > 0)\n-53.6 ms +- 2.66 ms per loop (mean +- std. dev. of 7 runs, 10 loops each)\n+15.8 ms +- 29.1 us per loop (mean +- std. dev. of 7 runs, 100 loops each)\n In [63]: %timeit pd.eval(\"(df1 > 0) & (df2 > 0) & (df3 > 0) & (df4 > 0)\")\n-11.3 ms +- 224 us per loop (mean +- std. dev. of 7 runs, 100 loops each)\n+5.07 ms +- 49.4 us per loop (mean +- std. dev. of 7 runs, 100 loops each)\n _\bD_\ba_\bt_\ba_\bF_\br_\ba_\bm_\be arithmetic with unaligned axes.\n In [64]: s = pd.Series(np.random.randn(50))\n \n In [65]: %timeit df1 + df2 + df3 + df4 + s\n-1.23 s +- 63.9 ms per loop (mean +- std. dev. of 7 runs, 1 loop each)\n+762 ms +- 43.3 ms per loop (mean +- std. dev. of 7 runs, 1 loop each)\n In [66]: %timeit pd.eval(\"df1 + df2 + df3 + df4 + s\")\n-252 ms +- 13.4 ms per loop (mean +- std. dev. of 7 runs, 1 loop each)\n+161 ms +- 4.23 ms per loop (mean +- std. dev. of 7 runs, 10 loops each)\n Note\n Operations such as\n 1 and 2 # would parse to 1 & 2, but should evaluate to 2\n 3 or 4 # would parse to 3 | 4, but should evaluate to 3\n ~1 # this is okay, but slower when using eval\n should be performed in Python. An exception will be raised if you try to\n perform any boolean/bitwise operations with scalar operands that are not of\n"}]}, {"source1": "./usr/share/doc/python-pandas-doc/html/user_guide/pyarrow.html", "source2": "./usr/share/doc/python-pandas-doc/html/user_guide/pyarrow.html", "unified_diff": "@@ -995,26 +995,19 @@\n Cell In[33], line 1\n ----> 1 table = pa.table([pa.array([1, 2, 3], type=pa.int64())], names=["a"])\n \n NameError: name 'pa' is not defined\n \n In [34]: df = table.to_pandas(types_mapper=pd.ArrowDtype)\n ---------------------------------------------------------------------------\n-AttributeError Traceback (most recent call last)\n-<ipython-input-34-64ec62289cb4> in ?()\n+NameError Traceback (most recent call last)\n+Cell In[34], line 1\n ----> 1 df = table.to_pandas(types_mapper=pd.ArrowDtype)\n \n-/usr/lib/python3/dist-packages/pandas/core/generic.py in ?(self, name)\n- 6295 and name not in self._accessors\n- 6296 and self._info_axis._can_hold_identifiers_and_holds_name(name)\n- 6297 ):\n- 6298 return self[name]\n--> 6299 return object.__getattribute__(self, name)\n-\n-AttributeError: 'DataFrame' object has no attribute 'to_pandas'\n+NameError: name 'table' is not defined\n \n In [35]: df\n Out[35]: \n a b\n 0 xxx yyy\n 1 \u00a1\u00a1 \u00a1\u00a1\n \n", "details": [{"source1": "html2text {}", "source2": "html2text {}", "unified_diff": "@@ -522,27 +522,19 @@\n Cell In[33], line 1\n ----> 1 table = pa.table([pa.array([1, 2, 3], type=pa.int64())], names=[\"a\"])\n \n NameError: name 'pa' is not defined\n \n In [34]: df = table.to_pandas(types_mapper=pd.ArrowDtype)\n ---------------------------------------------------------------------------\n-AttributeError Traceback (most recent call last)\n- in ?()\n+NameError Traceback (most recent call last)\n+Cell In[34], line 1\n ----> 1 df = table.to_pandas(types_mapper=pd.ArrowDtype)\n \n-/usr/lib/python3/dist-packages/pandas/core/generic.py in ?(self, name)\n- 6295 and name not in self._accessors\n- 6296 and self._info_axis._can_hold_identifiers_and_holds_name\n-(name)\n- 6297 ):\n- 6298 return self[name]\n--> 6299 return object.__getattribute__(self, name)\n-\n-AttributeError: 'DataFrame' object has no attribute 'to_pandas'\n+NameError: name 'table' is not defined\n \n In [35]: df\n Out[35]:\n a b\n 0 xxx yyy\n 1 \u00a1\u00a1 \u00a1\u00a1\n \n"}]}, {"source1": "./usr/share/doc/python-pandas-doc/html/user_guide/scale.html", "source2": "./usr/share/doc/python-pandas-doc/html/user_guide/scale.html", "unified_diff": "@@ -1095,16 +1095,16 @@\n ....: files = pathlib.Path("data/timeseries/").glob("ts*.parquet")\n ....: counts = pd.Series(dtype=int)\n ....: for path in files:\n ....: df = pd.read_parquet(path)\n ....: counts = counts.add(df["name"].value_counts(), fill_value=0)\n ....: counts.astype(int)\n ....: \n-CPU times: user 680 us, sys: 0 ns, total: 680 us\n-Wall time: 684 us\n+CPU times: user 408 us, sys: 362 us, total: 770 us\n+Wall time: 2.18 ms\n Out[32]: Series([], dtype: int32)\n
\n \n Some readers, like pandas.read_csv(), offer parameters to control the\n chunksize when reading a single file.
\n Manually chunking is an OK option for workflows that don\u2019t\n require too sophisticated of operations. Some operations, like pandas.DataFrame.groupby(), are\n", "details": [{"source1": "html2text {}", "source2": "html2text {}", "unified_diff": "@@ -640,16 +640,16 @@\n ....: files = pathlib.Path(\"data/timeseries/\").glob(\"ts*.parquet\")\n ....: counts = pd.Series(dtype=int)\n ....: for path in files:\n ....: df = pd.read_parquet(path)\n ....: counts = counts.add(df[\"name\"].value_counts(), fill_value=0)\n ....: counts.astype(int)\n ....:\n-CPU times: user 680 us, sys: 0 ns, total: 680 us\n-Wall time: 684 us\n+CPU times: user 408 us, sys: 362 us, total: 770 us\n+Wall time: 2.18 ms\n Out[32]: Series([], dtype: int32)\n Some readers, like _\bp_\ba_\bn_\bd_\ba_\bs_\b._\br_\be_\ba_\bd_\b__\bc_\bs_\bv_\b(_\b), offer parameters to control the chunksize\n when reading a single file.\n Manually chunking is an OK option for workflows that don\u2019t require too\n sophisticated of operations. Some operations, like _\bp_\ba_\bn_\bd_\ba_\bs_\b._\bD_\ba_\bt_\ba_\bF_\br_\ba_\bm_\be_\b._\bg_\br_\bo_\bu_\bp_\bb_\by_\b(_\b),\n are much harder to do chunkwise. In these cases, you may be better switching to\n a different library that implements these out-of-core algorithms for you.\n"}]}, {"source1": "./usr/share/doc/python-pandas-doc/html/user_guide/style.ipynb.gz", "source2": "./usr/share/doc/python-pandas-doc/html/user_guide/style.ipynb.gz", "unified_diff": null, "details": [{"source1": "style.ipynb", "source2": "style.ipynb", "unified_diff": null, "details": [{"source1": "Pretty-printed", "source2": "Pretty-printed", "comments": ["Similarity: 0.9985610875706213%", "Differences: {\"'cells'\": \"{1: {'metadata': {'execution': {'iopub.execute_input': '2025-12-11T23:36:27.300629Z', \"", " \"'iopub.status.busy': '2025-12-11T23:36:27.300243Z', 'iopub.status.idle': \"", " \"'2025-12-11T23:36:27.889580Z', 'shell.execute_reply': \"", " \"'2025-12-11T23:36:27.888991Z'}}}, 3: {'metadata': {'execution': \"", " \"{'iopub.execute_input': '2025-12-11T23:36:27.893615Z', 'iopub.status.busy': \"", " \"'2025-12-11T23:36:27.893061Z', 'iopub.status.idle': '2025-12-11T23:36:2 [\u2026]"], "unified_diff": "@@ -39,18 +39,18 @@\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 1,\n \"metadata\": {\n \"execution\": {\n- \"iopub.execute_input\": \"2024-11-08T14:51:53.436265Z\",\n- \"iopub.status.busy\": \"2024-11-08T14:51:53.435850Z\",\n- \"iopub.status.idle\": \"2024-11-08T14:51:54.474028Z\",\n- \"shell.execute_reply\": \"2024-11-08T14:51:54.472870Z\"\n+ \"iopub.execute_input\": \"2025-12-11T23:36:27.300629Z\",\n+ \"iopub.status.busy\": \"2025-12-11T23:36:27.300243Z\",\n+ \"iopub.status.idle\": \"2025-12-11T23:36:27.889580Z\",\n+ \"shell.execute_reply\": \"2025-12-11T23:36:27.888991Z\"\n },\n \"nbsphinx\": \"hidden\"\n },\n \"outputs\": [],\n \"source\": [\n \"import matplotlib.pyplot\\n\",\n \"# We have this here to trigger matplotlib's font cache stuff.\\n\",\n@@ -77,36 +77,36 @@\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 2,\n \"metadata\": {\n \"execution\": {\n- \"iopub.execute_input\": \"2024-11-08T14:51:54.479097Z\",\n- \"iopub.status.busy\": \"2024-11-08T14:51:54.478394Z\",\n- \"iopub.status.idle\": \"2024-11-08T14:51:55.099332Z\",\n- \"shell.execute_reply\": \"2024-11-08T14:51:55.098252Z\"\n+ \"iopub.execute_input\": \"2025-12-11T23:36:27.893615Z\",\n+ \"iopub.status.busy\": \"2025-12-11T23:36:27.893061Z\",\n+ \"iopub.status.idle\": \"2025-12-11T23:36:28.358030Z\",\n+ \"shell.execute_reply\": \"2025-12-11T23:36:28.357414Z\"\n }\n },\n \"outputs\": [],\n \"source\": [\n \"import pandas as pd\\n\",\n \"import numpy as np\\n\",\n \"import matplotlib as mpl\\n\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 3,\n \"metadata\": {\n \"execution\": {\n- \"iopub.execute_input\": \"2024-11-08T14:51:55.104847Z\",\n- \"iopub.status.busy\": \"2024-11-08T14:51:55.104268Z\",\n- \"iopub.status.idle\": \"2024-11-08T14:51:55.230295Z\",\n- \"shell.execute_reply\": \"2024-11-08T14:51:55.229357Z\"\n+ \"iopub.execute_input\": \"2025-12-11T23:36:28.360993Z\",\n+ \"iopub.status.busy\": \"2025-12-11T23:36:28.360514Z\",\n+ \"iopub.status.idle\": \"2025-12-11T23:36:28.431529Z\",\n+ \"shell.execute_reply\": \"2025-12-11T23:36:28.430970Z\"\n },\n \"nbsphinx\": \"hidden\"\n },\n \"outputs\": [],\n \"source\": [\n \"# For reproducibility - this doesn't respect uuid_len or positionally-passed uuid but the places here that use that coincidentally bypass this anyway\\n\",\n \"from pandas.io.formats.style import Styler\\n\",\n@@ -123,18 +123,18 @@\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 4,\n \"metadata\": {\n \"execution\": {\n- \"iopub.execute_input\": \"2024-11-08T14:51:55.236100Z\",\n- \"iopub.status.busy\": \"2024-11-08T14:51:55.234984Z\",\n- \"iopub.status.idle\": \"2024-11-08T14:51:55.251807Z\",\n- \"shell.execute_reply\": \"2024-11-08T14:51:55.250847Z\"\n+ \"iopub.execute_input\": \"2025-12-11T23:36:28.434079Z\",\n+ \"iopub.status.busy\": \"2025-12-11T23:36:28.433553Z\",\n+ \"iopub.status.idle\": \"2025-12-11T23:36:28.442917Z\",\n+ \"shell.execute_reply\": \"2025-12-11T23:36:28.442347Z\"\n }\n },\n \"outputs\": [\n {\n \"data\": {\n \"text/html\": [\n \"