Using Parallel Magics

IPython has a few magics for working with your engines.

This assumes you have started an IPython cluster, either with the notebook interface, or the ipcluster/controller/engine commands.

[1]:
import ipyparallel as ipp
cluster = ipp.Cluster()
cluster.start_cluster_sync(n=4)
rc = cluster.connect_client_sync()
rc.wait_for_engines(4)
dv = rc[:]
rc.ids
Sending output for ipcontroller-build-14291122-project-31407-ipyparallel-1626869428-q76i-642 to /home/docs/.ipython/profile_default/log/ipcontroller-build-14291122-project-31407-ipyparallel-1626869428-q76i-642.log
Starting 4 engines with <class 'ipyparallel.cluster.launcher.LocalEngineSetLauncher'>
Sending output for 0 to /home/docs/.ipython/profile_default/log/ipengine-build-14291122-project-31407-ipyparallel-1626869428-q76i-1626869429-ud6m-0.log
Sending output for 1 to /home/docs/.ipython/profile_default/log/ipengine-build-14291122-project-31407-ipyparallel-1626869428-q76i-1626869429-ud6m-1.log
Sending output for 2 to /home/docs/.ipython/profile_default/log/ipengine-build-14291122-project-31407-ipyparallel-1626869428-q76i-1626869429-ud6m-2.log
Sending output for 3 to /home/docs/.ipython/profile_default/log/ipengine-build-14291122-project-31407-ipyparallel-1626869428-q76i-1626869429-ud6m-3.log
100%|██████████| 4/4 [00:05<00:00,  1.31s/engine]
[1]:
[0, 1, 2, 3]
Creating a Client registers the parallel magics %px, %%px, %pxresult, pxconfig, and %autopx.
These magics are initially associated with a DirectView always associated with all currently registered engines.

Now we can execute code remotely with %px:

[2]:
%px a=5
[stderr:0] 2021-07-21 12:10:35.209 [IPEngine.0] Handling execute_request: 9e915fe9-3d2be6f32af45e07323638bb_1
[stderr:3] 2021-07-21 12:10:35.219 [IPEngine.3] Handling execute_request: 9e915fe9-3d2be6f32af45e07323638bb_4
[stderr:1] 2021-07-21 12:10:35.210 [IPEngine.1] Handling execute_request: 9e915fe9-3d2be6f32af45e07323638bb_2
[stderr:2] 2021-07-21 12:10:35.213 [IPEngine.2] Handling execute_request: 9e915fe9-3d2be6f32af45e07323638bb_3
[3]:
%px print(a)
[stdout:1] 5
[stdout:2] 5
[stderr:1] 2021-07-21 12:10:35.258 [IPEngine.1] Handling execute_request: 9e915fe9-3d2be6f32af45e07323638bb_6
[stderr:2] 2021-07-21 12:10:35.262 [IPEngine.2] Handling execute_request: 9e915fe9-3d2be6f32af45e07323638bb_7
[4]:
%px a
[stderr:0] 2021-07-21 12:10:35.279 [IPEngine.0] Handling execute_request: 9e915fe9-3d2be6f32af45e07323638bb_9
[stderr:1] 2021-07-21 12:10:35.285 [IPEngine.1] Handling execute_request: 9e915fe9-3d2be6f32af45e07323638bb_10
[stderr:3] 2021-07-21 12:10:35.285 [IPEngine.3] Handling execute_request: 9e915fe9-3d2be6f32af45e07323638bb_12
[stderr:2] 2021-07-21 12:10:35.291 [IPEngine.2] Handling execute_request: 9e915fe9-3d2be6f32af45e07323638bb_11
Out[0:3]: 5
Out[1:3]: 5
Out[2:3]: 5
Out[3:3]: 5
[5]:
with dv.sync_imports():
    import sys
importing sys on engine(s)
[6]:
%px from __future__ import print_function
%px print("ERROR", file=sys.stderr)
[stderr:0] 2021-07-21 12:10:35.749 [IPEngine.0] Handling execute_request: 9e915fe9-3d2be6f32af45e07323638bb_17
[stderr:2] 2021-07-21 12:10:35.754 [IPEngine.2] Handling execute_request: 9e915fe9-3d2be6f32af45e07323638bb_19
[stderr:1] 2021-07-21 12:10:35.754 [IPEngine.1] Handling execute_request: 9e915fe9-3d2be6f32af45e07323638bb_18
[stderr:3] 2021-07-21 12:10:35.759 [IPEngine.3] Handling execute_request: 9e915fe9-3d2be6f32af45e07323638bb_20
[stderr:0]
ERROR
2021-07-21 12:10:35.771 [IPEngine.0] Handling execute_request: 9e915fe9-3d2be6f32af45e07323638bb_21
[stderr:1]
ERROR2021-07-21 12:10:35.773 [IPEngine.1] Handling execute_request: 9e915fe9-3d2be6f32af45e07323638bb_22

[stderr:2]
2021-07-21 12:10:35.777 [IPEngine.2] Handling execute_request: 9e915fe9-3d2be6f32af45e07323638bb_23
ERROR
[stderr:3]
2021-07-21 12:10:35.784 [IPEngine.3] Handling execute_request: 9e915fe9-3d2be6f32af45e07323638bb_24
ERROR

You don’t have to wait for results. The %pxconfig magic lets you change the default blocking/targets for the %px magics:

[7]:
%pxconfig --noblock
[8]:
%px import time
%px time.sleep(1)
%px time.time()
[8]:
<AsyncResult: execute>

But you will notice that this didn’t output the result of the last command. For this, we have %pxresult, which displays the output of the latest request:

[9]:
%pxresult
[stderr:0] 2021-07-21 12:10:36.820 [IPEngine.0] Handling execute_request: 9e915fe9-3d2be6f32af45e07323638bb_33
[stderr:1] 2021-07-21 12:10:36.823 [IPEngine.1] Handling execute_request: 9e915fe9-3d2be6f32af45e07323638bb_34
[stderr:2] 2021-07-21 12:10:36.830 [IPEngine.2] Handling execute_request: 9e915fe9-3d2be6f32af45e07323638bb_35
[stderr:3] 2021-07-21 12:10:36.828 [IPEngine.3] Handling execute_request: 9e915fe9-3d2be6f32af45e07323638bb_36
Out[0:8]: 1626869436.8221579
Out[1:8]: 1626869436.8245614
Out[2:8]: 1626869436.8320296
Out[3:8]: 1626869436.8292444

Remember, an IPython engine is IPython, so you can do magics remotely as well!

[10]:
%pxconfig --block
%px %matplotlib inline
[stderr:0] 2021-07-21 12:10:36.860 [IPEngine.0] Handling execute_request: 9e915fe9-3d2be6f32af45e07323638bb_37
[stderr:1] 2021-07-21 12:10:36.861 [IPEngine.1] Handling execute_request: 9e915fe9-3d2be6f32af45e07323638bb_38
[stderr:3] 2021-07-21 12:10:36.875 [IPEngine.3] Handling execute_request: 9e915fe9-3d2be6f32af45e07323638bb_40
[stderr:2] 2021-07-21 12:10:36.871 [IPEngine.2] Handling execute_request: 9e915fe9-3d2be6f32af45e07323638bb_39
[stderr:0] WARNING:matplotlib.font_manager:Matplotlib is building the font cache; this may take a moment.
[stderr:1] WARNING:matplotlib.font_manager:Matplotlib is building the font cache; this may take a moment.
[stderr:3] WARNING:matplotlib.font_manager:Matplotlib is building the font cache; this may take a moment.
[stderr:2] WARNING:matplotlib.font_manager:Matplotlib is building the font cache; this may take a moment.

%%px can be used to lower the priority of the engines to improve system performance under heavy CPU load.

[11]:
%%px
import psutil
psutil.Process().nice(20 if psutil.POSIX else psutil.IDLE_PRIORITY_CLASS)
[stderr:0] 2021-07-21 12:10:45.352 [IPEngine.0] Handling execute_request: 9e915fe9-3d2be6f32af45e07323638bb_41
[stderr:3] 2021-07-21 12:10:45.356 [IPEngine.3] Handling execute_request: 9e915fe9-3d2be6f32af45e07323638bb_44
[stderr:2] 2021-07-21 12:10:45.355 [IPEngine.2] Handling execute_request: 9e915fe9-3d2be6f32af45e07323638bb_43
[stderr:1] 2021-07-21 12:10:45.364 [IPEngine.1] Handling execute_request: 9e915fe9-3d2be6f32af45e07323638bb_42
[12]:
%%px
import numpy as np
import matplotlib.pyplot as plt
[stderr:0] 2021-07-21 12:10:45.383 [IPEngine.0] Handling execute_request: 9e915fe9-3d2be6f32af45e07323638bb_45
[stderr:1] 2021-07-21 12:10:45.383 [IPEngine.1] Handling execute_request: 9e915fe9-3d2be6f32af45e07323638bb_46
[stderr:3] 2021-07-21 12:10:45.383 [IPEngine.3] Handling execute_request: 9e915fe9-3d2be6f32af45e07323638bb_48
[stderr:2] 2021-07-21 12:10:45.387 [IPEngine.2] Handling execute_request: 9e915fe9-3d2be6f32af45e07323638bb_47

%%px can also be used as a cell magic, for submitting whole blocks. This one acceps --block and --noblock flags to specify the blocking behavior, though the default is unchanged.

[13]:
dv.scatter('id', dv.targets, flatten=True)
dv['stride'] = len(dv)
[14]:
%%px --noblock
x = np.linspace(0,np.pi,1000)
for n in range(id,12, stride):
    print(n)
    plt.plot(x,np.sin(n*x))
plt.title("Plot %i" % id)
[14]:
<AsyncResult: execute>
[15]:
%pxresult
[stdout:0]
0
4
8
[stdout:1]
1
5
9
[stdout:2]
2
6
10
[stdout:3]
3
7
11
[stderr:0] 2021-07-21 12:10:45.446 [IPEngine.0] Handling execute_request: 9e915fe9-3d2be6f32af45e07323638bb_57
[stderr:1] 2021-07-21 12:10:45.451 [IPEngine.1] Handling execute_request: 9e915fe9-3d2be6f32af45e07323638bb_58
[stderr:2] 2021-07-21 12:10:45.452 [IPEngine.2] Handling execute_request: 9e915fe9-3d2be6f32af45e07323638bb_59
[stderr:3] 2021-07-21 12:10:45.447 [IPEngine.3] Handling execute_request: 9e915fe9-3d2be6f32af45e07323638bb_60
[output:0]
../_images/examples_Parallel_Magics_23_3.png
[output:1]
../_images/examples_Parallel_Magics_23_5.png
[output:2]
../_images/examples_Parallel_Magics_23_7.png
[output:3]
../_images/examples_Parallel_Magics_23_9.png
Out[0:12]: Text(0.5, 1.0, 'Plot 0')
Out[1:12]: Text(0.5, 1.0, 'Plot 1')
Out[2:12]: Text(0.5, 1.0, 'Plot 2')
Out[3:12]: Text(0.5, 1.0, 'Plot 3')

It also lets you choose some amount of the grouping of the outputs with --group-outputs:

The choices are:

  • engine - all of an engine’s output is collected together

  • type - where stdout of each engine is grouped, etc. (the default)

  • order - same as type, but individual displaypub outputs are interleaved. That is, it will output the first plot from each engine, then the second from each, etc.

[16]:
%%px --group-outputs=engine
x = np.linspace(0,np.pi,1000)
for n in range(id+1,12, stride):
    print(n)
    plt.figure()
    plt.plot(x,np.sin(n*x))
    plt.title("Plot %i" % n)
[stderr:0]
2021-07-21 12:10:45.922 [IPEngine.0] WARNING | No heartbeat in the last 3500 ms (1 time(s) in a row).
2021-07-21 12:10:46.016 [IPEngine.0] Handling execute_request: 9e915fe9-3d2be6f32af45e07323638bb_61
[stderr:1]
2021-07-21 12:10:45.990 [IPEngine.1] WARNING | No heartbeat in the last 3500 ms (1 time(s) in a row).
2021-07-21 12:10:46.020 [IPEngine.1] Handling execute_request: 9e915fe9-3d2be6f32af45e07323638bb_62
[stderr:3]
2021-07-21 12:10:45.992 [IPEngine.3] WARNING | No heartbeat in the last 3500 ms (1 time(s) in a row).
2021-07-21 12:10:46.018 [IPEngine.3] Handling execute_request: 9e915fe9-3d2be6f32af45e07323638bb_64
[stderr:2] 2021-07-21 12:10:46.022 [IPEngine.2] Handling execute_request: 9e915fe9-3d2be6f32af45e07323638bb_63
[stdout:1]
2
6
10
[stdout:3]
4
8
[stdout:0]
1
5
9
[stdout:2]
3
7
11
[output:3]
../_images/examples_Parallel_Magics_25_3.png
[output:0]
../_images/examples_Parallel_Magics_25_5.png
[output:2]
../_images/examples_Parallel_Magics_25_7.png
[output:3]
../_images/examples_Parallel_Magics_25_9.png
[output:1]
../_images/examples_Parallel_Magics_25_11.png
[output:2]
../_images/examples_Parallel_Magics_25_13.png
[output:0]
../_images/examples_Parallel_Magics_25_15.png
[output:1]
../_images/examples_Parallel_Magics_25_17.png
[output:0]
../_images/examples_Parallel_Magics_25_19.png
[output:2]
../_images/examples_Parallel_Magics_25_21.png
[output:1]
../_images/examples_Parallel_Magics_25_23.png

When you specify ‘order’, then individual display outputs (e.g. plots) will be interleaved.

%pxresult takes the same output-ordering arguments as %%px, so you can view the previous result in a variety of different ways with a few sequential calls to %pxresult:

[17]:
%pxresult --group-outputs=order
[stdout:0]
1
5
9
[stdout:1]
2
6
10
[stdout:2]
3
7
11
[stdout:3]
4
8
[stderr:0]
2021-07-21 12:10:45.922 [IPEngine.0] WARNING | No heartbeat in the last 3500 ms (1 time(s) in a row).
2021-07-21 12:10:46.016 [IPEngine.0] Handling execute_request: 9e915fe9-3d2be6f32af45e07323638bb_61
[stderr:1]
2021-07-21 12:10:45.990 [IPEngine.1] WARNING | No heartbeat in the last 3500 ms (1 time(s) in a row).
2021-07-21 12:10:46.020 [IPEngine.1] Handling execute_request: 9e915fe9-3d2be6f32af45e07323638bb_62
[stderr:2] 2021-07-21 12:10:46.022 [IPEngine.2] Handling execute_request: 9e915fe9-3d2be6f32af45e07323638bb_63
[stderr:3]
2021-07-21 12:10:45.992 [IPEngine.3] WARNING | No heartbeat in the last 3500 ms (1 time(s) in a row).
2021-07-21 12:10:46.018 [IPEngine.3] Handling execute_request: 9e915fe9-3d2be6f32af45e07323638bb_64
[output:0]
../_images/examples_Parallel_Magics_27_3.png
[output:1]
../_images/examples_Parallel_Magics_27_5.png
[output:2]
../_images/examples_Parallel_Magics_27_7.png
[output:0]
../_images/examples_Parallel_Magics_27_9.png
[output:1]
../_images/examples_Parallel_Magics_27_11.png
[output:2]
../_images/examples_Parallel_Magics_27_13.png
[output:0]
../_images/examples_Parallel_Magics_27_15.png
[output:1]
../_images/examples_Parallel_Magics_27_17.png
[output:2]
../_images/examples_Parallel_Magics_27_19.png

Single-engine views

When a DirectView has a single target, the output is a bit simpler (no prefixes on stdout/err, etc.):

[18]:
from __future__ import print_function

def generate_output():
    """function for testing output

    publishes two outputs of each type, and returns something
    """

    import sys,os
    from IPython.display import display, HTML, Math

    print("stdout")
    print("stderr", file=sys.stderr)

    display(HTML("<b>HTML</b>"))

    print("stdout2")
    print("stderr2", file=sys.stderr)

    display(Math(r"\alpha=\beta"))

    return os.getpid()

dv['generate_output'] = generate_output

You can also have more than one set of parallel magics registered at a time.

The View.activate() method takes a suffix argument, which is added to 'px'.

[19]:
e0 = rc[-1]
e0.block = True
e0.activate('0')
[20]:
%px0 generate_output()
[stdout:3] stdout
[stderr:3]
2021-07-21 12:10:47.327 [IPEngine.3] Handling execute_request: 9e915fe9-3d2be6f32af45e07323638bb_69
stderr
[output:3]
HTML
[stdout:3] stdout2
[stderr:3] stderr2
[output:3]
$\displaystyle \alpha=\beta$
Out[3:14]: 700
[21]:
%px generate_output()
[stdout:2] stdout
[stderr:2]
stderr
2021-07-21 12:10:47.352 [IPEngine.2] Handling execute_request: 9e915fe9-3d2be6f32af45e07323638bb_72
[output:2]
HTML
[stdout:2] stdout2
[stderr:2] stderr2
[output:2]
$\displaystyle \alpha=\beta$
[stdout:3] stdout
[stderr:3]
2021-07-21 12:10:47.370 [IPEngine.3] Handling execute_request: 9e915fe9-3d2be6f32af45e07323638bb_73
stderr
[output:3]
HTML
[stdout:3] stdout2
[stderr:3] stderr2
[output:3]
$\displaystyle \alpha=\beta$
[stdout:1] stdout
[stdout:0] stdout
[stderr:0]
2021-07-21 12:10:47.367 [IPEngine.0] Handling execute_request: 9e915fe9-3d2be6f32af45e07323638bb_70
stderr
[output:0]
HTML
[stderr:1]
2021-07-21 12:10:47.359 [IPEngine.1] Handling execute_request: 9e915fe9-3d2be6f32af45e07323638bb_71
stderr
[output:1]
HTML
[stdout:0] stdout2
[stdout:1] stdout2
[stderr:1] stderr2
[stderr:0] stderr2
[output:0]
$\displaystyle \alpha=\beta$
[output:1]
$\displaystyle \alpha=\beta$
Out[0:14]: 693
Out[1:14]: 695
Out[2:14]: 697
Out[3:15]: 700

As mentioned above, we can redisplay those same results with various grouping:

[22]:
%pxresult --group-outputs order
[stdout:0]
stdout
stdout2
[stdout:1]
stdout
stdout2
[stdout:2]
stdout
stdout2
[stdout:3]
stdout
stdout2
[stderr:0]
2021-07-21 12:10:47.367 [IPEngine.0] Handling execute_request: 9e915fe9-3d2be6f32af45e07323638bb_70
stderr
stderr2
[stderr:1]
2021-07-21 12:10:47.359 [IPEngine.1] Handling execute_request: 9e915fe9-3d2be6f32af45e07323638bb_71
stderr
stderr2
[stderr:2]
stderr
2021-07-21 12:10:47.352 [IPEngine.2] Handling execute_request: 9e915fe9-3d2be6f32af45e07323638bb_72
stderr2
[stderr:3]
2021-07-21 12:10:47.370 [IPEngine.3] Handling execute_request: 9e915fe9-3d2be6f32af45e07323638bb_73
stderr
stderr2
[output:0]
HTML
[output:1]
HTML
[output:2]
HTML
[output:3]
HTML
[output:0]
$\displaystyle \alpha=\beta$
[output:1]
$\displaystyle \alpha=\beta$
[output:2]
$\displaystyle \alpha=\beta$
[output:3]
$\displaystyle \alpha=\beta$
Out[0:14]: 693
Out[1:14]: 695
Out[2:14]: 697
Out[3:15]: 700
[23]:
%pxresult --group-outputs engine
[stdout:0]
stdout
stdout2
[stderr:0]
2021-07-21 12:10:47.367 [IPEngine.0] Handling execute_request: 9e915fe9-3d2be6f32af45e07323638bb_70
stderr
stderr2
[output:0]
HTML
$\displaystyle \alpha=\beta$
Out[0:14]: 693
[stdout:1]
stdout
stdout2
[stderr:1]
2021-07-21 12:10:47.359 [IPEngine.1] Handling execute_request: 9e915fe9-3d2be6f32af45e07323638bb_71
stderr
stderr2
[output:1]
HTML
$\displaystyle \alpha=\beta$
Out[1:14]: 695
[stdout:2]
stdout
stdout2
[stderr:2]
stderr
2021-07-21 12:10:47.352 [IPEngine.2] Handling execute_request: 9e915fe9-3d2be6f32af45e07323638bb_72
stderr2
[output:2]
HTML
$\displaystyle \alpha=\beta$
Out[2:14]: 697
[stdout:3]
stdout
stdout2
[stderr:3]
2021-07-21 12:10:47.370 [IPEngine.3] Handling execute_request: 9e915fe9-3d2be6f32af45e07323638bb_73
stderr
stderr2
[output:3]
HTML
$\displaystyle \alpha=\beta$
Out[3:15]: 700

Parallel Exceptions

When you raise exceptions with the parallel exception, the CompositeError raised locally will display your remote traceback.

[24]:
%%px
from numpy.random import random
A = random((100, 100, 'invalid shape'))
[stderr:2] 2021-07-21 12:10:47.472 [IPEngine.2] Handling execute_request: 9e915fe9-3d2be6f32af45e07323638bb_76
[stderr:3] 2021-07-21 12:10:47.475 [IPEngine.3] Handling execute_request: 9e915fe9-3d2be6f32af45e07323638bb_77
[stderr:0] 2021-07-21 12:10:47.475 [IPEngine.0] Handling execute_request: 9e915fe9-3d2be6f32af45e07323638bb_74
[stderr:1] 2021-07-21 12:10:47.475 [IPEngine.1] Handling execute_request: 9e915fe9-3d2be6f32af45e07323638bb_75
[stderr:1]
2021-07-21 12:10:47.810 [IPEngine.1] Exception in execute request:
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/tmp/ipykernel_695/1064401740.py in <module>
      1 from numpy.random import random
----> 2 A = random((100, 100, 'invalid shape'))

mtrand.pyx in numpy.random.mtrand.RandomState.random()

mtrand.pyx in numpy.random.mtrand.RandomState.random_sample()

_common.pyx in numpy.random._common.double_fill()

TypeError: 'str' object cannot be interpreted as an integer
[stderr:0]
2021-07-21 12:10:47.808 [IPEngine.0] Exception in execute request:
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/tmp/ipykernel_693/1064401740.py in <module>
      1 from numpy.random import random
----> 2 A = random((100, 100, 'invalid shape'))

mtrand.pyx in numpy.random.mtrand.RandomState.random()

mtrand.pyx in numpy.random.mtrand.RandomState.random_sample()

_common.pyx in numpy.random._common.double_fill()

TypeError: 'str' object cannot be interpreted as an integer
[stderr:3]
2021-07-21 12:10:47.819 [IPEngine.3] Exception in execute request:
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/tmp/ipykernel_700/1064401740.py in <module>
      1 from numpy.random import random
----> 2 A = random((100, 100, 'invalid shape'))

mtrand.pyx in numpy.random.mtrand.RandomState.random()

mtrand.pyx in numpy.random.mtrand.RandomState.random_sample()

_common.pyx in numpy.random._common.double_fill()

TypeError: 'str' object cannot be interpreted as an integer
[stderr:2]
2021-07-21 12:10:47.823 [IPEngine.2] Exception in execute request:
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/tmp/ipykernel_697/1064401740.py in <module>
      1 from numpy.random import random
----> 2 A = random((100, 100, 'invalid shape'))

mtrand.pyx in numpy.random.mtrand.RandomState.random()

mtrand.pyx in numpy.random.mtrand.RandomState.random_sample()

_common.pyx in numpy.random._common.double_fill()

TypeError: 'str' object cannot be interpreted as an integer
[0:execute]:
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/tmp/ipykernel_693/1064401740.py in <module>
      1 from numpy.random import random
----> 2 A = random((100, 100, 'invalid shape'))

mtrand.pyx in numpy.random.mtrand.RandomState.random()

mtrand.pyx in numpy.random.mtrand.RandomState.random_sample()

_common.pyx in numpy.random._common.double_fill()

TypeError: 'str' object cannot be interpreted as an integer

[1:execute]:
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/tmp/ipykernel_695/1064401740.py in <module>
      1 from numpy.random import random
----> 2 A = random((100, 100, 'invalid shape'))

mtrand.pyx in numpy.random.mtrand.RandomState.random()

mtrand.pyx in numpy.random.mtrand.RandomState.random_sample()

_common.pyx in numpy.random._common.double_fill()

TypeError: 'str' object cannot be interpreted as an integer

[2:execute]:
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/tmp/ipykernel_697/1064401740.py in <module>
      1 from numpy.random import random
----> 2 A = random((100, 100, 'invalid shape'))

mtrand.pyx in numpy.random.mtrand.RandomState.random()

mtrand.pyx in numpy.random.mtrand.RandomState.random_sample()

_common.pyx in numpy.random._common.double_fill()

TypeError: 'str' object cannot be interpreted as an integer

[3:execute]:
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/tmp/ipykernel_700/1064401740.py in <module>
      1 from numpy.random import random
----> 2 A = random((100, 100, 'invalid shape'))

mtrand.pyx in numpy.random.mtrand.RandomState.random()

mtrand.pyx in numpy.random.mtrand.RandomState.random_sample()

_common.pyx in numpy.random._common.double_fill()

TypeError: 'str' object cannot be interpreted as an integer

Remote Cell Magics

Remember, Engines are IPython too, so the cell that is run remotely by %%px can in turn use a cell magic.

[25]:
%%px
%%timeit
from numpy.random import random
from numpy.linalg import norm
A = random((100, 100))
norm(A, 2)
[stderr:1]
2021-07-21 12:10:47.817 [IPEngine.1] Finishing abort
2021-07-21 12:10:47.836 [IPEngine.1] Handling execute_request: 9e915fe9-3d2be6f32af45e07323638bb_79
[stderr:0]
2021-07-21 12:10:47.813 [IPEngine.0] Finishing abort
2021-07-21 12:10:47.841 [IPEngine.0] Handling execute_request: 9e915fe9-3d2be6f32af45e07323638bb_78
[stderr:3]
2021-07-21 12:10:47.823 [IPEngine.3] Finishing abort
2021-07-21 12:10:47.837 [IPEngine.3] Handling execute_request: 9e915fe9-3d2be6f32af45e07323638bb_81
[stderr:2]
2021-07-21 12:10:47.827 [IPEngine.2] Finishing abort
2021-07-21 12:10:47.837 [IPEngine.2] Handling execute_request: 9e915fe9-3d2be6f32af45e07323638bb_80
[stdout:2] 5.99 ms ± 92.3 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
[stdout:0] 6.02 ms ± 78.6 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
[stdout:3] 6 ms ± 72.6 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
[stdout:1] 6.06 ms ± 95.6 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

Local Execution

You can instruct %%px to also execute the cell locally. This is useful for interactive definitions, or if you want to load a data source everywhere, not just on the engines.

[26]:
%%px --local
import os
thispid = os.getpid()
print(thispid)
642
[stdout:0] 693
[stdout:1] 695
[stdout:2] 697
[stdout:3] 700
[stderr:0] 2021-07-21 12:10:52.766 [IPEngine.0] Handling execute_request: 9e915fe9-3d2be6f32af45e07323638bb_82
[stderr:1] 2021-07-21 12:10:52.762 [IPEngine.1] Handling execute_request: 9e915fe9-3d2be6f32af45e07323638bb_83
[stderr:2] 2021-07-21 12:10:52.765 [IPEngine.2] Handling execute_request: 9e915fe9-3d2be6f32af45e07323638bb_84
[stderr:3] 2021-07-21 12:10:52.768 [IPEngine.3] Handling execute_request: 9e915fe9-3d2be6f32af45e07323638bb_85