Using Parallel Magics

IPython has a few magics for working with your engines.

This assumes you have started an IPython cluster, either with the notebook interface, or the ipcluster/controller/engine commands.

[1]:
import ipyparallel as ipp

rc = ipp.Cluster(n=4).start_and_connect_sync()
dv = rc[:]
rc.ids
Using existing profile dir: '/home/docs/.ipython/profile_default'
Starting 4 engines with <class 'ipyparallel.cluster.launcher.LocalEngineSetLauncher'>
100%|██████████| 4/4 [00:08<00:00,  2.06s/engine]
[1]:
[0, 1, 2, 3]
Creating a Client registers the parallel magics %px, %%px, %pxresult, pxconfig, and %autopx.
These magics are initially associated with a DirectView always associated with all currently registered engines.

Now we can execute code remotely with %px:

[2]:
%px a=5
[3]:
%px print(a)
[stdout:3] 5

[stdout:2] 5

[stdout:1] 5

[4]:
%px a
Out[0:3]: 5
Out[1:3]: 5
Out[2:3]: 5
Out[3:3]: 5
[5]:
with dv.sync_imports():
    import sys
importing sys on engine(s)
[6]:
%px from __future__ import print_function
%px print("ERROR", file=sys.stderr)
[stderr:3] ERROR

[stderr:2] ERROR

[stderr:0] ERROR

[stderr:1] ERROR

You don’t have to wait for results. The %pxconfig magic lets you change the default blocking/targets for the %px magics:

[7]:
%pxconfig --noblock
[8]:
%px import time
%px time.sleep(1)
%px time.time()
[8]:
<AsyncResult: %px>

But you will notice that this didn’t output the result of the last command. For this, we have %pxresult, which displays the output of the latest request:

[9]:
%pxresult
Out[0:8]: 1634554417.9787152
Out[1:8]: 1634554417.9747453
Out[2:8]: 1634554417.9819129
Out[3:8]: 1634554417.985727

Remember, an IPython engine is IPython, so you can do magics remotely as well!

[10]:
%pxconfig --block
%px %matplotlib inline
%px:   0%|          | 0/4 [00:03<?, ?tasks/s]
[stderr:2] WARNING:matplotlib.font_manager:Matplotlib is building the font cache; this may take a moment.

%px:   0%|          | 0/4 [00:03<?, ?tasks/s]
[stderr:3] WARNING:matplotlib.font_manager:Matplotlib is building the font cache; this may take a moment.

[stderr:0] WARNING:matplotlib.font_manager:Matplotlib is building the font cache; this may take a moment.

%px:   0%|          | 0/4 [00:03<?, ?tasks/s]
[stderr:1] WARNING:matplotlib.font_manager:Matplotlib is building the font cache; this may take a moment.

%px: 100%|██████████| 4/4 [00:07<00:00,  1.76s/tasks]

%%px can be used to lower the priority of the engines to improve system performance under heavy CPU load.

[11]:
%%px
import psutil
psutil.Process().nice(20 if psutil.POSIX else psutil.IDLE_PRIORITY_CLASS)
[12]:
%%px
import numpy as np
import matplotlib.pyplot as plt

%%px can also be used as a cell magic, for submitting whole blocks. This one acceps --block and --noblock flags to specify the blocking behavior, though the default is unchanged.

[13]:
dv.scatter('id', dv.targets, flatten=True)
dv['stride'] = len(dv)
[14]:
%%px --noblock
x = np.linspace(0,np.pi,1000)
for n in range(id,12, stride):
    print(n)
    plt.plot(x,np.sin(n*x))
plt.title("Plot %i" % id)
[14]:
<AsyncResult: %px>
[15]:
%pxresult
[stdout:0]
0
4
8
[stdout:1]
1
5
9
[stdout:2]
2
6
10
[stdout:3]
3
7
11
[output:0]
../_images/examples_Parallel_Magics_23_2.png
[output:1]
../_images/examples_Parallel_Magics_23_4.png
[output:2]
../_images/examples_Parallel_Magics_23_6.png
[output:3]
../_images/examples_Parallel_Magics_23_8.png
Out[0:12]: Text(0.5, 1.0, 'Plot 0')
Out[1:12]: Text(0.5, 1.0, 'Plot 1')
Out[2:12]: Text(0.5, 1.0, 'Plot 2')
Out[3:12]: Text(0.5, 1.0, 'Plot 3')

It also lets you choose some amount of the grouping of the outputs with --group-outputs:

The choices are:

  • engine - all of an engine’s output is collected together

  • type - where stdout of each engine is grouped, etc. (the default)

  • order - same as type, but individual displaypub outputs are interleaved. That is, it will output the first plot from each engine, then the second from each, etc.

[16]:
%%px --group-outputs=engine
x = np.linspace(0,np.pi,1000)
for n in range(id+1,12, stride):
    print(n)
    plt.figure()
    plt.plot(x,np.sin(n*x))
    plt.title("Plot %i" % n)
[stdout:3] 4
8

[stdout:2] 3
7
11

[stdout:0] 1
5
9

[stdout:1] 2
6
10

[output:3]
../_images/examples_Parallel_Magics_25_5.png
[output:2]
../_images/examples_Parallel_Magics_25_7.png
[output:1]
../_images/examples_Parallel_Magics_25_9.png
[output:3]
../_images/examples_Parallel_Magics_25_11.png
[output:0]
../_images/examples_Parallel_Magics_25_13.png
[output:2]
../_images/examples_Parallel_Magics_25_15.png
[output:0]
../_images/examples_Parallel_Magics_25_17.png
[output:1]
../_images/examples_Parallel_Magics_25_19.png
[output:2]
../_images/examples_Parallel_Magics_25_21.png
[output:0]
../_images/examples_Parallel_Magics_25_23.png
[output:1]
../_images/examples_Parallel_Magics_25_25.png

When you specify ‘order’, then individual display outputs (e.g. plots) will be interleaved.

%pxresult takes the same output-ordering arguments as %%px, so you can view the previous result in a variety of different ways with a few sequential calls to %pxresult:

[17]:
%pxresult --group-outputs=order
[stdout:0]
1
5
9
[stdout:1]
2
6
10
[stdout:2]
3
7
11
[stdout:3]
4
8
[output:0]
../_images/examples_Parallel_Magics_27_2.png
[output:1]
../_images/examples_Parallel_Magics_27_4.png
[output:2]
../_images/examples_Parallel_Magics_27_6.png
[output:0]
../_images/examples_Parallel_Magics_27_8.png
[output:1]
../_images/examples_Parallel_Magics_27_10.png
[output:2]
../_images/examples_Parallel_Magics_27_12.png
[output:0]
../_images/examples_Parallel_Magics_27_14.png
[output:1]
../_images/examples_Parallel_Magics_27_16.png
[output:2]
../_images/examples_Parallel_Magics_27_18.png

Single-engine views

When a DirectView has a single target, the output is a bit simpler (no prefixes on stdout/err, etc.):

[18]:
from __future__ import print_function

def generate_output():
    """function for testing output

    publishes two outputs of each type, and returns something
    """

    import sys,os
    from IPython.display import display, HTML, Math

    print("stdout")
    print("stderr", file=sys.stderr)

    display(HTML("<b>HTML</b>"))

    print("stdout2")
    print("stderr2", file=sys.stderr)

    display(Math(r"\alpha=\beta"))

    return os.getpid()

dv['generate_output'] = generate_output

You can also have more than one set of parallel magics registered at a time.

The View.activate() method takes a suffix argument, which is added to 'px'.

[19]:
e0 = rc[-1]
e0.block = True
e0.activate('0')
[20]:
%px0 generate_output()
[stdout:3] stdout
stdout2

[stderr:3] stderr
stderr2

[output:3]
HTML
[output:3]
$\displaystyle \alpha=\beta$
Out[3:14]: 2555
[21]:
%px generate_output()
[stdout:3] stdout
stdout2

[stdout:0] stdout
stdout2

[stderr:0] stderr
stderr2

[stdout:1] stdout
stdout2

[output:0]
HTML
[stderr:3] stderr
stderr2

[stdout:2] stdout
stdout2

[stderr:2] stderr
stderr2

[stderr:1] stderr
stderr2

[output:2]
HTML
[output:3]
HTML
[output:1]
HTML
[output:0]
$\displaystyle \alpha=\beta$
[output:2]
$\displaystyle \alpha=\beta$
[output:3]
$\displaystyle \alpha=\beta$
[output:1]
$\displaystyle \alpha=\beta$
Out[0:14]: 2548
Out[1:14]: 2550
Out[2:14]: 2552
Out[3:15]: 2555

As mentioned above, we can redisplay those same results with various grouping:

[22]:
%pxresult --group-outputs order
[stdout:0]
stdout
stdout2
[stdout:1]
stdout
stdout2
[stdout:2]
stdout
stdout2
[stdout:3]
stdout
stdout2
[stderr:0]
stderr
stderr2
[stderr:1]
stderr
stderr2
[stderr:2]
stderr
stderr2
[stderr:3]
stderr
stderr2
[output:0]
HTML
[output:1]
HTML
[output:2]
HTML
[output:3]
HTML
[output:0]
$\displaystyle \alpha=\beta$
[output:1]
$\displaystyle \alpha=\beta$
[output:2]
$\displaystyle \alpha=\beta$
[output:3]
$\displaystyle \alpha=\beta$
Out[0:14]: 2548
Out[1:14]: 2550
Out[2:14]: 2552
Out[3:15]: 2555
[23]:
%pxresult --group-outputs engine
[stdout:0]
stdout
stdout2
[stderr:0]
stderr
stderr2
[output:0]
HTML
$\displaystyle \alpha=\beta$
Out[0:14]: 2548
[stdout:1]
stdout
stdout2
[stderr:1]
stderr
stderr2
[output:1]
HTML
$\displaystyle \alpha=\beta$
Out[1:14]: 2550
[stdout:2]
stdout
stdout2
[stderr:2]
stderr
stderr2
[output:2]
HTML
$\displaystyle \alpha=\beta$
Out[2:14]: 2552
[stdout:3]
stdout
stdout2
[stderr:3]
stderr
stderr2
[output:3]
HTML
$\displaystyle \alpha=\beta$
Out[3:15]: 2555

Parallel Exceptions

When you raise exceptions with the parallel exception, the CompositeError raised locally will display your remote traceback.

[24]:
%%px
from numpy.random import random
A = random((100, 100, 'invalid shape'))
[0:execute]:
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/tmp/ipykernel_2548/1064401740.py in <module>
      1 from numpy.random import random
----> 2 A = random((100, 100, 'invalid shape'))

mtrand.pyx in numpy.random.mtrand.RandomState.random()

mtrand.pyx in numpy.random.mtrand.RandomState.random_sample()

_common.pyx in numpy.random._common.double_fill()

TypeError: 'str' object cannot be interpreted as an integer

[1:execute]:
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/tmp/ipykernel_2550/1064401740.py in <module>
      1 from numpy.random import random
----> 2 A = random((100, 100, 'invalid shape'))

mtrand.pyx in numpy.random.mtrand.RandomState.random()

mtrand.pyx in numpy.random.mtrand.RandomState.random_sample()

_common.pyx in numpy.random._common.double_fill()

TypeError: 'str' object cannot be interpreted as an integer

[2:execute]:
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/tmp/ipykernel_2552/1064401740.py in <module>
      1 from numpy.random import random
----> 2 A = random((100, 100, 'invalid shape'))

mtrand.pyx in numpy.random.mtrand.RandomState.random()

mtrand.pyx in numpy.random.mtrand.RandomState.random_sample()

_common.pyx in numpy.random._common.double_fill()

TypeError: 'str' object cannot be interpreted as an integer

[3:execute]:
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/tmp/ipykernel_2555/1064401740.py in <module>
      1 from numpy.random import random
----> 2 A = random((100, 100, 'invalid shape'))

mtrand.pyx in numpy.random.mtrand.RandomState.random()

mtrand.pyx in numpy.random.mtrand.RandomState.random_sample()

_common.pyx in numpy.random._common.double_fill()

TypeError: 'str' object cannot be interpreted as an integer

Remote Cell Magics

Remember, Engines are IPython too, so the cell that is run remotely by %%px can in turn use a cell magic.

[25]:
%%px
%%timeit
from numpy.random import random
from numpy.linalg import norm
A = random((100, 100))
norm(A, 2)
%px:   0%|          | 0/4 [00:02<?, ?tasks/s]
[stdout:3] 5.85 ms ± 81.4 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

%px:  25%|██▌       | 1/4 [00:02<00:00,  9.89tasks/s]
[stdout:0] 6.1 ms ± 242 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

[stdout:2] 6.14 ms ± 302 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

[stdout:1] 6.16 ms ± 240 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

%px: 100%|██████████| 4/4 [00:03<00:00,  1.32tasks/s]

Local Execution

You can instruct %%px to also execute the cell locally. This is useful for interactive definitions, or if you want to load a data source everywhere, not just on the engines.

[26]:
%%px --local
import os
thispid = os.getpid()
print(thispid)
2494
[stdout:0]
2548
[stdout:1]
2550
[stdout:2]
2552
[stdout:3]
2555