Ctrl+K

Using widgets in IPython Parallel#

IPython Parallel 7.1 introduces basic support for using Jupyter widgets from a notebook to engines.

This allows things like progress bars for incremental stages, when IPP’s own task-level progress doesn’t show you enough information.

As always, we start by creating and connecting to a cluster:

import ipyparallel as ipp
rc = ipp.Cluster(n=4).start_and_connect_sync()
rc.activate()

Using existing profile dir: '/Users/minrk/.ipython/profile_default'
Starting 4 engines with <class 'ipyparallel.cluster.launcher.LocalEngineSetLauncher'>

<DirectView all>

IPython widgets support updateable readouts of things like progress.

There are lots of widgets to choose from, but for our purposes, we are going to use IntProgress

import time

import ipywidgets as W
from IPython.display import display

progress = W.IntProgress(min=0, max=10, description="Step 0", readout=True)
display(progress)

for i in range(10):
    progress.value += 1
    progress.description = f"Step {progress.value}/{progress.max}"
    time.sleep(0.1)

progress.bar_style = "success" # change color when it's done

IPython Parallel supports progress and interactive waits on AsyncResult objects, which is great when your tasks are small and you have a lot of them:

view = rc.load_balanced_view()
view.map_async(lambda x: x * 2, range(1000)).wait_interactive()

but the unit of progress is each function call.

If you have only a few large IPython tasks, you only get feedback when the whole task is done on a given engine:

%%px --block
import time


def do_some_work(t):
    time.sleep(t)


do_some_work(5)

Combining widgets with tasks allows us to show our progress in stages, including showing progress for every engine.

This way we get to see progress within the task, even though IPython Parallel can’t tell what’s going on within your black-box task

rc[:].scatter("rank", rc.ids, flatten=True)

<AsyncResult: scatter>

%%px --block
import random

import ipywidgets as W
from IPython.display import display

n_steps = 10

progress = W.IntProgress(max=n_steps, description=f"rank {rank}: 0")
def _set_color(change):
    if change.new == progress.max:
        progress.bar_style = "success"
    else:
        progress.bar_style = ""
progress.observe(_set_color, "value")
display(progress)

for i in range(n_steps):
    time.sleep(random.random())
    progress.value += 1
    progress.description = f"rank {rank}: {progress.value}/{progress.max}"

[output:3]

[output:2]

[output:0]

[output:1]

Producing progress bars for every engine can get very unwieldy if you have more than a few engines.

Here’s an example where only rank 0 reports its progress, giving a more succinct presentation, assuming rank 0 knows sufficient information about what’s going on (e.g. an MPI-based simulation).

%%px --block
import random
import time

import ipywidgets as W
from IPython.display import display

n_steps = 10

if rank == 0:
    progress = W.IntProgress(max=n_steps, description="Step 0")
    def _set_color(change):
        if change.new == progress.max:
            progress.bar_style = "success"
        else:
            progress.bar_style = ""
    progress.observe(_set_color, "value")
    display(progress)

for i in range(n_steps):
    time.sleep(random.random())
    if rank == 0:
        progress.value += 1
        progress.description = f"Step {progress.value}/{progress.max}"

print(f"rank {rank}: done!")

[output:0]

[stdout:2] rank 2: done!
[stdout:1] rank 1: done!
[stdout:0] rank 0: done!
[stdout:3] rank 3: done!

Widgets also support two-way communication, using buttons and text areas for input. These also work in IPython Parallel, allowing feedback between engines and a notebook.

%%px
button = W.Button(description=f"step")
progress = W.IntProgress(max=5, description="0 / 5")
count = 0


def click_increment(_):
    progress.value += 1
    progress.description = f"{progress.value} / {progress.max}"
    if progress.value == progress.max:
        button.disabled = True
        progress.bar_style = "success"


button.on_click(click_increment)

display(W.HBox([W.Label(f"Rank {rank}"), button, progress]))

[output:0]

[output:1]

[output:2]

[output:3]

This lets you build interactive displays with parallel execution on remote engines. Useful? Maybe!

Show Source