Ctrl+K

Using IPython Parallel as a joblib backend#

joblib is a tool for running tasks, which includes support for implementing custom parallel backends. IPython defines one such backend, so you can use IPython parallel with joblib.

The simplest way to set this up is a single call:

import ipyparallel as ipp

ipp.register_joblib_backend()

This registers the ‘ipyparallel’ backend with all the defaults.

from joblib import Parallel, delayed

def neg(x):
    return -x

Parallel(backend='ipyparallel')(delayed(neg)(i) for i in range(32))

Using existing profile dir: '/Users/minrk/.ipython/profile_default'
Using existing profile dir: '/Users/minrk/.ipython/profile_default'
Starting 4 engines with <class 'ipyparallel.cluster.launcher.LocalEngineSetLauncher'>

Stopping engine(s): 1630490654
engine set stopped 1630490654: {'0': {'exit_code': 0, 'pid': 14275, 'identifier': '0'}, '1': {'exit_code': 0, 'pid': 14276, 'identifier': '1'}, '2': {'exit_code': 0, 'pid': 14277, 'identifier': '2'}, '3': {'exit_code': 0, 'pid': 14278, 'identifier': '3'}}
Stopping controller
Controller stopped: {'exit_code': 0, 'pid': 14261, 'identifier': 'ipcontroller-1630490653-glt9-14256'}

[0,
 -1,
 -2,
 -3,
 -4,
 -5,
 -6,
 -7,
 -8,
 -9,
 -10,
 -11,
 -12,
 -13,
 -14,
 -15,
 -16,
 -17,
 -18,
 -19,
 -20,
 -21,
 -22,
 -23,
 -24,
 -25,
 -26,
 -27,
 -28,
 -29,
 -30,
 -31]

You can also configure your own View, and register it explicitly, or even as the default:

rc = ipp.Cluster(n=4).start_and_connect_sync()

rc[:].use_cloudpickle()
view = rc.load_balanced_view()
view.register_joblib_backend(make_default=True)

Using existing profile dir: '/Users/minrk/.ipython/profile_default'
Starting 4 engines with <class 'ipyparallel.cluster.launcher.LocalEngineSetLauncher'>

import os
Parallel()(delayed(os.getpid)() for i in range(10))

[14317, 14316, 14315, 14306, 14306, 14306, 14306, 14315, 14306, 14306]

Joblib also provides a context manager for selecting a particular backend:

even = rc.load_balanced_view(targets=rc.ids[::2])
even.register_joblib_backend('even')

from joblib import parallel_backend
with parallel_backend('even'):
    result = Parallel()(delayed(os.getpid)() for i in range(5))
result

[14316, 14306, 14316, 14306, 14316]
Show Source