Using IPython Parallel as a joblib backend

joblib 0.10 adds the ability to register a custom parallel backend. IPython 5.1 defines one such backend, so you can use IPython parallel with joblib.

The simplest way to set this up is a single call:

[1]:
import ipyparallel as ipp
ipp.register_joblib_backend()

This registers the ‘ipyparallel’ backend with all the defaults.

[2]:
from joblib import Parallel, delayed

def neg(x):
    return -x

Parallel(backend='ipyparallel')(delayed(neg)(i) for i in range(32))
[2]:
[0,
 -1,
 -2,
 -3,
 -4,
 -5,
 -6,
 -7,
 -8,
 -9,
 -10,
 -11,
 -12,
 -13,
 -14,
 -15,
 -16,
 -17,
 -18,
 -19,
 -20,
 -21,
 -22,
 -23,
 -24,
 -25,
 -26,
 -27,
 -28,
 -29,
 -30,
 -31]

You can also configure your own View, and register it explicitly, or even as the default:

[3]:
import ipyparallel as ipp
rc = ipp.Client()
rc[:].use_cloudpickle()
view = rc.load_balanced_view()
view.register_joblib_backend(make_default=True)
[4]:
import os
Parallel()(delayed(os.getpid)() for i in range(10))
[4]:
[10625, 10621, 10647, 10637, 10622, 10620, 10642, 10631, 10625, 10621]

Joblib also provides a context manager for selecting a particular backend:

[5]:
even = rc.load_balanced_view(targets=rc.ids[::2])
even.register_joblib_backend('even')

from joblib import parallel_backend
with parallel_backend('even'):
    result = Parallel()(delayed(os.getpid)() for i in range(5))
result
[5]:
[10622, 10620, 10642, 10631, 10622]