.options.future = list(conditions = NULL) would
be ignored as if it would never had been specified.Add %dofuture% operator, which can be used like
%dopar%, but without the need for
registerDoFuture(),
e.g. y <- foreach(x = 1:3) %dofuture% { slow_fcn(x) }.
One advantage, contrary to using %dopar%, is that a
developer then has full control on foreach(); with
%dopar% the exact behavior depends also on what
%dopar% adapter the user has registered. Another advantage
is that %dofuture% can hand over random number generation,
identification of globals, and error handling to the future ecosystem,
which result in a more predictable and concise behavior, similar to that
provided by future.apply and
furrr.
Now future operators such as
%stdout% and %conditions% can be used to
control the corresponding options.future arguments,
e.g. y <- foreach(i = 1:3) %dopar% { my_fun(i) } %stdout% FALSE
is the same as
y <- foreach(i = 1:3, .options.future = list(stdout = FALSE)) %dopar% { my_fun(i) }.
Add withDoRNG() for evaluating a foreach
%dopar% expression with doRNG::registerDoRNG()
temporarily set.
options(doFuture.rng.onMisuse = "ignore") is now a tad
faster than before.registerDoFuture() with popular packages
(e.g. BiocParallel, foreach,
plyr, caret, glmnet,
NSP, and TSP) relying on
foreach to a separate
doFuture.tests.extra package. This reduced the size of
the package tarball from 140 kB to 40 kB.Now registerDoFuture() returns the previously set
foreach backend, making it possible to reset the the foreach backend to
the previous settings.
Now doFuture recognizes when it is called via the BiocParallel package in which case it skips the check whether or not RNG was used by mistake.
Add option doFuture.rng.onMisuse which can be used
to temporarily override option future.rng.onMisuse when the
doFuture adaptor is running.
Add option doFuture.workarounds, which can be set by
environment variable R_DOFUTURE_WORKAROUNDS when the
package is loaded.
Adding "BiocParallel.DoParam.errors" to option
doFuture.workarounds will prefix
RngFutureError messages with
"task <index> failed - " in order for such errors to
be recognized by the BiocParallel DoParam
backend.
foreach() argument
.options.snow = list(preschedule = <logical>) is
now acknowledged as a fallback to analogous argument
.options.multicore - two arguments defined by the
doMC and the doSNOW packages and also
used by the doParallel package. As before, argument
.options.future will always take precedence if
list(scheduling = <logical>/<numeric>) or
list(chunk.size = <integer>) is given.
Warnings and errors produced when using the RNG without using
%dorng% of the doRNG package are now
tailored to the doFuture package.
foreach() argument .noexport was
completely ignored by doFuture.doFuture.foreach.export values
"automatic" and "automatic-unless-.export".
They were made defunct in doFuture 0.8.2.%dorng% of the doRNG package no longer
produces a warning on ‘Foreach loop had changed the current RNG type:
RNG was restored to same type, next state’ when using the
doFuture adapter.chunk_size,
which now has been corrected to mention chunk.size.Now doFuture sets a label on each future that
reflects its name and the index of the chunk,
e.g. "doFuture-3".
doFuture will now detect when doRNG is in use allowing underlying futures to skip the test of incorrectly generated random numbers - an optional validation of parallel RNG that will be added to future (>= 1.16.0).
.options.future = list(chunk.size = <count>) to
foreach().doFuture.foreach.export values
automatic-unless-.export and automatic are
defunct. They have been deprecated since doFuture
0.7.0..Random.seed to NULL, instead of
removing it, which in turn would produce a warning on “.Random.seed is
not an integer vector but of type NULL, so ignored” when
the next random number generated._R_CHECK_LENGTH_1_LOGIC2_=true bug. This bug did
not affect how the package worked or any of its results.The foreach() argument .options.future
(a named list) can already be used to control whether “chunking” should
take place or not, and if so, how much,
e.g. .options.future = list(scheduling = 2.0)). As an
alternative to scheduling, this can now be specified by
chunk.size - the number of elements processed per future
(“chunk”). In R 3.5.0, the parallel package introduced
argument chunk.size.
Elements can be processed in random order by setting attribute
ordering of .options.future elements
chunk.size or scheduling,
e.g. .options.future = list(chunk.size = structure(TRUE, ordering = "random")).
This improve load balancing in cases where there is a correlation
between processing time and ordering of the elements. Note that the
order of the returned values is not affected when randomizing the
processing order.
Passing argument
.options.future = list(stdout = ...) can be used to to
control how standard output should be relayed. See
?future::Future for further details. Analogously,
.options.future = list(conditions = ...) can be used to
control how messages and warnings are relayed, if at all.
Debug messages are now prepended with a timestamp.
foreach() iterates over,
e.g. foreach(f = F, g = G) %dopar% { f() + g() }.The maximum total size of globals allowed (option
future.globals.maxSize) per future (“chunk”) is now scaled
up by the number of elements processed by the future (“chunk”) making
the protection approximately invariant to the amount of chunking done by
foreach.
Added support for option
doFuture.foreach.export = "manual", which will strictly
follow argument .export of foreach() for
identifying global variables. None of the future +
globals framework for identifying global variables will
be used. This is useful for asserting that the .export
argument is correct.
help("doFuture") and updated
example("doFuture") with a best-practices RNG example.TESTS: The opt-in tests for third-party packages now run their
examples with example(..., run.dontrun = TRUE) to cover
even more use cases.
TESTS: Added future.callr to the backend opt-in tested with plyr.
If the doFuture package is missing on a worker, an error on “length(results) == nbr_of_elements is not TRUE” would be produced. Now a more informative error is produced.
foreach(..., .export) with .export
containing "..." would produce an error when using
globals (<= 0.11.0).
foreach() would not relay captured conditions as
provided by future (>= 0.11.0).
Previously deprecated option
doFuture.foreach.nullexport is defunct.
Option doFuture.foreach.export values
automatic-unless-.export and automatic are
defunct and will fall back to
export-and-automatic.
doFuture now respects option
future.globals.resolve instead of being hardcoded to always
resolve globals (future.globals.resolve = TRUE). This makes
doFuture consistent with other future
frontends.
Added option doFuture.foreach.export making it
possible ignore a faulty .export argument to
foreach() and instead rely on the future framework to
identify globals. For instance, all examples of caret
6.0-77 works with doFuture and any backend when setting
this option to "automatic" (or
".export-and-automatic") whereas they will only work on
forked backends if using ".export" or the default
"automatic-unless-.export". If using
".export-and-automatic-with-warning", a warning that lists
globals potentially missing from the .export argument is
produced
foreach()
code.help("doFuture.options").TESTS: The doFuture package gained more opt-in tests for third-party packages across all known future backends. These tests are not performed on the CRAN servers; instead they are performed on Travis CI. Third-party packages that are currently tested are: caret, foreach, glmnet, NMF, plyr, and TSP.
TESTS: Testing global functions that call themselves recursively.
foreach() option to control whether scheduling
(“chunking”) should take place or not, and if so, how granular it should
be. This is specified as
foreach(..., .options.future = list(scheduling = <value>)).
With scheduling = 1.0 (or equivalently
scheduling = TRUE), the the elements (iterations) will be
split up in equally sized chunks such that each backend worker will
process exactly one chunk. With scheduling = Inf (or
equivalently scheduling = FALSE), chunking is disabled,
i.e. each worker process exactly one element at the time. If
scheduling = 0.0, then a single workers processes all
elements (and the other workers will not be used). If 2.0,
then each worker will process two chunks, and so on. If above option is
not set, then
.options.multicore = list(preschedule = <logical>))
which is defined by doParallel, is used to mean
scheduling = preschedule. If that is not set, then
scheduling = 1.0 is used by default..parallel = TRUE. The default is to test
against all of the future strategies that comes with the
future package, but it is possible to also test against
future.BatchJobs, future.batchtools,
and so on. These tests are performed on all possible future backends
before each release (as well as via continuous integration).Using a nested foreach() call would incorrectly
produce an error on not being able to locate the iterator variable of
the inner-most foreach() as a global variable.
If a foreach() call would result in an error, the
error thrown would report on “object ‘expr’ not found” and not the
actually error message.
%dopar% backend now processes all elements in chunks such
that each backend worker will process a subset of data at once (and only
once). This significantly speeds up processing time when iterating over
a large number of elements that each has short a processing time.Now the package tests future.batchtools with
foreach by itself, in combination with
plyr (parallel = TRUE) as well as with
BiocParallel::bplapply() and friends. Similar tests are
already done using future.BatchJobs.
Added test for foreach::times() %dopar% { ... }.
Especially, it is now tested that global variables are properly
identified. Note that times() does not allow you to specify
neither .export nor .packages so it is not
really designed for processing in external R process. Having said this,
times() does indeed work also in those cases when used with
doFuture because it internally handles this
automatically.
ROBUSTNESS: The package redundancy tests (not run by
R CMD check; needed to be run manually for now) that run 89
plyr examples with the doFuture
foreach adapter, now forces testing of
.parallel = TRUE for all plyr functions
that support that argument. Each example is run across various future
strategies, including ‘sequential’, ‘multicore’, ‘multisession’, and
‘cluster’, as well as ‘batchjobs_local’ and ‘batchtools_local’, if
installed. See doFuture 0.2.0 notes below for how to
run these tests.
.export of foreach() is
acknowledged such that if a character vector of variables names to be
exported is specified, then those variables and nothing else are
exported to the future. If NULL, then automatic lookup of global
variables is used.%dopar% calls, because doFuture forgot to
remind foreach that doFuture should be
used also deeper down. Thank you Alex Vorobiev for reporting on
this.ROBUSTNESS: Added package redundancy tests that runs all examples
of the foreach and the plyr packages
using doFuture and all known types of futures. These
tests are not package tests and need to be run manually. The test
scripts are available in package directory
path <- system.file("tests2", package="doFuture") and
can be run as
source(file.path(path, "plyr", "examples.R")).
ROBUSTNESS: Added package tests validating foreach()
on regular as well as future.BatchJobs futures. Same
for plyr and BiocParallel apply
functions.
help("doFuture").foreach::getDoParWorkers() gives useful information
with registerDoFuture() in most cases. In cases where the
number of workers cannot be inferred easily from
future::plan() it will default to returning a large number
(= 99).foreach::getDoParName() and
foreach::getDoParVersion() gives useful information with
registerDoFuture().