cascading.flow.hadoop

## Class MultiMapReduceFlow

- java.lang.Object
- - cascading.flow.BaseFlow<JobConf>
  - - cascading.flow.hadoop.HadoopFlow
    - - cascading.flow.hadoop.BaseMapReduceFlow
      - - cascading.flow.hadoop.MultiMapReduceFlow

All Implemented Interfaces:
  Flow<JobConf>, UnitOfWork<FlowStats>, ProcessLogger

    public class MultiMapReduceFlow
    extends BaseMapReduceFlow

Class MultiMapReduceFlow is a `HadoopFlow` subclass that supports custom MapReduce jobs pre-configured via one or more `JobConf` objects.

Use this class to group multiple JobConf instances together as a single Flow. MultiMapReduceFlow will automatically topologically order the JobConf instances and schedule them on the cluster once `BaseFlow.start()` or `complete()` are called.

If you have a single JobConf instance, see `MapReduceFlow` as a alternative to this class.

This class will not delete any sinks before execution, it is up to the developer to make sure any intermediate and sink paths be removed/deleted before calling `BaseFlow.start()` or `complete()`, otherwise Hadoop will throw an exception.

JobConf instances can be incrementally added at any point before the `complete()` method is called. But they must logically (topologically) come after any previously provided JobConf instances. In practice the Flow will fail if the input source path is missing because a prior JobConf was not provided before the Flow was started.

The ordering is done by comparing the input and output paths of the given JobConf instances. By default, this class only works with JobConf instances that read and write from the Hadoop FileSystem (HDFS) (any path that would work with the `Hfs` Tap.

If the configured JobConf instance uses some other identifier instead of Hadoop FS paths, you should override the `BaseMapReduceFlow.createSources(org.apache.hadoop.mapred.JobConf)`, `BaseMapReduceFlow.createSinks(org.apache.hadoop.mapred.JobConf)`, and `BaseMapReduceFlow.createTraps(org.apache.hadoop.mapred.JobConf)` methods to properly resolve the configured paths into usable `Tap` instances. By default createTraps returns an empty collection and should probably be left alone.

MultiMapReduceFlow supports both org.apache.hadoop.mapred.* and org.apache.hadoop.mapreduce.* API Jobs.

### Nested Class Summary

### Nested classes/interfaces inherited from class cascading.flow.BaseFlow

`BaseFlow.FlowHolder`

### Field Summary

### Fields inherited from class cascading.flow.hadoop.BaseMapReduceFlow

`deleteSinkOnInit`

### Fields inherited from class cascading.flow.BaseFlow

`completed, flowCanonicalHash, flowDescriptor, flowElementGraph, flowStats, flowStepGraph, platformInfo, sinks, sources, steps, stop, stopJobsOnExit, thread, throwable`

### Fields inherited from interface cascading.flow.Flow

`CASCADING_FLOW_ID`

### Fields inherited from interface cascading.util.ProcessLogger

`NULL`

### Constructor Summary

Constructors
`` `MultiMapReduceFlow(java.util.Map<java.lang.Object,java.lang.Object> properties, java.lang.String name, JobConf jobConf, JobConf... jobConfs)` Constructor MultiMapReduceFlow creates a new MultiMapReduceFlow instance.
`` `MultiMapReduceFlow(java.util.Map<java.lang.Object,java.lang.Object> properties, java.lang.String name, java.util.Map<java.lang.String,java.lang.String> flowDescriptor, boolean stopJobsOnExit, JobConf jobConf, JobConf... jobConfs)` Constructor MultiMapReduceFlow creates a new MultiMapReduceFlow instance.
`` `MultiMapReduceFlow(java.util.Map<java.lang.Object,java.lang.Object> properties, java.lang.String name, java.util.Map<java.lang.String,java.lang.String> flowDescriptor, JobConf jobConf, JobConf... jobConfs)` Constructor MultiMapReduceFlow creates a new MultiMapReduceFlow instance.
`protected` `MultiMapReduceFlow(PlatformInfo platformInfo, java.util.Map<java.lang.Object,java.lang.Object> properties, java.lang.String name)` Constructor MultiMapReduceFlow creates a new MultiMapReduceFlow instance.
`protected` `MultiMapReduceFlow(PlatformInfo platformInfo, java.util.Map<java.lang.Object,java.lang.Object> properties, java.lang.String name, java.util.Map<java.lang.String,java.lang.String> flowDescriptor)` Constructor MultiMapReduceFlow creates a new MultiMapReduceFlow instance.
`` `MultiMapReduceFlow(java.lang.String name, JobConf jobConf, JobConf... jobConfs)` Constructor MultiMapReduceFlow creates a new MultiMapReduceFlow instance.

### Method Summary

Methods
`protected void` `addFlowStep(MapReduceFlowStep flowStep)`
`void` `attachFlowStep(JobConf jobConf)`
`protected boolean` `blockingContinuePollingSteps()`
`void` `complete()`
`protected MapReduceFlowStep` `createMapReduceFlowStep(JobConf jobConf)`
`protected Tap` `createTap(JobConf jobConf, Path path, SinkMode sinkMode)`
`protected FlowStepGraph` `getOrCreateFlowStepGraph()`
`protected void` `initializeFrom(java.util.List<JobConf> jobConfs)`
`void` `notifyComplete()`
`protected boolean` `spawnSteps()`
`protected FlowStepGraph` `updateFlowStepGraph(FlowStepGraph flowStepGraph, java.util.Collection<MapReduceFlowStep> flowSteps)`
`protected java.util.Collection<MapReduceFlowStep>` `updateWithFlowSteps(java.util.Collection<MapReduceFlowStep> flowSteps)`

### Methods inherited from class cascading.flow.hadoop.BaseMapReduceFlow

`createFlowStep, createSinks, createSources, createTraps, fileInputToTaps, fileOutputToTaps, makeNameFromPath, makeStepGraph, toSinkTap, toSourceTap`

### Methods inherited from class cascading.flow.hadoop.HadoopFlow

`copyToDistributedCache, getConfig, getConfigAsProperties, getConfigCopy, getFlowProcess, getMaxNumParallelSteps, getProperty, getTotalSliceCPUMilliSeconds, initConfig, initFromProperties, internalClean, internalShutdown, internalStart, isPreserveTemporaryFiles, newConfig, registerHadoopShutdownHook, setConfigProperty, stepsAreLocal`

### Methods inherited from class cascading.flow.BaseFlow

`addListener, addPlannerProperties, addSessionProperties, addStepListener, areSinksStale, areSourcesNewer, cleanup, createConfig, createFlowCanonicalHash, createFlowStats, createFlowThread, createPrepareFlowStats, deleteCheckpointsIfNotUpdate, deleteCheckpointsIfReplace, deleteSinks, deleteSinksIfNotUpdate, deleteSinksIfReplace, deleteTrapsIfNotUpdate, deleteTrapsIfReplace, fireOnCompleted, fireOnStarting, fireOnStopping, fireOnThrowable, fireOnThrowable, getCascadeID, getCascadingServices, getCheckpointNames, getCheckpoints, getCheckpointsCollection, getClassPath, getClientState, getEligibleJobsSize, getFieldsFor, getFlowCanonicalHash, getFlowDescriptor, getFlowElementGraph, getFlowSession, getFlowSkipStrategy, getFlowStats, getFlowStepGraph, getFlowSteps, getFlowStepStrategy, getHolder, getID, getJobMapCallables, getName, getPlannerInfo, getPlatformInfo, getRunID, getSink, getSink, getSinkModified, getSinkNames, getSinks, getSinksCollection, getSource, getSourceNames, getSources, getSourcesCollection, getSpawnStrategy, getStats, getSubmitPriority, getTags, getTrapNames, getTraps, getTrapsCollection, handleExecutorShutdown, hasListeners, hasStepListeners, initialize, initializeChildStats, initializeNewJobsMap, initSteps, internalStopAllJobs, isDebugEnabled, isInfoEnabled, isJobsMapInitialized, isSkipFlow, isStopJobsOnExit, logDebug, logError, logError, logInfo, logWarn, logWarn, logWarn, openSink, openSink, openSource, openSource, openTapForRead, openTapForWrite, openTrap, openTrap, prepare, presentSinkFields, presentSourceFields, registerShutdownHook, removeListener, removeListeners, removeStepListener, resourceExists, retrieveSinkFields, retrieveSourceFields, setCascade, setCheckpoints, setFlowElementGraph, setFlowSkipStrategy, setFlowStepGraph, setFlowStepStrategy, setName, setPlannerInfo, setSinks, setSources, setSpawnStrategy, setSubmitPriority, setTraps, start, stop, toString, updateJobsMap, updateSchemes, writeDOT, writeStepsDOT`

### Methods inherited from class java.lang.Object

`clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait`

### Constructor Detail

#### MultiMapReduceFlow

    public MultiMapReduceFlow(java.lang.String name,
                      JobConf jobConf,
                      JobConf... jobConfs)

Constructor MultiMapReduceFlow creates a new MultiMapReduceFlow instance.

Parameters:
  `name` - of String
  `jobConf` - of JobConf
  `jobConfs` - of JobConf...

#### MultiMapReduceFlow

    public MultiMapReduceFlow(java.util.Map<java.lang.Object,java.lang.Object> properties,
                      java.lang.String name,
                      JobConf jobConf,
                      JobConf... jobConfs)

Constructor MultiMapReduceFlow creates a new MultiMapReduceFlow instance.

Parameters:
  `properties` - of Map<Object, Object>
  `name` - of String
  `jobConf` - of JobConf
  `jobConfs` - of JobConf...

#### MultiMapReduceFlow

    public MultiMapReduceFlow(java.util.Map<java.lang.Object,java.lang.Object> properties,
                      java.lang.String name,
                      java.util.Map<java.lang.String,java.lang.String> flowDescriptor,
                      JobConf jobConf,
                      JobConf... jobConfs)

Constructor MultiMapReduceFlow creates a new MultiMapReduceFlow instance.

Parameters:
  `properties` - of Map<Object, Object>
  `name` - of String
  `flowDescriptor` - of Map<String, String>
  `jobConf` - of JobConf
  `jobConfs` - of JobConf...

#### MultiMapReduceFlow

    public MultiMapReduceFlow(java.util.Map<java.lang.Object,java.lang.Object> properties,
                      java.lang.String name,
                      java.util.Map<java.lang.String,java.lang.String> flowDescriptor,
                      boolean stopJobsOnExit,
                      JobConf jobConf,
                      JobConf... jobConfs)

Constructor MultiMapReduceFlow creates a new MultiMapReduceFlow instance.

Parameters:
  `properties` - of Map<Object, Object>
  `name` - of String
  `flowDescriptor` - of Map<String, String>
  `stopJobsOnExit` - of boolean
  `jobConf` - of JobConf
  `jobConfs` - of JobConf...

#### MultiMapReduceFlow

    protected MultiMapReduceFlow(PlatformInfo platformInfo,
                      java.util.Map<java.lang.Object,java.lang.Object> properties,
                      java.lang.String name)

Constructor MultiMapReduceFlow creates a new MultiMapReduceFlow instance.

Parameters:
  `platformInfo` - of PlatformInfo
  `properties` - of Map<Object, Object>
  `name` - of String

#### MultiMapReduceFlow

    protected MultiMapReduceFlow(PlatformInfo platformInfo,
                      java.util.Map<java.lang.Object,java.lang.Object> properties,
                      java.lang.String name,
                      java.util.Map<java.lang.String,java.lang.String> flowDescriptor)

Constructor MultiMapReduceFlow creates a new MultiMapReduceFlow instance.

Parameters:
  `platformInfo` - of PlatformInfo
  `properties` - of Map<Object, Object>
  `name` - of String
  `flowDescriptor` - of Map<String, String>

### Method Detail

#### initializeFrom

    protected void initializeFrom(java.util.List<JobConf> jobConfs)

#### createMapReduceFlowStep

    protected MapReduceFlowStep createMapReduceFlowStep(JobConf jobConf)

#### notifyComplete

    public void notifyComplete()

#### complete

    public void complete()

Specified by:
  `complete` in interface `Flow<JobConf>`
Specified by:
  `complete` in interface `UnitOfWork<FlowStats>`
Overrides:
  `complete` in class `BaseFlow<JobConf>`

#### spawnSteps

    protected boolean spawnSteps()
                          throws java.lang.InterruptedException,
                                 java.util.concurrent.ExecutionException

Overrides:
  `spawnSteps` in class `BaseFlow<JobConf>`
Throws:
  `java.lang.InterruptedException`
  `java.util.concurrent.ExecutionException`

#### blockingContinuePollingSteps

    protected boolean blockingContinuePollingSteps()

#### createTap

    protected Tap createTap(JobConf jobConf,
                Path path,
                SinkMode sinkMode)

Overrides:
  `createTap` in class `BaseMapReduceFlow`

#### attachFlowStep

    public void attachFlowStep(JobConf jobConf)

#### addFlowStep

    protected void addFlowStep(MapReduceFlowStep flowStep)

#### getOrCreateFlowStepGraph

    protected FlowStepGraph getOrCreateFlowStepGraph()

#### updateWithFlowSteps

    protected java.util.Collection<MapReduceFlowStep> updateWithFlowSteps(java.util.Collection<MapReduceFlowStep> flowSteps)

#### updateFlowStepGraph

    protected FlowStepGraph updateFlowStepGraph(FlowStepGraph flowStepGraph,
                                    java.util.Collection<MapReduceFlowStep> flowSteps)