cascading.flow.hadoop

## Class MapReduceFlow

- java.lang.Object
- - cascading.flow.BaseFlow<JobConf>
  - - cascading.flow.hadoop.HadoopFlow
    - - cascading.flow.hadoop.BaseMapReduceFlow
      - - cascading.flow.hadoop.MapReduceFlow

All Implemented Interfaces:
  Flow<JobConf>, UnitOfWork<FlowStats>, ProcessLogger

    public class MapReduceFlow
    extends BaseMapReduceFlow

Class MapReduceFlow is a `HadoopFlow` subclass that supports custom MapReduce jobs pre-configured via the `JobConf` object.

Use this class to allow custom MapReduce jobs to participate in the `Cascade` scheduler. If other Flow instances in the Cascade share resources with this Flow instance, all participants will be scheduled according to their dependencies (topologically).

Set the parameter `deleteSinkOnInit` to `true` if the outputPath in the jobConf should be deleted before executing the MapReduce job.

MapReduceFlow assumes the underlying input and output paths are compatible with the `Hfs` Tap.

If the configured JobConf instance uses some other identifier instead of Hadoop FS paths, you should override the `BaseMapReduceFlow.createSources(org.apache.hadoop.mapred.JobConf)`, `BaseMapReduceFlow.createSinks(org.apache.hadoop.mapred.JobConf)`, and `BaseMapReduceFlow.createTraps(org.apache.hadoop.mapred.JobConf)` methods to properly resolve the configured paths into usable `Tap` instances. By default createTraps returns an empty collection and should probably be left alone.

MapReduceFlow supports both org.apache.hadoop.mapred.* and org.apache.hadoop.mapreduce.* API Jobs.

### Nested Class Summary

### Nested classes/interfaces inherited from class cascading.flow.BaseFlow

`BaseFlow.FlowHolder`

### Field Summary

### Fields inherited from class cascading.flow.hadoop.BaseMapReduceFlow

`deleteSinkOnInit`

### Fields inherited from class cascading.flow.BaseFlow

`completed, flowCanonicalHash, flowDescriptor, flowElementGraph, flowStats, flowStepGraph, platformInfo, sinks, sources, steps, stop, stopJobsOnExit, thread, throwable`

### Fields inherited from interface cascading.flow.Flow

`CASCADING_FLOW_ID`

### Fields inherited from interface cascading.util.ProcessLogger

`NULL`

### Constructor Summary

Constructors
Constructor and Description
`MapReduceFlow(JobConf jobConf)` Constructor MapReduceFlow creates a new MapReduceFlow instance.
`MapReduceFlow(JobConf jobConf, boolean deleteSinkOnInit)` Constructor MapReduceFlow creates a new MapReduceFlow instance.
`MapReduceFlow(java.util.Properties properties, java.lang.String name, JobConf jobConf, boolean deleteSinkOnInit)` Constructor MapReduceFlow creates a new MapReduceFlow instance.
`MapReduceFlow(java.util.Properties properties, java.lang.String name, JobConf jobConf, java.util.Map<java.lang.String,java.lang.String> flowDescriptor, boolean deleteSinkOnInit)` Constructor MapReduceFlow creates a new MapReduceFlow instance.
`MapReduceFlow(java.util.Properties properties, java.lang.String name, JobConf jobConf, java.util.Map<java.lang.String,java.lang.String> flowDescriptor, boolean deleteSinkOnInit, boolean stopJobsOnExit)` Constructor MapReduceFlow creates a new MapReduceFlow instance.
`MapReduceFlow(java.lang.String name, JobConf jobConf)` Constructor MapReduceFlow creates a new MapReduceFlow instance.
`MapReduceFlow(java.lang.String name, JobConf jobConf, boolean deleteSinkOnInit)` Constructor MapReduceFlow creates a new MapReduceFlow instance.

### Method Summary

Methods
`protected void` `initializeFrom(JobConf jobConf)`

### Methods inherited from class cascading.flow.hadoop.BaseMapReduceFlow

`createFlowStep, createSinks, createSources, createTap, createTraps, fileInputToTaps, fileOutputToTaps, makeNameFromPath, makeStepGraph, toSinkTap, toSourceTap`

### Methods inherited from class cascading.flow.hadoop.HadoopFlow

`copyToDistributedCache, getConfig, getConfigAsProperties, getConfigCopy, getFlowProcess, getMaxNumParallelSteps, getProperty, getTotalSliceCPUMilliSeconds, initConfig, initFromProperties, internalClean, internalShutdown, internalStart, isPreserveTemporaryFiles, newConfig, registerHadoopShutdownHook, setConfigProperty, stepsAreLocal`

### Methods inherited from class cascading.flow.BaseFlow

`addListener, addPlannerProperties, addSessionProperties, addStepListener, areSinksStale, areSourcesNewer, cleanup, complete, createConfig, createFlowCanonicalHash, createFlowStats, createFlowThread, createPrepareFlowStats, deleteCheckpointsIfNotUpdate, deleteCheckpointsIfReplace, deleteSinks, deleteSinksIfNotUpdate, deleteSinksIfReplace, deleteTrapsIfNotUpdate, deleteTrapsIfReplace, fireOnCompleted, fireOnStarting, fireOnStopping, fireOnThrowable, fireOnThrowable, getCascadeID, getCascadingServices, getCheckpointNames, getCheckpoints, getCheckpointsCollection, getClassPath, getClientState, getEligibleJobsSize, getFieldsFor, getFlowCanonicalHash, getFlowDescriptor, getFlowElementGraph, getFlowSession, getFlowSkipStrategy, getFlowStats, getFlowStepGraph, getFlowSteps, getFlowStepStrategy, getHolder, getID, getJobMapCallables, getName, getPlannerInfo, getPlatformInfo, getRunID, getSink, getSink, getSinkModified, getSinkNames, getSinks, getSinksCollection, getSource, getSourceNames, getSources, getSourcesCollection, getSpawnStrategy, getStats, getSubmitPriority, getTags, getTrapNames, getTraps, getTrapsCollection, handleExecutorShutdown, hasListeners, hasStepListeners, initialize, initializeChildStats, initializeNewJobsMap, initSteps, internalStopAllJobs, isDebugEnabled, isInfoEnabled, isJobsMapInitialized, isSkipFlow, isStopJobsOnExit, logDebug, logError, logError, logInfo, logWarn, logWarn, logWarn, openSink, openSink, openSource, openSource, openTapForRead, openTapForWrite, openTrap, openTrap, prepare, presentSinkFields, presentSourceFields, registerShutdownHook, removeListener, removeListeners, removeStepListener, resourceExists, retrieveSinkFields, retrieveSourceFields, setCascade, setCheckpoints, setFlowElementGraph, setFlowSkipStrategy, setFlowStepGraph, setFlowStepStrategy, setName, setPlannerInfo, setSinks, setSources, setSpawnStrategy, setSubmitPriority, setTraps, spawnSteps, start, stop, toString, updateJobsMap, updateSchemes, writeDOT, writeStepsDOT`

### Methods inherited from class java.lang.Object

`clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait`

### Constructor Detail

#### MapReduceFlow

    @ConstructorProperties(value="jobConf")
    public MapReduceFlow(JobConf jobConf)

Constructor MapReduceFlow creates a new MapReduceFlow instance.

Parameters:
  `jobConf` - of type JobConf

#### MapReduceFlow

    @ConstructorProperties(value={"jobConf","deleteSinkOnInit"})
    public MapReduceFlow(JobConf jobConf,
                                       boolean deleteSinkOnInit)

Constructor MapReduceFlow creates a new MapReduceFlow instance.

Parameters:
  `jobConf` - of type JobConf
  `deleteSinkOnInit` - of type boolean

#### MapReduceFlow

    @ConstructorProperties(value={"name","jobConf"})
    public MapReduceFlow(java.lang.String name,
                                       JobConf jobConf)

Constructor MapReduceFlow creates a new MapReduceFlow instance.

Parameters:
  `name` - of type String
  `jobConf` - of type JobConf

#### MapReduceFlow

    @ConstructorProperties(value={"name","jobConf","deleteSinkOnInit"})
    public MapReduceFlow(java.lang.String name,
                                       JobConf jobConf,
                                       boolean deleteSinkOnInit)

Constructor MapReduceFlow creates a new MapReduceFlow instance.

Parameters:
  `name` - of type String
  `jobConf` - of type JobConf
  `deleteSinkOnInit` - of type boolean

#### MapReduceFlow

    @ConstructorProperties(value={"properties","name","jobConf","deleteSinkOnInit"})
    public MapReduceFlow(java.util.Properties properties,
                                       java.lang.String name,
                                       JobConf jobConf,
                                       boolean deleteSinkOnInit)

Constructor MapReduceFlow creates a new MapReduceFlow instance.

Parameters:
  `properties` - of type Properties
  `name` - of type String
  `jobConf` - of type JobConf
  `deleteSinkOnInit` - of type boolean

#### MapReduceFlow

    @ConstructorProperties(value={"properties","name","jobConf","flowDescriptor","deleteSinkOnInit"})
    public MapReduceFlow(java.util.Properties properties,
                                       java.lang.String name,
                                       JobConf jobConf,
                                       java.util.Map<java.lang.String,java.lang.String> flowDescriptor,
                                       boolean deleteSinkOnInit)

Constructor MapReduceFlow creates a new MapReduceFlow instance.

Parameters:
  `properties` - of type Properties
  `name` - of type String
  `jobConf` - of type JobConf
  `flowDescriptor` - of type Map<String, String>
  `deleteSinkOnInit` - of type boolean

#### MapReduceFlow

    @ConstructorProperties(value={"properties","name","jobConf","flowDescriptor","deleteSinkOnInit","stopJobsOnExit"})
    public MapReduceFlow(java.util.Properties properties,
                                       java.lang.String name,
                                       JobConf jobConf,
                                       java.util.Map<java.lang.String,java.lang.String> flowDescriptor,
                                       boolean deleteSinkOnInit,
                                       boolean stopJobsOnExit)

Constructor MapReduceFlow creates a new MapReduceFlow instance.

Parameters:
  `properties` - of type Properties
  `name` - of type String
  `jobConf` - of type JobConf
  `flowDescriptor` - of type Map<String, String>
  `deleteSinkOnInit` - of type boolean
  `stopJobsOnExit` - of type boolean

### Method Detail

#### initializeFrom

    protected void initializeFrom(JobConf jobConf)