cascading.flow.hadoop

## Class MapReduceFlow

- java.lang.Object
- - cascading.flow.BaseFlow<org.apache.hadoop.mapred.JobConf>
  - - cascading.flow.hadoop.HadoopFlow
    - - cascading.flow.hadoop.MapReduceFlow

All Implemented Interfaces:
  Flow<org.apache.hadoop.mapred.JobConf>, UnitOfWork<FlowStats>, ProcessLogger

    public class MapReduceFlow
    extends HadoopFlow

Class MapReduceFlow is a `HadoopFlow` subclass that supports custom MapReduce jobs pre-configured via the `JobConf` object.

Use this class to allow custom MapReduce jobs to participate in the `Cascade` scheduler. If other Flow instances in the Cascade share resources with this Flow instance, all participants will be scheduled according to their dependencies (topologically).

Set the parameter `deleteSinkOnInit` to `true` if the outputPath in the jobConf should be deleted before executing the MapReduce job.

MapReduceFlow assumes the underlying input and output paths are compatible with the `Hfs` Tap.

If the configured JobConf instance uses some other identifier instead of Hadoop FS paths, you should override the `createSources(org.apache.hadoop.mapred.JobConf)`, `createSinks(org.apache.hadoop.mapred.JobConf)`, and `createTraps(org.apache.hadoop.mapred.JobConf)` methods to properly resolve the configured paths into usable `Tap` instances. By default createTraps returns an empty collection and should probably be left alone.

MapReduceFlow supports both org.apache.hadoop.mapred.* and org.apache.hadoop.mapreduce.* API Jobs.

### Nested Class Summary

### Nested classes/interfaces inherited from class cascading.flow.BaseFlow

`BaseFlow.FlowHolder`

### Field Summary

Fields
`protected boolean` `deleteSinkOnInit` Field deleteSinkOnInit

### Fields inherited from class cascading.flow.BaseFlow

`flowCanonicalHash, flowDescriptor, flowElementGraph, flowStats, flowStepGraph, platformInfo, sinks, sources, steps, stop, stopJobsOnExit, thread`

### Fields inherited from interface cascading.flow.Flow

`CASCADING_FLOW_ID`

### Fields inherited from interface cascading.util.ProcessLogger

`NULL`

### Constructor Summary

Constructors
Constructor and Description
`MapReduceFlow(org.apache.hadoop.mapred.JobConf jobConf)` Constructor MapReduceFlow creates a new MapReduceFlow instance.
`MapReduceFlow(org.apache.hadoop.mapred.JobConf jobConf, boolean deleteSinkOnInit)` Constructor MapReduceFlow creates a new MapReduceFlow instance.
`MapReduceFlow(Properties properties, String name, org.apache.hadoop.mapred.JobConf jobConf, boolean deleteSinkOnInit)` Constructor MapReduceFlow creates a new MapReduceFlow instance.
`MapReduceFlow(Properties properties, String name, org.apache.hadoop.mapred.JobConf jobConf, Map<String,String> flowDescriptor, boolean deleteSinkOnInit)` Constructor MapReduceFlow creates a new MapReduceFlow instance.
`MapReduceFlow(Properties properties, String name, org.apache.hadoop.mapred.JobConf jobConf, Map<String,String> flowDescriptor, boolean deleteSinkOnInit, boolean stopJobsOnExit)` Constructor MapReduceFlow creates a new MapReduceFlow instance.
`MapReduceFlow(String name, org.apache.hadoop.mapred.JobConf jobConf)` Constructor MapReduceFlow creates a new MapReduceFlow instance.
`MapReduceFlow(String name, org.apache.hadoop.mapred.JobConf jobConf, boolean deleteSinkOnInit)` Constructor MapReduceFlow creates a new MapReduceFlow instance.

### Method Summary

Methods
`protected FlowStep<org.apache.hadoop.mapred.JobConf>` `createFlowStep(org.apache.hadoop.mapred.JobConf jobConf, Tap sink)`
`protected Map<String,Tap>` `createSinks(org.apache.hadoop.mapred.JobConf jobConf)`
`protected Map<String,Tap>` `createSources(org.apache.hadoop.mapred.JobConf jobConf)`
`protected Map<String,Tap>` `createTraps(org.apache.hadoop.mapred.JobConf jobConf)`
`protected Map<String,Tap>` `fileInputToTaps(org.apache.hadoop.mapred.JobConf jobConf)`
`protected Map<String,Tap>` `fileOutputToTaps(org.apache.hadoop.mapred.JobConf jobConf)`
`protected void` `initializeFrom(org.apache.hadoop.mapred.JobConf jobConf)`
`protected String` `makeNameFromPath(Map<String,Tap> taps, org.apache.hadoop.fs.Path path)`
`protected FlowStepGraph` `makeStepGraph(org.apache.hadoop.mapred.JobConf jobConf)`
`protected Tap` `toSinkTap(Map<String,Tap> taps, org.apache.hadoop.fs.Path path)`
`protected Tap` `toSourceTap(Map<String,Tap> taps, org.apache.hadoop.fs.Path path)`

### Methods inherited from class cascading.flow.hadoop.HadoopFlow

`copyToDistributedCache, getConfig, getConfigAsProperties, getConfigCopy, getFlowProcess, getMaxNumParallelSteps, getProperty, getTotalSliceCPUMilliSeconds, initConfig, initFromProperties, internalClean, internalShutdown, internalStart, isPreserveTemporaryFiles, newConfig, registerHadoopShutdownHook, setConfigProperty, stepsAreLocal`

### Methods inherited from class cascading.flow.BaseFlow

`addListener, addPlannerProperties, addSessionProperties, addStepListener, areSinksStale, areSourcesNewer, cleanup, complete, createConfig, createFlowCanonicalHash, createFlowStats, createFlowThread, createPrepareFlowStats, deleteCheckpointsIfNotUpdate, deleteCheckpointsIfReplace, deleteSinks, deleteSinksIfNotUpdate, deleteSinksIfReplace, deleteTrapsIfNotUpdate, deleteTrapsIfReplace, fireOnCompleted, fireOnStarting, fireOnStopping, fireOnThrowable, fireOnThrowable, getCascadeID, getCascadingServices, getCheckpointNames, getCheckpoints, getCheckpointsCollection, getClassPath, getClientState, getFieldsFor, getFlowCanonicalHash, getFlowDescriptor, getFlowSession, getFlowSkipStrategy, getFlowStats, getFlowSteps, getFlowStepStrategy, getHolder, getID, getName, getPlannerInfo, getPlatformInfo, getRunID, getSink, getSink, getSinkModified, getSinkNames, getSinks, getSinksCollection, getSource, getSourceNames, getSources, getSourcesCollection, getSpawnStrategy, getStats, getSubmitPriority, getTags, getTrapNames, getTraps, getTrapsCollection, handleExecutorShutdown, hasListeners, hasStepListeners, initialize, initializeChildStats, initializeNewJobsMap, initSteps, internalStopAllJobs, isDebugEnabled, isInfoEnabled, isSkipFlow, isStopJobsOnExit, logDebug, logError, logError, logInfo, logWarn, logWarn, logWarn, openSink, openSink, openSource, openSource, openTapForRead, openTapForWrite, openTrap, openTrap, prepare, presentSinkFields, presentSourceFields, registerShutdownHook, removeListener, removeStepListener, resourceExists, retrieveSinkFields, retrieveSourceFields, setCascade, setCheckpoints, setFlowSkipStrategy, setFlowStepGraph, setFlowStepStrategy, setName, setPlannerInfo, setSinks, setSources, setSpawnStrategy, setSubmitPriority, setTraps, start, stop, toString, updateSchemes, writeDOT, writeStepsDOT`

### Methods inherited from class java.lang.Object

`clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait`

### Field Detail

#### deleteSinkOnInit

    protected boolean deleteSinkOnInit

Field deleteSinkOnInit

### Constructor Detail

#### MapReduceFlow

    @ConstructorProperties(value="jobConf")
    public MapReduceFlow(org.apache.hadoop.mapred.JobConf jobConf)

Constructor MapReduceFlow creates a new MapReduceFlow instance.

Parameters:
  `jobConf` - of type JobConf

#### MapReduceFlow

    @ConstructorProperties(value={"jobConf","deleteSinkOnInit"})
    public MapReduceFlow(org.apache.hadoop.mapred.JobConf jobConf,
                                            boolean deleteSinkOnInit)

Constructor MapReduceFlow creates a new MapReduceFlow instance.

Parameters:
  `jobConf` - of type JobConf
  `deleteSinkOnInit` - of type boolean

#### MapReduceFlow

    @ConstructorProperties(value={"name","jobConf"})
    public MapReduceFlow(String name,
                                            org.apache.hadoop.mapred.JobConf jobConf)

Constructor MapReduceFlow creates a new MapReduceFlow instance.

Parameters:
  `name` - of type String
  `jobConf` - of type JobConf

#### MapReduceFlow

    @ConstructorProperties(value={"name","jobConf","deleteSinkOnInit"})
    public MapReduceFlow(String name,
                                            org.apache.hadoop.mapred.JobConf jobConf,
                                            boolean deleteSinkOnInit)

Constructor MapReduceFlow creates a new MapReduceFlow instance.

Parameters:
  `name` - of type String
  `jobConf` - of type JobConf
  `deleteSinkOnInit` - of type boolean

#### MapReduceFlow

    @ConstructorProperties(value={"properties","name","jobConf","deleteSinkOnInit"})
    public MapReduceFlow(Properties properties,
                                            String name,
                                            org.apache.hadoop.mapred.JobConf jobConf,
                                            boolean deleteSinkOnInit)

Constructor MapReduceFlow creates a new MapReduceFlow instance.

Parameters:
  `properties` - of type Properties
  `name` - of type String
  `jobConf` - of type JobConf
  `deleteSinkOnInit` - of type boolean

#### MapReduceFlow

    @ConstructorProperties(value={"properties","name","jobConf","flowDescriptor","deleteSinkOnInit"})
    public MapReduceFlow(Properties properties,
                                            String name,
                                            org.apache.hadoop.mapred.JobConf jobConf,
                                            Map<String,String> flowDescriptor,
                                            boolean deleteSinkOnInit)

Constructor MapReduceFlow creates a new MapReduceFlow instance.

Parameters:
  `properties` - of type Properties
  `name` - of type String
  `jobConf` - of type JobConf
  `flowDescriptor` - of type Map<String, String>
  `deleteSinkOnInit` - of type boolean

#### MapReduceFlow

    @ConstructorProperties(value={"properties","name","jobConf","flowDescriptor","deleteSinkOnInit","stopJobsOnExit"})
    public MapReduceFlow(Properties properties,
                                            String name,
                                            org.apache.hadoop.mapred.JobConf jobConf,
                                            Map<String,String> flowDescriptor,
                                            boolean deleteSinkOnInit,
                                            boolean stopJobsOnExit)

Constructor MapReduceFlow creates a new MapReduceFlow instance.

Parameters:
  `properties` - of type Properties
  `name` - of type String
  `jobConf` - of type JobConf
  `flowDescriptor` - of type Map<String, String>
  `deleteSinkOnInit` - of type boolean
  `stopJobsOnExit` - of type boolean

### Method Detail

#### initializeFrom

    protected void initializeFrom(org.apache.hadoop.mapred.JobConf jobConf)

#### makeStepGraph

    protected FlowStepGraph makeStepGraph(org.apache.hadoop.mapred.JobConf jobConf)

#### createFlowStep

    protected FlowStep<org.apache.hadoop.mapred.JobConf> createFlowStep(org.apache.hadoop.mapred.JobConf jobConf,
                                                            Tap sink)

#### createSources

    protected Map<String,Tap> createSources(org.apache.hadoop.mapred.JobConf jobConf)

#### fileInputToTaps

    protected Map<String,Tap> fileInputToTaps(org.apache.hadoop.mapred.JobConf jobConf)

#### toSourceTap

    protected Tap toSourceTap(Map<String,Tap> taps,
                  org.apache.hadoop.fs.Path path)

#### createSinks

    protected Map<String,Tap> createSinks(org.apache.hadoop.mapred.JobConf jobConf)

#### fileOutputToTaps

    protected Map<String,Tap> fileOutputToTaps(org.apache.hadoop.mapred.JobConf jobConf)

#### toSinkTap

    protected Tap toSinkTap(Map<String,Tap> taps,
                org.apache.hadoop.fs.Path path)

#### makeNameFromPath

    protected String makeNameFromPath(Map<String,Tap> taps,
                          org.apache.hadoop.fs.Path path)

#### createTraps

    protected Map<String,Tap> createTraps(org.apache.hadoop.mapred.JobConf jobConf)