cascading.flow.hadoop

## Class MapReduceFlow

- java.lang.Object
- - cascading.flow.BaseFlow<org.apache.hadoop.mapred.JobConf>
  - - cascading.flow.hadoop.HadoopFlow
    - - cascading.flow.hadoop.BaseMapReduceFlow
      - - cascading.flow.hadoop.MapReduceFlow

- All Implemented Interfaces:
    Flow<org.apache.hadoop.mapred.JobConf>, UnitOfWork<FlowStats>, ProcessLogger

      public class MapReduceFlow
      extends BaseMapReduceFlow

  Class MapReduceFlow is a `HadoopFlow` subclass that supports custom MapReduce jobs pre-configured via the `JobConf` object.

  Use this class to allow custom MapReduce jobs to participate in the `Cascade` scheduler. If other Flow instances in the Cascade share resources with this Flow instance, all participants will be scheduled according to their dependencies (topologically).

  Set the parameter `deleteSinkOnInit` to `true` if the outputPath in the jobConf should be deleted before executing the MapReduce job.

  MapReduceFlow assumes the underlying input and output paths are compatible with the `Hfs` Tap.

  If the configured JobConf instance uses some other identifier instead of Hadoop FS paths, you should override the `BaseMapReduceFlow.createSources(org.apache.hadoop.mapred.JobConf)`, `BaseMapReduceFlow.createSinks(org.apache.hadoop.mapred.JobConf)`, and `BaseMapReduceFlow.createTraps(org.apache.hadoop.mapred.JobConf)` methods to properly resolve the configured paths into usable `Tap` instances. By default createTraps returns an empty collection and should probably be left alone.

  MapReduceFlow supports both org.apache.hadoop.mapred.* and org.apache.hadoop.mapreduce.* API Jobs.

- - ### Nested Class Summary

    - ### Nested classes/interfaces inherited from class cascading.flow.BaseFlow

      `BaseFlow.FlowHolder`

  - ### Field Summary

    - ### Fields inherited from class cascading.flow.hadoop.BaseMapReduceFlow

      `deleteSinkOnInit`

    - ### Fields inherited from class cascading.flow.BaseFlow

      `completed, flowCanonicalHash, flowDescriptor, flowElementGraph, flowStats, flowStepGraph, platformInfo, sinks, sources, steps, stop, stopJobsOnExit, thread, throwable`

    - ### Fields inherited from interface cascading.flow.Flow

      `CASCADING_FLOW_ID`

    - ### Fields inherited from interface cascading.util.ProcessLogger

      `NULL`

  - ### Constructor Summary

    Constructors
    Constructor and Description
    `MapReduceFlow(org.apache.hadoop.mapred.JobConf jobConf)` Constructor MapReduceFlow creates a new MapReduceFlow instance.
    `MapReduceFlow(org.apache.hadoop.mapred.JobConf jobConf, boolean deleteSinkOnInit)` Constructor MapReduceFlow creates a new MapReduceFlow instance.
    `MapReduceFlow(java.util.Properties properties, java.lang.String name, org.apache.hadoop.mapred.JobConf jobConf, boolean deleteSinkOnInit)` Constructor MapReduceFlow creates a new MapReduceFlow instance.
    `MapReduceFlow(java.util.Properties properties, java.lang.String name, org.apache.hadoop.mapred.JobConf jobConf, java.util.Map<java.lang.String,java.lang.String> flowDescriptor, boolean deleteSinkOnInit)` Constructor MapReduceFlow creates a new MapReduceFlow instance.
    `MapReduceFlow(java.util.Properties properties, java.lang.String name, org.apache.hadoop.mapred.JobConf jobConf, java.util.Map<java.lang.String,java.lang.String> flowDescriptor, boolean deleteSinkOnInit, boolean stopJobsOnExit)` Constructor MapReduceFlow creates a new MapReduceFlow instance.
    `MapReduceFlow(java.lang.String name, org.apache.hadoop.mapred.JobConf jobConf)` Constructor MapReduceFlow creates a new MapReduceFlow instance.
    `MapReduceFlow(java.lang.String name, org.apache.hadoop.mapred.JobConf jobConf, boolean deleteSinkOnInit)` Constructor MapReduceFlow creates a new MapReduceFlow instance.

  - ### Method Summary

    Methods
    Modifier and Type | Method and Description
    --- | ---
    `protected void` | `initializeFrom(org.apache.hadoop.mapred.JobConf jobConf)`

    - ### Methods inherited from class cascading.flow.hadoop.BaseMapReduceFlow

      `createFlowStep, createSinks, createSources, createTap, createTraps, fileInputToTaps, fileOutputToTaps, makeNameFromPath, makeStepGraph, toSinkTap, toSourceTap`

    - ### Methods inherited from class cascading.flow.hadoop.HadoopFlow

      `copyToDistributedCache, getConfig, getConfigAsProperties, getConfigCopy, getFlowProcess, getMaxNumParallelSteps, getProperty, getTotalSliceCPUMilliSeconds, initConfig, initFromProperties, internalClean, internalShutdown, internalStart, isPreserveTemporaryFiles, newConfig, registerHadoopShutdownHook, setConfigProperty, stepsAreLocal`

    - ### Methods inherited from class cascading.flow.BaseFlow

      `addListener, addPlannerProperties, addSessionProperties, addStepListener, areSinksStale, areSourcesNewer, cleanup, complete, createConfig, createFlowCanonicalHash, createFlowStats, createFlowThread, createPrepareFlowStats, deleteCheckpointsIfNotUpdate, deleteCheckpointsIfReplace, deleteSinks, deleteSinksIfNotUpdate, deleteSinksIfReplace, deleteTrapsIfNotUpdate, deleteTrapsIfReplace, fireOnCompleted, fireOnStarting, fireOnStopping, fireOnThrowable, fireOnThrowable, getCascadeID, getCascadingServices, getCheckpointNames, getCheckpoints, getCheckpointsCollection, getClassPath, getClientState, getEligibleJobsSize, getFieldsFor, getFlowCanonicalHash, getFlowDescriptor, getFlowElementGraph, getFlowSession, getFlowSkipStrategy, getFlowStats, getFlowStepGraph, getFlowSteps, getFlowStepStrategy, getHolder, getID, getJobMapCallables, getName, getPlannerInfo, getPlatformInfo, getRunID, getSink, getSink, getSinkModified, getSinkNames, getSinks, getSinksCollection, getSource, getSourceNames, getSources, getSourcesCollection, getSpawnStrategy, getStats, getSubmitPriority, getTags, getTrapNames, getTraps, getTrapsCollection, handleExecutorShutdown, hasListeners, hasStepListeners, initialize, initializeChildStats, initializeNewJobsMap, initSteps, internalStopAllJobs, isDebugEnabled, isInfoEnabled, isJobsMapInitialized, isSkipFlow, isStopJobsOnExit, logDebug, logError, logError, logInfo, logWarn, logWarn, logWarn, openSink, openSink, openSource, openSource, openTapForRead, openTapForWrite, openTrap, openTrap, prepare, presentSinkFields, presentSourceFields, registerShutdownHook, removeListener, removeListeners, removeStepListener, resourceExists, retrieveSinkFields, retrieveSourceFields, setCascade, setCheckpoints, setFlowElementGraph, setFlowSkipStrategy, setFlowStepGraph, setFlowStepStrategy, setName, setPlannerInfo, setSinks, setSources, setSpawnStrategy, setSubmitPriority, setTraps, spawnSteps, start, stop, toString, updateJobsMap, updateSchemes, writeDOT, writeStepsDOT`

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait`

- - ### Constructor Detail

    - #### MapReduceFlow

          @ConstructorProperties(value="jobConf")
          public MapReduceFlow(org.apache.hadoop.mapred.JobConf jobConf)

      Constructor MapReduceFlow creates a new MapReduceFlow instance.
      Parameters:
        `jobConf` - of type JobConf

    - #### MapReduceFlow

          @ConstructorProperties(value={"jobConf","deleteSinkOnInit"})
          public MapReduceFlow(org.apache.hadoop.mapred.JobConf jobConf,
                                             boolean deleteSinkOnInit)

      Constructor MapReduceFlow creates a new MapReduceFlow instance.
      Parameters:
        `jobConf` - of type JobConf
        `deleteSinkOnInit` - of type boolean

    - #### MapReduceFlow

          @ConstructorProperties(value={"name","jobConf"})
          public MapReduceFlow(java.lang.String name,
                                             org.apache.hadoop.mapred.JobConf jobConf)

      Constructor MapReduceFlow creates a new MapReduceFlow instance.
      Parameters:
        `name` - of type String
        `jobConf` - of type JobConf

    - #### MapReduceFlow

          @ConstructorProperties(value={"name","jobConf","deleteSinkOnInit"})
          public MapReduceFlow(java.lang.String name,
                                             org.apache.hadoop.mapred.JobConf jobConf,
                                             boolean deleteSinkOnInit)

      Constructor MapReduceFlow creates a new MapReduceFlow instance.
      Parameters:
        `name` - of type String
        `jobConf` - of type JobConf
        `deleteSinkOnInit` - of type boolean

    - #### MapReduceFlow

          @ConstructorProperties(value={"properties","name","jobConf","deleteSinkOnInit"})
          public MapReduceFlow(java.util.Properties properties,
                                             java.lang.String name,
                                             org.apache.hadoop.mapred.JobConf jobConf,
                                             boolean deleteSinkOnInit)

      Constructor MapReduceFlow creates a new MapReduceFlow instance.
      Parameters:
        `properties` - of type Properties
        `name` - of type String
        `jobConf` - of type JobConf
        `deleteSinkOnInit` - of type boolean

    - #### MapReduceFlow

          @ConstructorProperties(value={"properties","name","jobConf","flowDescriptor","deleteSinkOnInit"})
          public MapReduceFlow(java.util.Properties properties,
                                             java.lang.String name,
                                             org.apache.hadoop.mapred.JobConf jobConf,
                                             java.util.Map<java.lang.String,java.lang.String> flowDescriptor,
                                             boolean deleteSinkOnInit)

      Constructor MapReduceFlow creates a new MapReduceFlow instance.
      Parameters:
        `properties` - of type Properties
        `name` - of type String
        `jobConf` - of type JobConf
        `flowDescriptor` - of type Map
        `deleteSinkOnInit` - of type boolean

    - #### MapReduceFlow

          @ConstructorProperties(value={"properties","name","jobConf","flowDescriptor","deleteSinkOnInit","stopJobsOnExit"})
          public MapReduceFlow(java.util.Properties properties,
                                             java.lang.String name,
                                             org.apache.hadoop.mapred.JobConf jobConf,
                                             java.util.Map<java.lang.String,java.lang.String> flowDescriptor,
                                             boolean deleteSinkOnInit,
                                             boolean stopJobsOnExit)

      Constructor MapReduceFlow creates a new MapReduceFlow instance.
      Parameters:
        `properties` - of type Properties
        `name` - of type String
        `jobConf` - of type JobConf
        `flowDescriptor` - of type Map
        `deleteSinkOnInit` - of type boolean
        `stopJobsOnExit` - of type boolean

  - ### Method Detail

    - #### initializeFrom

          protected void initializeFrom(org.apache.hadoop.mapred.JobConf jobConf)

Copyright © 2007-2015 Xplenty, Inc. All Rights Reserved.