cascading.flow.hadoop

## Class MultiMapReduceFlow

- java.lang.Object
- - cascading.flow.BaseFlow<org.apache.hadoop.mapred.JobConf>
  - - cascading.flow.hadoop.HadoopFlow
    - - cascading.flow.hadoop.BaseMapReduceFlow
      - - cascading.flow.hadoop.MultiMapReduceFlow

- All Implemented Interfaces:
    Flow<org.apache.hadoop.mapred.JobConf>, UnitOfWork<FlowStats>, ProcessLogger

      public class MultiMapReduceFlow
      extends BaseMapReduceFlow

  Class MultiMapReduceFlow is a `HadoopFlow` subclass that supports custom MapReduce jobs pre-configured via one or more `JobConf` objects.

  Use this class to group multiple JobConf instances together as a single Flow. MultiMapReduceFlow will automatically topologically order the JobConf instances and schedule them on the cluster once `BaseFlow.start()` or `complete()` are called.

  If you have a single JobConf instance, see `MapReduceFlow` as a alternative to this class.

  This class will not delete any sinks before execution, it is up to the developer to make sure any intermediate and sink paths be removed/deleted before calling `BaseFlow.start()` or `complete()`, otherwise Hadoop will throw an exception.

  JobConf instances can be incrementally added at any point before the `complete()` method is called. But they must logically (topologically) come after any previously provided JobConf instances. In practice the Flow will fail if the input source path is missing because a prior JobConf was not provided before the Flow was started.

  The ordering is done by comparing the input and output paths of the given JobConf instances. By default, this class only works with JobConf instances that read and write from the Hadoop FileSystem (HDFS) (any path that would work with the `Hfs` Tap.

  If the configured JobConf instance uses some other identifier instead of Hadoop FS paths, you should override the `BaseMapReduceFlow.createSources(org.apache.hadoop.mapred.JobConf)`, `BaseMapReduceFlow.createSinks(org.apache.hadoop.mapred.JobConf)`, and `BaseMapReduceFlow.createTraps(org.apache.hadoop.mapred.JobConf)` methods to properly resolve the configured paths into usable `Tap` instances. By default createTraps returns an empty collection and should probably be left alone.

  MultiMapReduceFlow supports both org.apache.hadoop.mapred.* and org.apache.hadoop.mapreduce.* API Jobs.

- - ### Nested Class Summary

    - ### Nested classes/interfaces inherited from class cascading.flow.BaseFlow

      `BaseFlow.FlowHolder`

  - ### Field Summary

    - ### Fields inherited from class cascading.flow.hadoop.BaseMapReduceFlow

      `deleteSinkOnInit`

    - ### Fields inherited from class cascading.flow.BaseFlow

      `completed, flowCanonicalHash, flowDescriptor, flowElementGraph, flowStats, flowStepGraph, platformInfo, sinks, sources, steps, stop, stopJobsOnExit, thread, throwable`

    - ### Fields inherited from interface cascading.flow.Flow

      `CASCADING_FLOW_ID`

    - ### Fields inherited from interface cascading.util.ProcessLogger

      `NULL`

  - ### Constructor Summary

    Constructors
    Modifier | Constructor and Description
    --- | ---
    `` | `MultiMapReduceFlow(java.util.Map<java.lang.Object,java.lang.Object> properties, java.lang.String name, org.apache.hadoop.mapred.JobConf jobConf, org.apache.hadoop.mapred.JobConf... jobConfs)` Constructor MultiMapReduceFlow creates a new MultiMapReduceFlow instance.
    `` | `MultiMapReduceFlow(java.util.Map<java.lang.Object,java.lang.Object> properties, java.lang.String name, java.util.Map<java.lang.String,java.lang.String> flowDescriptor, boolean stopJobsOnExit, org.apache.hadoop.mapred.JobConf jobConf, org.apache.hadoop.mapred.JobConf... jobConfs)` Constructor MultiMapReduceFlow creates a new MultiMapReduceFlow instance.
    `` | `MultiMapReduceFlow(java.util.Map<java.lang.Object,java.lang.Object> properties, java.lang.String name, java.util.Map<java.lang.String,java.lang.String> flowDescriptor, org.apache.hadoop.mapred.JobConf jobConf, org.apache.hadoop.mapred.JobConf... jobConfs)` Constructor MultiMapReduceFlow creates a new MultiMapReduceFlow instance.
    `protected` | `MultiMapReduceFlow(PlatformInfo platformInfo, java.util.Map<java.lang.Object,java.lang.Object> properties, java.lang.String name)` Constructor MultiMapReduceFlow creates a new MultiMapReduceFlow instance.
    `protected` | `MultiMapReduceFlow(PlatformInfo platformInfo, java.util.Map<java.lang.Object,java.lang.Object> properties, java.lang.String name, java.util.Map<java.lang.String,java.lang.String> flowDescriptor)` Constructor MultiMapReduceFlow creates a new MultiMapReduceFlow instance.
    `` | `MultiMapReduceFlow(java.lang.String name, org.apache.hadoop.mapred.JobConf jobConf, org.apache.hadoop.mapred.JobConf... jobConfs)` Constructor MultiMapReduceFlow creates a new MultiMapReduceFlow instance.

  - ### Method Summary

    Methods
    Modifier and Type | Method and Description
    --- | ---
    `protected void` | `addFlowStep(MapReduceFlowStep flowStep)`
    `void` | `attachFlowStep(org.apache.hadoop.mapred.JobConf jobConf)`
    `protected boolean` | `blockingContinuePollingSteps()`
    `void` | `complete()` Method complete starts the current Flow instance if it has not be previously started, then block until completion.
    `protected MapReduceFlowStep` | `createMapReduceFlowStep(org.apache.hadoop.mapred.JobConf jobConf)`
    `protected Tap` | `createTap(org.apache.hadoop.mapred.JobConf jobConf, org.apache.hadoop.fs.Path path, SinkMode sinkMode)`
    `protected FlowStepGraph` | `getOrCreateFlowStepGraph()`
    `protected void` | `initializeFrom(java.util.List<org.apache.hadoop.mapred.JobConf> jobConfs)`
    `void` | `notifyComplete()`
    `protected boolean` | `spawnSteps()` Reentrant method to launch all step jobs, returns false if any of the current steps failed.
    `protected FlowStepGraph` | `updateFlowStepGraph(FlowStepGraph flowStepGraph, java.util.Collection<MapReduceFlowStep> flowSteps)`
    `protected java.util.Collection<MapReduceFlowStep>` | `updateWithFlowSteps(java.util.Collection<MapReduceFlowStep> flowSteps)`

    - ### Methods inherited from class cascading.flow.hadoop.BaseMapReduceFlow

      `createFlowStep, createSinks, createSources, createTraps, fileInputToTaps, fileOutputToTaps, makeNameFromPath, makeStepGraph, toSinkTap, toSourceTap`

    - ### Methods inherited from class cascading.flow.hadoop.HadoopFlow

      `copyToDistributedCache, getConfig, getConfigAsProperties, getConfigCopy, getFlowProcess, getMaxNumParallelSteps, getProperty, getTotalSliceCPUMilliSeconds, initConfig, initFromProperties, internalClean, internalShutdown, internalStart, isPreserveTemporaryFiles, newConfig, registerHadoopShutdownHook, setConfigProperty, stepsAreLocal`

    - ### Methods inherited from class cascading.flow.BaseFlow

      `addListener, addPlannerProperties, addSessionProperties, addStepListener, areSinksStale, areSourcesNewer, cleanup, createConfig, createFlowCanonicalHash, createFlowStats, createFlowThread, createPrepareFlowStats, deleteCheckpointsIfNotUpdate, deleteCheckpointsIfReplace, deleteSinks, deleteSinksIfNotUpdate, deleteSinksIfReplace, deleteTrapsIfNotUpdate, deleteTrapsIfReplace, fireOnCompleted, fireOnStarting, fireOnStopping, fireOnThrowable, fireOnThrowable, getCascadeID, getCascadingServices, getCheckpointNames, getCheckpoints, getCheckpointsCollection, getClassPath, getClientState, getEligibleJobsSize, getFieldsFor, getFlowCanonicalHash, getFlowDescriptor, getFlowElementGraph, getFlowSession, getFlowSkipStrategy, getFlowStats, getFlowStepGraph, getFlowSteps, getFlowStepStrategy, getHolder, getID, getJobMapCallables, getName, getPlannerInfo, getPlatformInfo, getRunID, getSink, getSink, getSinkModified, getSinkNames, getSinks, getSinksCollection, getSource, getSourceNames, getSources, getSourcesCollection, getSpawnStrategy, getStats, getSubmitPriority, getTags, getTrapNames, getTraps, getTrapsCollection, handleExecutorShutdown, hasListeners, hasStepListeners, initialize, initializeChildStats, initializeNewJobsMap, initSteps, internalStopAllJobs, isDebugEnabled, isInfoEnabled, isJobsMapInitialized, isSkipFlow, isStopJobsOnExit, logDebug, logError, logError, logInfo, logWarn, logWarn, logWarn, openSink, openSink, openSource, openSource, openTapForRead, openTapForWrite, openTrap, openTrap, prepare, presentSinkFields, presentSourceFields, registerShutdownHook, removeListener, removeListeners, removeStepListener, resourceExists, retrieveSinkFields, retrieveSourceFields, setCascade, setCheckpoints, setFlowElementGraph, setFlowSkipStrategy, setFlowStepGraph, setFlowStepStrategy, setName, setPlannerInfo, setSinks, setSources, setSpawnStrategy, setSubmitPriority, setTraps, start, stop, toString, updateJobsMap, updateSchemes, writeDOT, writeStepsDOT`

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait`

- - ### Constructor Detail

    - #### MultiMapReduceFlow

          public MultiMapReduceFlow(java.lang.String name,
                            org.apache.hadoop.mapred.JobConf jobConf,
                            org.apache.hadoop.mapred.JobConf... jobConfs)

      Constructor MultiMapReduceFlow creates a new MultiMapReduceFlow instance.
      Parameters:
        `name` - of String
        `jobConf` - of JobConf
        `jobConfs` - of JobConf...

    - #### MultiMapReduceFlow

          public MultiMapReduceFlow(java.util.Map<java.lang.Object,java.lang.Object> properties,
                            java.lang.String name,
                            org.apache.hadoop.mapred.JobConf jobConf,
                            org.apache.hadoop.mapred.JobConf... jobConfs)

      Constructor MultiMapReduceFlow creates a new MultiMapReduceFlow instance.
      Parameters:
        `properties` - of Map
        `name` - of String
        `jobConf` - of JobConf
        `jobConfs` - of JobConf...

    - #### MultiMapReduceFlow

          public MultiMapReduceFlow(java.util.Map<java.lang.Object,java.lang.Object> properties,
                            java.lang.String name,
                            java.util.Map<java.lang.String,java.lang.String> flowDescriptor,
                            org.apache.hadoop.mapred.JobConf jobConf,
                            org.apache.hadoop.mapred.JobConf... jobConfs)

      Constructor MultiMapReduceFlow creates a new MultiMapReduceFlow instance.
      Parameters:
        `properties` - of Map
        `name` - of String
        `flowDescriptor` - of Map
        `jobConf` - of JobConf
        `jobConfs` - of JobConf...

    - #### MultiMapReduceFlow

          public MultiMapReduceFlow(java.util.Map<java.lang.Object,java.lang.Object> properties,
                            java.lang.String name,
                            java.util.Map<java.lang.String,java.lang.String> flowDescriptor,
                            boolean stopJobsOnExit,
                            org.apache.hadoop.mapred.JobConf jobConf,
                            org.apache.hadoop.mapred.JobConf... jobConfs)

      Constructor MultiMapReduceFlow creates a new MultiMapReduceFlow instance.
      Parameters:
        `properties` - of Map
        `name` - of String
        `flowDescriptor` - of Map
        `stopJobsOnExit` - of boolean
        `jobConf` - of JobConf
        `jobConfs` - of JobConf...

    - #### MultiMapReduceFlow

          protected MultiMapReduceFlow(PlatformInfo platformInfo,
                            java.util.Map<java.lang.Object,java.lang.Object> properties,
                            java.lang.String name)

      Constructor MultiMapReduceFlow creates a new MultiMapReduceFlow instance.
      Parameters:
        `platformInfo` - of PlatformInfo
        `properties` - of Map
        `name` - of String

    - #### MultiMapReduceFlow

          protected MultiMapReduceFlow(PlatformInfo platformInfo,
                            java.util.Map<java.lang.Object,java.lang.Object> properties,
                            java.lang.String name,
                            java.util.Map<java.lang.String,java.lang.String> flowDescriptor)

      Constructor MultiMapReduceFlow creates a new MultiMapReduceFlow instance.
      Parameters:
        `platformInfo` - of PlatformInfo
        `properties` - of Map
        `name` - of String
        `flowDescriptor` - of Map

  - ### Method Detail

    - #### initializeFrom

          protected void initializeFrom(java.util.List<org.apache.hadoop.mapred.JobConf> jobConfs)

    - #### createMapReduceFlowStep

          protected MapReduceFlowStep createMapReduceFlowStep(org.apache.hadoop.mapred.JobConf jobConf)

    - #### notifyComplete

          public void notifyComplete()

    - #### complete

          public void complete()

      Description copied from interface: `Flow`
      Method complete starts the current Flow instance if it has not be previously started, then block until completion.
      Specified by:
        `complete` in interface `Flow<org.apache.hadoop.mapred.JobConf>`
      Specified by:
        `complete` in interface `UnitOfWork<FlowStats>`
      Overrides:
        `complete` in class `BaseFlow<org.apache.hadoop.mapred.JobConf>`

    - #### spawnSteps

          protected boolean spawnSteps()
                                throws java.lang.InterruptedException,
                                       java.util.concurrent.ExecutionException

      Description copied from class: `BaseFlow`
      Reentrant method to launch all step jobs, returns false if any of the current steps failed.
      Overrides:
        `spawnSteps` in class `BaseFlow<org.apache.hadoop.mapred.JobConf>`
      Returns:
      Throws:
        `java.lang.InterruptedException`
        `java.util.concurrent.ExecutionException`

    - #### blockingContinuePollingSteps

          protected boolean blockingContinuePollingSteps()

    - #### createTap

          protected Tap createTap(org.apache.hadoop.mapred.JobConf jobConf,
                      org.apache.hadoop.fs.Path path,
                      SinkMode sinkMode)

      Overrides:
        `createTap` in class `BaseMapReduceFlow`

    - #### attachFlowStep

          public void attachFlowStep(org.apache.hadoop.mapred.JobConf jobConf)

    - #### addFlowStep

          protected void addFlowStep(MapReduceFlowStep flowStep)

    - #### getOrCreateFlowStepGraph

          protected FlowStepGraph getOrCreateFlowStepGraph()

    - #### updateWithFlowSteps

          protected java.util.Collection<MapReduceFlowStep> updateWithFlowSteps(java.util.Collection<MapReduceFlowStep> flowSteps)

    - #### updateFlowStepGraph

          protected FlowStepGraph updateFlowStepGraph(FlowStepGraph flowStepGraph,
                                          java.util.Collection<MapReduceFlowStep> flowSteps)

Copyright © 2007-2015 Xplenty, Inc. All Rights Reserved.