cascading.flow.hadoop

## Class HadoopFlow

- java.lang.Object
- - cascading.flow.BaseFlow<org.apache.hadoop.mapred.JobConf>
  - - cascading.flow.hadoop.HadoopFlow

- All Implemented Interfaces:
    Flow<org.apache.hadoop.mapred.JobConf>, UnitOfWork<FlowStats>, ProcessLogger
  Direct Known Subclasses:
    BaseMapReduceFlow

      public class HadoopFlow
      extends BaseFlow<org.apache.hadoop.mapred.JobConf>

  Class HadoopFlow is the Apache Hadoop specific implementation of a `Flow`.

  HadoopFlow must be created through a `FlowConnector` sub-class instance.

  If classpath paths are provided on the `FlowDef`, the Hadoop distributed cache mechanism will be used to augment the remote classpath.

  Any path elements that are relative will be uploaded to HDFS, and the HDFS URI will be used on the JobConf. Note all paths are added as "files" to the JobConf, not archives, so they aren't needlessly uncompressed cluster side.
  See Also:
    `FlowConnector`

- - ### Nested Class Summary

    - ### Nested classes/interfaces inherited from class cascading.flow.BaseFlow

      `BaseFlow.FlowHolder`

  - ### Field Summary

    - ### Fields inherited from class cascading.flow.BaseFlow

      `completed, flowCanonicalHash, flowDescriptor, flowElementGraph, flowStats, flowStepGraph, platformInfo, sinks, sources, steps, stop, stopJobsOnExit, thread, throwable`

    - ### Fields inherited from interface cascading.flow.Flow

      `CASCADING_FLOW_ID`

    - ### Fields inherited from interface cascading.util.ProcessLogger

      `NULL`

  - ### Constructor Summary

    Constructors
    Modifier | Constructor and Description
    --- | ---
    `protected` | `HadoopFlow()`
    `` | `HadoopFlow(PlatformInfo platformInfo, java.util.Map<java.lang.Object,java.lang.Object> properties, org.apache.hadoop.mapred.JobConf jobConf, FlowDef flowDef)`
    `protected` | `HadoopFlow(PlatformInfo platformInfo, java.util.Map<java.lang.Object,java.lang.Object> properties, org.apache.hadoop.mapred.JobConf jobConf, java.lang.String name, java.util.Map<java.lang.String,java.lang.String> flowDescriptor)`

  - ### Method Summary

    Methods
    Modifier and Type | Method and Description
    --- | ---
    `protected void` | `copyToDistributedCache()`
    `org.apache.hadoop.mapred.JobConf` | `getConfig()` Method getConfig returns the internal configuration object.
    `java.util.Map<java.lang.Object,java.lang.Object>` | `getConfigAsProperties()` Method getConfigAsProperties converts the internal configuration object into a `Map` of key value pairs.
    `org.apache.hadoop.mapred.JobConf` | `getConfigCopy()` Method getConfigCopy returns a copy of the internal configuration object.
    `FlowProcess<org.apache.hadoop.mapred.JobConf>` | `getFlowProcess()`
    `protected int` | `getMaxNumParallelSteps()`
    `java.lang.String` | `getProperty(java.lang.String key)` Method getProperty returns the value associated with the given key from the underlying properties system.
    `protected long` | `getTotalSliceCPUMilliSeconds()`
    `protected void` | `initConfig(java.util.Map<java.lang.Object,java.lang.Object> properties, org.apache.hadoop.mapred.JobConf parentConfig)` This method creates a new internal Config with the parentConfig as defaults using the properties to override the defaults.
    `protected void` | `initFromProperties(java.util.Map<java.lang.Object,java.lang.Object> properties)`
    `protected void` | `internalClean(boolean stop)`
    `protected void` | `internalShutdown()`
    `protected void` | `internalStart()`
    `boolean` | `isPreserveTemporaryFiles()` Method isPreserveTemporaryFiles returns false if temporary files will be cleaned when this Flow completes.
    `protected org.apache.hadoop.mapred.JobConf` | `newConfig(org.apache.hadoop.mapred.JobConf defaultConfig)`
    `protected void` | `registerHadoopShutdownHook()`
    `protected void` | `setConfigProperty(org.apache.hadoop.mapred.JobConf config, java.lang.Object key, java.lang.Object value)`
    `boolean` | `stepsAreLocal()` Method jobsAreLocal returns true if all jobs are executed in-process as a single map and reduce task.

    - ### Methods inherited from class cascading.flow.BaseFlow

      `addListener, addPlannerProperties, addSessionProperties, addStepListener, areSinksStale, areSourcesNewer, cleanup, complete, createConfig, createFlowCanonicalHash, createFlowStats, createFlowThread, createPrepareFlowStats, deleteCheckpointsIfNotUpdate, deleteCheckpointsIfReplace, deleteSinks, deleteSinksIfNotUpdate, deleteSinksIfReplace, deleteTrapsIfNotUpdate, deleteTrapsIfReplace, fireOnCompleted, fireOnStarting, fireOnStopping, fireOnThrowable, fireOnThrowable, getCascadeID, getCascadingServices, getCheckpointNames, getCheckpoints, getCheckpointsCollection, getClassPath, getClientState, getEligibleJobsSize, getFieldsFor, getFlowCanonicalHash, getFlowDescriptor, getFlowElementGraph, getFlowSession, getFlowSkipStrategy, getFlowStats, getFlowStepGraph, getFlowSteps, getFlowStepStrategy, getHolder, getID, getJobMapCallables, getName, getPlannerInfo, getPlatformInfo, getRunID, getSink, getSink, getSinkModified, getSinkNames, getSinks, getSinksCollection, getSource, getSourceNames, getSources, getSourcesCollection, getSpawnStrategy, getStats, getSubmitPriority, getTags, getTrapNames, getTraps, getTrapsCollection, handleExecutorShutdown, hasListeners, hasStepListeners, initialize, initializeChildStats, initializeNewJobsMap, initSteps, internalStopAllJobs, isDebugEnabled, isInfoEnabled, isJobsMapInitialized, isSkipFlow, isStopJobsOnExit, logDebug, logError, logError, logInfo, logWarn, logWarn, logWarn, openSink, openSink, openSource, openSource, openTapForRead, openTapForWrite, openTrap, openTrap, prepare, presentSinkFields, presentSourceFields, registerShutdownHook, removeListener, removeListeners, removeStepListener, resourceExists, retrieveSinkFields, retrieveSourceFields, setCascade, setCheckpoints, setFlowElementGraph, setFlowSkipStrategy, setFlowStepGraph, setFlowStepStrategy, setName, setPlannerInfo, setSinks, setSources, setSpawnStrategy, setSubmitPriority, setTraps, spawnSteps, start, stop, toString, updateJobsMap, updateSchemes, writeDOT, writeStepsDOT`

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait`

- - ### Constructor Detail

    - #### HadoopFlow

          protected HadoopFlow()

    - #### HadoopFlow

          protected HadoopFlow(PlatformInfo platformInfo,
                    java.util.Map<java.lang.Object,java.lang.Object> properties,
                    org.apache.hadoop.mapred.JobConf jobConf,
                    java.lang.String name,
                    java.util.Map<java.lang.String,java.lang.String> flowDescriptor)

    - #### HadoopFlow

          public HadoopFlow(PlatformInfo platformInfo,
                    java.util.Map<java.lang.Object,java.lang.Object> properties,
                    org.apache.hadoop.mapred.JobConf jobConf,
                    FlowDef flowDef)

  - ### Method Detail

    - #### initFromProperties

          protected void initFromProperties(java.util.Map<java.lang.Object,java.lang.Object> properties)

      Overrides:
        `initFromProperties` in class `BaseFlow<org.apache.hadoop.mapred.JobConf>`

    - #### initConfig

          protected void initConfig(java.util.Map<java.lang.Object,java.lang.Object> properties,
                        org.apache.hadoop.mapred.JobConf parentConfig)

      Description copied from class: `BaseFlow`
      This method creates a new internal Config with the parentConfig as defaults using the properties to override the defaults.
      Specified by:
        `initConfig` in class `BaseFlow<org.apache.hadoop.mapred.JobConf>`
      Parameters:
        `properties` - of type Map
        `parentConfig` - of type Config

    - #### setConfigProperty

          protected void setConfigProperty(org.apache.hadoop.mapred.JobConf config,
                               java.lang.Object key,
                               java.lang.Object value)

      Specified by:
        `setConfigProperty` in class `BaseFlow<org.apache.hadoop.mapred.JobConf>`

    - #### newConfig

          protected org.apache.hadoop.mapred.JobConf newConfig(org.apache.hadoop.mapred.JobConf defaultConfig)

      Specified by:
        `newConfig` in class `BaseFlow<org.apache.hadoop.mapred.JobConf>`

    - #### getConfig

          @ProcessConfiguration
          public org.apache.hadoop.mapred.JobConf getConfig()

      Description copied from interface: `Flow`
      Method getConfig returns the internal configuration object.

      Any changes to this object will not be reflected in child steps. See `FlowConnector` for setting default properties visible to children. Or see `FlowStepStrategy` for setting properties on individual steps before they are executed.
      Returns:
        the default configuration of this Flow

    - #### getConfigCopy

          public org.apache.hadoop.mapred.JobConf getConfigCopy()

      Description copied from interface: `Flow`
      Method getConfigCopy returns a copy of the internal configuration object. This object can be safely modified.
      Returns:
        a copy of the default configuration of this Flow

    - #### getConfigAsProperties

          public java.util.Map<java.lang.Object,java.lang.Object> getConfigAsProperties()

      Description copied from interface: `Flow`
      Method getConfigAsProperties converts the internal configuration object into a `Map` of key value pairs.
      Returns:
        a Map of key/value pairs

    - #### getProperty

          public java.lang.String getProperty(java.lang.String key)

      Method getProperty returns the value associated with the given key from the underlying properties system.
      Parameters:
        `key` - of type String
      Returns:
        String

    - #### getFlowProcess

          public FlowProcess<org.apache.hadoop.mapred.JobConf> getFlowProcess()

    - #### isPreserveTemporaryFiles

          public boolean isPreserveTemporaryFiles()

      Method isPreserveTemporaryFiles returns false if temporary files will be cleaned when this Flow completes.
      Returns:
        the preserveTemporaryFiles (type boolean) of this Flow object.

    - #### internalStart

          protected void internalStart()

      Specified by:
        `internalStart` in class `BaseFlow<org.apache.hadoop.mapred.JobConf>`

    - #### registerHadoopShutdownHook

          protected void registerHadoopShutdownHook()

    - #### copyToDistributedCache

          protected void copyToDistributedCache()

    - #### stepsAreLocal

          public boolean stepsAreLocal()

      Description copied from interface: `Flow`
      Method jobsAreLocal returns true if all jobs are executed in-process as a single map and reduce task.
      Returns:
        boolean

    - #### internalClean

          protected void internalClean(boolean stop)

      Specified by:
        `internalClean` in class `BaseFlow<org.apache.hadoop.mapred.JobConf>`

    - #### internalShutdown

          protected void internalShutdown()

      Specified by:
        `internalShutdown` in class `BaseFlow<org.apache.hadoop.mapred.JobConf>`

    - #### getMaxNumParallelSteps

          protected int getMaxNumParallelSteps()

      Specified by:
        `getMaxNumParallelSteps` in class `BaseFlow<org.apache.hadoop.mapred.JobConf>`

    - #### getTotalSliceCPUMilliSeconds

          protected long getTotalSliceCPUMilliSeconds()

      Overrides:
        `getTotalSliceCPUMilliSeconds` in class `BaseFlow<org.apache.hadoop.mapred.JobConf>`

Copyright © 2007-2015 Xplenty, Inc. All Rights Reserved.