cascading.flow.hadoop

## Class HadoopFlow

- java.lang.Object
- - cascading.flow.BaseFlow<JobConf>
  - - cascading.flow.hadoop.HadoopFlow

All Implemented Interfaces:
  Flow<JobConf>, UnitOfWork<FlowStats>, ProcessLogger

Direct Known Subclasses:
  BaseMapReduceFlow

    public class HadoopFlow
    extends BaseFlow<JobConf>

Class HadoopFlow is the Apache Hadoop specific implementation of a `Flow`.

HadoopFlow must be created through a `FlowConnector` sub-class instance.

If classpath paths are provided on the `FlowDef`, the Hadoop distributed cache mechanism will be used to augment the remote classpath.

Any path elements that are relative will be uploaded to HDFS, and the HDFS URI will be used on the JobConf. Note all paths are added as "files" to the JobConf, not archives, so they aren't needlessly uncompressed cluster side.

See Also:
  `FlowConnector`

### Nested Class Summary

### Nested classes/interfaces inherited from class cascading.flow.BaseFlow

`BaseFlow.FlowHolder`

### Field Summary

### Fields inherited from class cascading.flow.BaseFlow

`completed, flowCanonicalHash, flowDescriptor, flowElementGraph, flowStats, flowStepGraph, platformInfo, sinks, sources, steps, stop, stopJobsOnExit, thread, throwable`

### Fields inherited from interface cascading.flow.Flow

`CASCADING_FLOW_ID`

### Fields inherited from interface cascading.util.ProcessLogger

`NULL`

### Constructor Summary

Constructors
`protected` `HadoopFlow()`
`` `HadoopFlow(PlatformInfo platformInfo, java.util.Map<java.lang.Object,java.lang.Object> properties, JobConf jobConf, FlowDef flowDef)`
`protected` `HadoopFlow(PlatformInfo platformInfo, java.util.Map<java.lang.Object,java.lang.Object> properties, JobConf jobConf, java.lang.String name, java.util.Map<java.lang.String,java.lang.String> flowDescriptor)`

### Method Summary

Methods
`protected void` `copyToDistributedCache()`
`JobConf` `getConfig()`
`java.util.Map<java.lang.Object,java.lang.Object>` `getConfigAsProperties()`
`JobConf` `getConfigCopy()`
`FlowProcess<JobConf>` `getFlowProcess()`
`protected int` `getMaxNumParallelSteps()`
`java.lang.String` `getProperty(java.lang.String key)` Method getProperty returns the value associated with the given key from the underlying properties system.
`protected long` `getTotalSliceCPUMilliSeconds()`
`protected void` `initConfig(java.util.Map<java.lang.Object,java.lang.Object> properties, JobConf parentConfig)`
`protected void` `initFromProperties(java.util.Map<java.lang.Object,java.lang.Object> properties)`
`protected void` `internalClean(boolean stop)`
`protected void` `internalShutdown()`
`protected void` `internalStart()`
`boolean` `isPreserveTemporaryFiles()` Method isPreserveTemporaryFiles returns false if temporary files will be cleaned when this Flow completes.
`protected JobConf` `newConfig(JobConf defaultConfig)`
`protected void` `registerHadoopShutdownHook()`
`protected void` `setConfigProperty(JobConf config, java.lang.Object key, java.lang.Object value)`
`boolean` `stepsAreLocal()`

### Methods inherited from class cascading.flow.BaseFlow

`addListener, addPlannerProperties, addSessionProperties, addStepListener, areSinksStale, areSourcesNewer, cleanup, complete, createConfig, createFlowCanonicalHash, createFlowStats, createFlowThread, createPrepareFlowStats, deleteCheckpointsIfNotUpdate, deleteCheckpointsIfReplace, deleteSinks, deleteSinksIfNotUpdate, deleteSinksIfReplace, deleteTrapsIfNotUpdate, deleteTrapsIfReplace, fireOnCompleted, fireOnStarting, fireOnStopping, fireOnThrowable, fireOnThrowable, getCascadeID, getCascadingServices, getCheckpointNames, getCheckpoints, getCheckpointsCollection, getClassPath, getClientState, getEligibleJobsSize, getFieldsFor, getFlowCanonicalHash, getFlowDescriptor, getFlowElementGraph, getFlowSession, getFlowSkipStrategy, getFlowStats, getFlowStepGraph, getFlowSteps, getFlowStepStrategy, getHolder, getID, getJobMapCallables, getName, getPlannerInfo, getPlatformInfo, getRunID, getSink, getSink, getSinkModified, getSinkNames, getSinks, getSinksCollection, getSource, getSourceNames, getSources, getSourcesCollection, getSpawnStrategy, getStats, getSubmitPriority, getTags, getTrapNames, getTraps, getTrapsCollection, handleExecutorShutdown, hasListeners, hasStepListeners, initialize, initializeChildStats, initializeNewJobsMap, initSteps, internalStopAllJobs, isDebugEnabled, isInfoEnabled, isJobsMapInitialized, isSkipFlow, isStopJobsOnExit, logDebug, logError, logError, logInfo, logWarn, logWarn, logWarn, openSink, openSink, openSource, openSource, openTapForRead, openTapForWrite, openTrap, openTrap, prepare, presentSinkFields, presentSourceFields, registerShutdownHook, removeListener, removeListeners, removeStepListener, resourceExists, retrieveSinkFields, retrieveSourceFields, setCascade, setCheckpoints, setFlowElementGraph, setFlowSkipStrategy, setFlowStepGraph, setFlowStepStrategy, setName, setPlannerInfo, setSinks, setSources, setSpawnStrategy, setSubmitPriority, setTraps, spawnSteps, start, stop, toString, updateJobsMap, updateSchemes, writeDOT, writeStepsDOT`

### Methods inherited from class java.lang.Object

`clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait`

### Constructor Detail

#### HadoopFlow

    protected HadoopFlow()

#### HadoopFlow

    protected HadoopFlow(PlatformInfo platformInfo,
              java.util.Map<java.lang.Object,java.lang.Object> properties,
              JobConf jobConf,
              java.lang.String name,
              java.util.Map<java.lang.String,java.lang.String> flowDescriptor)

#### HadoopFlow

    public HadoopFlow(PlatformInfo platformInfo,
              java.util.Map<java.lang.Object,java.lang.Object> properties,
              JobConf jobConf,
              FlowDef flowDef)

### Method Detail

#### initFromProperties

    protected void initFromProperties(java.util.Map<java.lang.Object,java.lang.Object> properties)

Overrides:
  `initFromProperties` in class `BaseFlow<JobConf>`

#### initConfig

    protected void initConfig(java.util.Map<java.lang.Object,java.lang.Object> properties,
                  JobConf parentConfig)

Specified by:
  `initConfig` in class `BaseFlow<JobConf>`

#### setConfigProperty

    protected void setConfigProperty(JobConf config,
                         java.lang.Object key,
                         java.lang.Object value)

Specified by:
  `setConfigProperty` in class `BaseFlow<JobConf>`

#### newConfig

    protected JobConf newConfig(JobConf defaultConfig)

Specified by:
  `newConfig` in class `BaseFlow<JobConf>`

#### getConfig

    @ProcessConfiguration
    public JobConf getConfig()

#### getConfigCopy

    public JobConf getConfigCopy()

#### getConfigAsProperties

    public java.util.Map<java.lang.Object,java.lang.Object> getConfigAsProperties()

#### getProperty

    public java.lang.String getProperty(java.lang.String key)

Method getProperty returns the value associated with the given key from the underlying properties system.

Parameters:
  `key` - of type String
Returns:
  String

#### getFlowProcess

    public FlowProcess<JobConf> getFlowProcess()

#### isPreserveTemporaryFiles

    public boolean isPreserveTemporaryFiles()

Method isPreserveTemporaryFiles returns false if temporary files will be cleaned when this Flow completes.

Returns:
  the preserveTemporaryFiles (type boolean) of this Flow object.

#### internalStart

    protected void internalStart()

Specified by:
  `internalStart` in class `BaseFlow<JobConf>`

#### registerHadoopShutdownHook

    protected void registerHadoopShutdownHook()

#### copyToDistributedCache

    protected void copyToDistributedCache()

#### stepsAreLocal

    public boolean stepsAreLocal()

#### internalClean

    protected void internalClean(boolean stop)

Specified by:
  `internalClean` in class `BaseFlow<JobConf>`

#### internalShutdown

    protected void internalShutdown()

Specified by:
  `internalShutdown` in class `BaseFlow<JobConf>`

#### getMaxNumParallelSteps

    protected int getMaxNumParallelSteps()

Specified by:
  `getMaxNumParallelSteps` in class `BaseFlow<JobConf>`

#### getTotalSliceCPUMilliSeconds

    protected long getTotalSliceCPUMilliSeconds()

Overrides:
  `getTotalSliceCPUMilliSeconds` in class `BaseFlow<JobConf>`