cascading.flow.tez

## Class Hadoop2TezFlow

- java.lang.Object
- - cascading.flow.BaseFlow<org.apache.tez.dag.api.TezConfiguration>
  - - cascading.flow.tez.Hadoop2TezFlow

- All Implemented Interfaces:
    Flow<org.apache.tez.dag.api.TezConfiguration>, UnitOfWork<FlowStats>, ProcessLogger

      public class Hadoop2TezFlow
      extends BaseFlow<org.apache.tez.dag.api.TezConfiguration>

  Class HadoopFlow is the Apache Hadoop specific implementation of a `Flow`.

  HadoopFlow must be created through a `Hadoop2TezFlowConnector` instance.

  If classpath paths are provided on the `FlowDef`, the Hadoop distributed cache mechanism will be used to augment the remote classpath.

  Any path elements that are relative will be uploaded to HDFS, and the HDFS URI will be used on the JobConf. Note all paths are added as "files" to the JobConf, not archives, so they aren't needlessly uncompressed cluster side.
  See Also:
    `Hadoop2TezFlowConnector`

- - ### Nested Class Summary

    - ### Nested classes/interfaces inherited from class cascading.flow.BaseFlow

      `BaseFlow.FlowHolder`

  - ### Field Summary

    - ### Fields inherited from class cascading.flow.BaseFlow

      `completed, flowCanonicalHash, flowDescriptor, flowElementGraph, flowStats, flowStepGraph, platformInfo, sinks, sources, steps, stop, stopJobsOnExit, thread, throwable`

    - ### Fields inherited from interface cascading.flow.Flow

      `CASCADING_FLOW_ID`

    - ### Fields inherited from interface cascading.util.ProcessLogger

      `NULL`

  - ### Constructor Summary

    Constructors
    Modifier | Constructor and Description
    --- | ---
    `protected` | `Hadoop2TezFlow()`
    `` | `Hadoop2TezFlow(PlatformInfo platformInfo, java.util.Map<java.lang.Object,java.lang.Object> properties, org.apache.tez.dag.api.TezConfiguration flowConf, FlowDef flowDef)`

  - ### Method Summary

    Methods
    Modifier and Type | Method and Description
    --- | ---
    `org.apache.tez.dag.api.TezConfiguration` | `getConfig()`
    `java.util.Map<java.lang.Object,java.lang.Object>` | `getConfigAsProperties()`
    `org.apache.tez.dag.api.TezConfiguration` | `getConfigCopy()`
    `FlowProcess<org.apache.tez.dag.api.TezConfiguration>` | `getFlowProcess()`
    `java.lang.String` | `getFlowStagingPath()`
    `protected int` | `getMaxNumParallelSteps()`
    `java.lang.String` | `getProperty(java.lang.String key)` Method getProperty returns the value associated with the given key from the underlying properties system.
    `protected long` | `getTotalSliceCPUMilliSeconds()`
    `protected void` | `initConfig(java.util.Map<java.lang.Object,java.lang.Object> properties, org.apache.tez.dag.api.TezConfiguration parentConfig)`
    `protected void` | `initFromProperties(java.util.Map<java.lang.Object,java.lang.Object> properties)`
    `protected void` | `internalClean(boolean stop)`
    `protected void` | `internalShutdown()`
    `protected void` | `internalStart()`
    `boolean` | `isPreserveTemporaryFiles()` Method isPreserveTemporaryFiles returns false if temporary files will be cleaned when this Flow completes.
    `protected org.apache.tez.dag.api.TezConfiguration` | `newConfig(org.apache.tez.dag.api.TezConfiguration defaultConfig)`
    `protected void` | `setConfigProperty(org.apache.tez.dag.api.TezConfiguration config, java.lang.Object key, java.lang.Object value)`
    `boolean` | `stepsAreLocal()`

    - ### Methods inherited from class cascading.flow.BaseFlow

      `addListener, addPlannerProperties, addSessionProperties, addStepListener, areSinksStale, areSourcesNewer, cleanup, complete, createConfig, createFlowCanonicalHash, createFlowStats, createFlowThread, createPrepareFlowStats, deleteCheckpointsIfNotUpdate, deleteCheckpointsIfReplace, deleteSinks, deleteSinksIfNotUpdate, deleteSinksIfReplace, deleteTrapsIfNotUpdate, deleteTrapsIfReplace, fireOnCompleted, fireOnStarting, fireOnStopping, fireOnThrowable, fireOnThrowable, getCascadeID, getCascadingServices, getCheckpointNames, getCheckpoints, getCheckpointsCollection, getClassPath, getClientState, getEligibleJobsSize, getFieldsFor, getFlowCanonicalHash, getFlowDescriptor, getFlowElementGraph, getFlowSession, getFlowSkipStrategy, getFlowStats, getFlowStepGraph, getFlowSteps, getFlowStepStrategy, getHolder, getID, getJobMapCallables, getName, getPlannerInfo, getPlatformInfo, getRunID, getSink, getSink, getSinkModified, getSinkNames, getSinks, getSinksCollection, getSource, getSourceNames, getSources, getSourcesCollection, getSpawnStrategy, getStats, getSubmitPriority, getTags, getTrapNames, getTraps, getTrapsCollection, handleExecutorShutdown, hasListeners, hasStepListeners, initialize, initializeChildStats, initializeNewJobsMap, initSteps, internalStopAllJobs, isDebugEnabled, isInfoEnabled, isJobsMapInitialized, isSkipFlow, isStopJobsOnExit, logDebug, logError, logError, logInfo, logWarn, logWarn, logWarn, openSink, openSink, openSource, openSource, openTapForRead, openTapForWrite, openTrap, openTrap, prepare, presentSinkFields, presentSourceFields, registerShutdownHook, removeListener, removeListeners, removeStepListener, resourceExists, retrieveSinkFields, retrieveSourceFields, setCascade, setCheckpoints, setFlowElementGraph, setFlowSkipStrategy, setFlowStepGraph, setFlowStepStrategy, setName, setPlannerInfo, setSinks, setSources, setSpawnStrategy, setSubmitPriority, setTraps, spawnSteps, start, stop, toString, updateJobsMap, updateSchemes, writeDOT, writeStepsDOT`

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait`

- - ### Constructor Detail

    - #### Hadoop2TezFlow

          protected Hadoop2TezFlow()

    - #### Hadoop2TezFlow

          public Hadoop2TezFlow(PlatformInfo platformInfo,
                        java.util.Map<java.lang.Object,java.lang.Object> properties,
                        org.apache.tez.dag.api.TezConfiguration flowConf,
                        FlowDef flowDef)

  - ### Method Detail

    - #### initFromProperties

          protected void initFromProperties(java.util.Map<java.lang.Object,java.lang.Object> properties)

      Overrides:
        `initFromProperties` in class `BaseFlow<org.apache.tez.dag.api.TezConfiguration>`

    - #### initConfig

          protected void initConfig(java.util.Map<java.lang.Object,java.lang.Object> properties,
                        org.apache.tez.dag.api.TezConfiguration parentConfig)

      Specified by:
        `initConfig` in class `BaseFlow<org.apache.tez.dag.api.TezConfiguration>`

    - #### getFlowStagingPath

          public java.lang.String getFlowStagingPath()

    - #### setConfigProperty

          protected void setConfigProperty(org.apache.tez.dag.api.TezConfiguration config,
                               java.lang.Object key,
                               java.lang.Object value)

      Specified by:
        `setConfigProperty` in class `BaseFlow<org.apache.tez.dag.api.TezConfiguration>`

    - #### newConfig

          protected org.apache.tez.dag.api.TezConfiguration newConfig(org.apache.tez.dag.api.TezConfiguration defaultConfig)

      Specified by:
        `newConfig` in class `BaseFlow<org.apache.tez.dag.api.TezConfiguration>`

    - #### getConfig

          @ProcessConfiguration
          public org.apache.tez.dag.api.TezConfiguration getConfig()

    - #### getConfigCopy

          public org.apache.tez.dag.api.TezConfiguration getConfigCopy()

    - #### getConfigAsProperties

          public java.util.Map<java.lang.Object,java.lang.Object> getConfigAsProperties()

    - #### getProperty

          public java.lang.String getProperty(java.lang.String key)

      Method getProperty returns the value associated with the given key from the underlying properties system.
      Parameters:
        `key` - of type String
      Returns:
        String

    - #### getFlowProcess

          public FlowProcess<org.apache.tez.dag.api.TezConfiguration> getFlowProcess()

    - #### isPreserveTemporaryFiles

          public boolean isPreserveTemporaryFiles()

      Method isPreserveTemporaryFiles returns false if temporary files will be cleaned when this Flow completes.
      Returns:
        the preserveTemporaryFiles (type boolean) of this Flow object.

    - #### internalStart

          protected void internalStart()

      Specified by:
        `internalStart` in class `BaseFlow<org.apache.tez.dag.api.TezConfiguration>`

    - #### stepsAreLocal

          public boolean stepsAreLocal()

    - #### internalClean

          protected void internalClean(boolean stop)

      Specified by:
        `internalClean` in class `BaseFlow<org.apache.tez.dag.api.TezConfiguration>`

    - #### internalShutdown

          protected void internalShutdown()

      Specified by:
        `internalShutdown` in class `BaseFlow<org.apache.tez.dag.api.TezConfiguration>`

    - #### getMaxNumParallelSteps

          protected int getMaxNumParallelSteps()

      Specified by:
        `getMaxNumParallelSteps` in class `BaseFlow<org.apache.tez.dag.api.TezConfiguration>`

    - #### getTotalSliceCPUMilliSeconds

          protected long getTotalSliceCPUMilliSeconds()

      Overrides:
        `getTotalSliceCPUMilliSeconds` in class `BaseFlow<org.apache.tez.dag.api.TezConfiguration>`

Copyright © 2007-2015 Xplenty, Inc. All Rights Reserved.