cascading.flow.hadoop

## Class HadoopFlowProcess

- java.lang.Object
- - cascading.flow.FlowProcess<org.apache.hadoop.mapred.JobConf>
  - - cascading.flow.hadoop.HadoopFlowProcess

All Implemented Interfaces:
  MapRed

    public class HadoopFlowProcess
    extends FlowProcess<org.apache.hadoop.mapred.JobConf>
    implements MapRed

Class HadoopFlowProcess is an implementation of `FlowProcess` for Hadoop. Use this interface to get direct access to the Hadoop JobConf and Reporter interfaces.

Be warned that coupling to this implementation will cause custom `Operation`s to fail if they are executed on a system other than Hadoop.

See Also:
  `FlowSession`, `JobConf`, `Reporter`

### Nested Class Summary

### Nested classes/interfaces inherited from class cascading.flow.FlowProcess

`FlowProcess.NullFlowProcess`

### Field Summary

### Fields inherited from class cascading.flow.FlowProcess

`NULL`

### Constructor Summary

Constructors
Constructor and Description
`HadoopFlowProcess()`
`HadoopFlowProcess(org.apache.hadoop.conf.Configuration jobConf)`
`HadoopFlowProcess(FlowSession flowSession, org.apache.hadoop.mapred.JobConf jobConf)`
`HadoopFlowProcess(FlowSession flowSession, org.apache.hadoop.mapred.JobConf jobConf, boolean isMapper)` Constructor HadoopFlowProcess creates a new HadoopFlowProcess instance.
`HadoopFlowProcess(HadoopFlowProcess flowProcess, org.apache.hadoop.mapred.JobConf jobConf)`
`HadoopFlowProcess(org.apache.hadoop.mapred.JobConf jobConf)`

### Method Summary

Methods
`<C> C` `copyConfig(C config)`
`FlowProcess` `copyWith(org.apache.hadoop.mapred.JobConf jobConf)`
`<C> java.util.Map<java.lang.String,java.lang.String>` `diffConfigIntoMap(C defaultConfig, C updatedConfig)`
`org.apache.hadoop.mapred.JobConf` `getConfig()` Method getConfig returns the actual instance of the underlying configuration instance.
`org.apache.hadoop.mapred.JobConf` `getConfigCopy()`
`long` `getCounterValue(java.lang.Enum counter)` Method getCounterValue is used to retrieve a counter value.
`long` `getCounterValue(java.lang.String group, java.lang.String counter)` Method getCounterValue is used to retrieve a counter value.
`int` `getCurrentNumMappers()`
`int` `getCurrentNumReducers()`
`int` `getCurrentSliceNum()` Method getCurrentTaskNum returns the task number of this task.
`org.apache.hadoop.mapred.JobConf` `getJobConf()` Method getJobConf returns the jobConf of this HadoopFlowProcess object.
`int` `getNumProcessSlices()` Method getNumProcessSlices returns the number of parallel slices or tasks allocated for this process execution.
`org.apache.hadoop.mapred.OutputCollector` `getOutputCollector()`
`java.lang.Object` `getProperty(java.lang.String key)` Method getProperty should be used to return configuration parameters from the underlying system.
`java.util.Collection<java.lang.String>` `getPropertyKeys()` Method getPropertyKeys returns an immutable collection of all available property key values.
`org.apache.hadoop.mapred.Reporter` `getReporter()`
`void` `increment(java.lang.Enum counter, long amount)` Method increment is used to increment a custom counter.
`void` `increment(java.lang.String group, java.lang.String counter, long amount)` Method increment is used to increment a custom counter.
`boolean` `isCounterStatusInitialized()` Method isCounterStatusInitialized returns true if it is safe to increment a counter or set a status.
`boolean` `isMapper()` Method isMapper returns true if this part of the FlowProcess is a MapReduce mapper.
`void` `keepAlive()` Method keepAlive notifies the system that the current process is still alive.
`org.apache.hadoop.mapred.JobConf` `mergeMapIntoConfig(org.apache.hadoop.mapred.JobConf defaultConfig, java.util.Map<java.lang.String,java.lang.String> map)`
`java.lang.Object` `newInstance(java.lang.String className)` Method newInstance creates a new object instance from the given className argument delegating to any platform specific instantiation and configuration routines.
`TupleEntryCollector` `openSystemIntermediateForWrite()`
`TupleEntryIterator` `openTapForRead(Tap tap)` Method openTapForRead return a `TupleEntryIterator` for the given Tap instance.
`TupleEntryCollector` `openTapForWrite(Tap tap)` Method openTapForWrite returns a (@link TupleCollector} for the given Tap instance.
`TupleEntryCollector` `openTrapForWrite(Tap trap)` Method openTrapForWrite returns a (@link TupleCollector} for the given Tap instance.
`void` `setOutputCollector(org.apache.hadoop.mapred.OutputCollector outputCollector)`
`void` `setReporter(org.apache.hadoop.mapred.Reporter reporter)` Method setReporter sets the reporter of this HadoopFlowProcess object.
`void` `setStatus(java.lang.String status)` Method setStatus is used to set the status of the current operation.

### Methods inherited from class cascading.flow.FlowProcess

`closeTrapCollectors, getBooleanProperty, getBooleanProperty, getCurrentSession, getID, getIntegerProperty, getIntegerProperty, getStringProperty, getStringProperty, getTrapCollectorFor, getTrapCollectors, setCurrentSession`

### Methods inherited from class java.lang.Object

`clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

### Constructor Detail

#### HadoopFlowProcess

    public HadoopFlowProcess()

#### HadoopFlowProcess

    public HadoopFlowProcess(org.apache.hadoop.conf.Configuration jobConf)

#### HadoopFlowProcess

    public HadoopFlowProcess(org.apache.hadoop.mapred.JobConf jobConf)

#### HadoopFlowProcess

    public HadoopFlowProcess(FlowSession flowSession,
                     org.apache.hadoop.mapred.JobConf jobConf)

#### HadoopFlowProcess

    public HadoopFlowProcess(FlowSession flowSession,
                     org.apache.hadoop.mapred.JobConf jobConf,
                     boolean isMapper)

Constructor HadoopFlowProcess creates a new HadoopFlowProcess instance.

Parameters:
  `flowSession` - of type FlowSession
  `jobConf` - of type JobConf

#### HadoopFlowProcess

    public HadoopFlowProcess(HadoopFlowProcess flowProcess,
                     org.apache.hadoop.mapred.JobConf jobConf)

### Method Detail

#### copyWith

    public FlowProcess copyWith(org.apache.hadoop.mapred.JobConf jobConf)

Specified by:
  `copyWith` in class `FlowProcess<org.apache.hadoop.mapred.JobConf>`

#### getJobConf

    public org.apache.hadoop.mapred.JobConf getJobConf()

Method getJobConf returns the jobConf of this HadoopFlowProcess object.

Returns:
  the jobConf (type JobConf) of this HadoopFlowProcess object.

#### getConfig

    public org.apache.hadoop.mapred.JobConf getConfig()

Description copied from class: `FlowProcess`
Method getConfig returns the actual instance of the underlying configuration instance.

This instance should not be modified or cached, see `FlowProcess.getConfigCopy()` for a modifiable instance.

Specified by:
  `getConfig` in class `FlowProcess<org.apache.hadoop.mapred.JobConf>`
Returns:
  an instance of the configuration

#### getConfigCopy

    public org.apache.hadoop.mapred.JobConf getConfigCopy()

Specified by:
  `getConfigCopy` in class `FlowProcess<org.apache.hadoop.mapred.JobConf>`

#### isMapper

    public boolean isMapper()

Method isMapper returns true if this part of the FlowProcess is a MapReduce mapper. If false, it is a reducer.

Returns:
  boolean

#### getCurrentNumMappers

    public int getCurrentNumMappers()

#### getCurrentNumReducers

    public int getCurrentNumReducers()

#### getCurrentSliceNum

    public int getCurrentSliceNum()

Method getCurrentTaskNum returns the task number of this task. Task 0 is the first task.

Specified by:
  `getCurrentSliceNum` in class `FlowProcess<org.apache.hadoop.mapred.JobConf>`
Returns:
  int

#### getNumProcessSlices

    public int getNumProcessSlices()

Description copied from class: `FlowProcess`
Method getNumProcessSlices returns the number of parallel slices or tasks allocated for this process execution.

For MapReduce platforms, this is the same as the number of tasks for a given MapReduce job.

Specified by:
  `getNumProcessSlices` in class `FlowProcess<org.apache.hadoop.mapred.JobConf>`
Returns:
  an int

#### setReporter

    public void setReporter(org.apache.hadoop.mapred.Reporter reporter)

Method setReporter sets the reporter of this HadoopFlowProcess object.

Parameters:
  `reporter` - the reporter of this HadoopFlowProcess object.

#### getReporter

    public org.apache.hadoop.mapred.Reporter getReporter()

Specified by:
  `getReporter` in interface `MapRed`

#### setOutputCollector

    public void setOutputCollector(org.apache.hadoop.mapred.OutputCollector outputCollector)

#### getOutputCollector

    public org.apache.hadoop.mapred.OutputCollector getOutputCollector()

#### getProperty

    public java.lang.Object getProperty(java.lang.String key)

Description copied from class: `FlowProcess`
Method getProperty should be used to return configuration parameters from the underlying system.

In the case of Hadoop, the current Configuration will be queried.

Specified by:
  `getProperty` in class `FlowProcess<org.apache.hadoop.mapred.JobConf>`
Parameters:
  `key` - of type String
Returns:
  an Object

#### getPropertyKeys

    public java.util.Collection<java.lang.String> getPropertyKeys()

Description copied from class: `FlowProcess`
Method getPropertyKeys returns an immutable collection of all available property key values.

Specified by:
  `getPropertyKeys` in class `FlowProcess<org.apache.hadoop.mapred.JobConf>`
Returns:
  a Collection<String>

#### newInstance

    public java.lang.Object newInstance(java.lang.String className)

Description copied from class: `FlowProcess`
Method newInstance creates a new object instance from the given className argument delegating to any platform specific instantiation and configuration routines.

Specified by:
  `newInstance` in class `FlowProcess<org.apache.hadoop.mapred.JobConf>`
Returns:
  an instance of className

#### keepAlive

    public void keepAlive()

Description copied from class: `FlowProcess`
Method keepAlive notifies the system that the current process is still alive. Use this method if a particular `Operation` takes some moments to complete. Each system is different, so calling ping every few seconds to every minute or so would be best.

This method will fail silently if the underlying mechanism to notify keepAlive status are not initialized.

Specified by:
  `keepAlive` in class `FlowProcess<org.apache.hadoop.mapred.JobConf>`

#### increment

    public void increment(java.lang.Enum counter,
                 long amount)

Description copied from class: `FlowProcess`
Method increment is used to increment a custom counter. Counters must be of type Enum. The amount to increment must be a integer value.

This method will fail if the underlying counter infrastructure is unavailable. See `FlowProcess.isCounterStatusInitialized()`.

Specified by:
  `increment` in class `FlowProcess<org.apache.hadoop.mapred.JobConf>`
Parameters:
  `counter` - of type Enum
  `amount` - of type int

#### increment

    public void increment(java.lang.String group,
                 java.lang.String counter,
                 long amount)

Description copied from class: `FlowProcess`
Method increment is used to increment a custom counter. The amount to increment must be a integer value.

This method will fail if the underlying counter infrastructure is unavailable. See `FlowProcess.isCounterStatusInitialized()`.

Specified by:
  `increment` in class `FlowProcess<org.apache.hadoop.mapred.JobConf>`
Parameters:
  `group` - of type String
  `counter` - of type String
  `amount` - of type int

#### getCounterValue

    public long getCounterValue(java.lang.Enum counter)

Description copied from class: `FlowProcess`
Method getCounterValue is used to retrieve a counter value.

This method will fail if the underlying counter infrastructure is unavailable. See `FlowProcess.isCounterStatusInitialized()`.

Specified by:
  `getCounterValue` in class `FlowProcess<org.apache.hadoop.mapred.JobConf>`
Parameters:
  `counter` - of type Enum

#### getCounterValue

    public long getCounterValue(java.lang.String group,
                       java.lang.String counter)

Description copied from class: `FlowProcess`
Method getCounterValue is used to retrieve a counter value.

This method will fail if the underlying counter infrastructure is unavailable. See `FlowProcess.isCounterStatusInitialized()`.

Specified by:
  `getCounterValue` in class `FlowProcess<org.apache.hadoop.mapred.JobConf>`
Parameters:
  `group` - of type String
  `counter` - of type String

#### setStatus

    public void setStatus(java.lang.String status)

Description copied from class: `FlowProcess`
Method setStatus is used to set the status of the current operation.

This method will fail if the underlying counter infrastructure is unavailable. See `FlowProcess.isCounterStatusInitialized()`.

Specified by:
  `setStatus` in class `FlowProcess<org.apache.hadoop.mapred.JobConf>`
Parameters:
  `status` - of type String

#### isCounterStatusInitialized

    public boolean isCounterStatusInitialized()

Description copied from class: `FlowProcess`
Method isCounterStatusInitialized returns true if it is safe to increment a counter or set a status.

Specified by:
  `isCounterStatusInitialized` in class `FlowProcess<org.apache.hadoop.mapred.JobConf>`
Returns:
  boolean

#### openTapForRead

    public TupleEntryIterator openTapForRead(Tap tap)
                                      throws java.io.IOException

Description copied from class: `FlowProcess`
Method openTapForRead return a `TupleEntryIterator` for the given Tap instance.

Note the returned iterator will return the same instance of `TupleEntry` on every call, thus a copy must be made of either the TupleEntry or the underlying `Tuple` instance if they are to be stored in a Collection.

Specified by:
  `openTapForRead` in class `FlowProcess<org.apache.hadoop.mapred.JobConf>`
Parameters:
  `tap` - of type Tap
Returns:
  TupleIterator
Throws:
  `java.io.IOException` - when there is a failure opening the resource

#### openTapForWrite

    public TupleEntryCollector openTapForWrite(Tap tap)
                                        throws java.io.IOException

Description copied from class: `FlowProcess`
Method openTapForWrite returns a (@link TupleCollector} for the given Tap instance.

Specified by:
  `openTapForWrite` in class `FlowProcess<org.apache.hadoop.mapred.JobConf>`
Parameters:
  `tap` - of type Tap
Returns:
  TupleCollector
Throws:
  `java.io.IOException` - when there is a failure opening the resource

#### openTrapForWrite

    public TupleEntryCollector openTrapForWrite(Tap trap)
                                         throws java.io.IOException

Description copied from class: `FlowProcess`
Method openTrapForWrite returns a (@link TupleCollector} for the given Tap instance.

Specified by:
  `openTrapForWrite` in class `FlowProcess<org.apache.hadoop.mapred.JobConf>`
Parameters:
  `trap` - of type Tap
Returns:
  TupleCollector
Throws:
  `java.io.IOException` - when there is a failure opening the resource

#### openSystemIntermediateForWrite

    public TupleEntryCollector openSystemIntermediateForWrite()
                                                       throws java.io.IOException

Specified by:
  `openSystemIntermediateForWrite` in class `FlowProcess<org.apache.hadoop.mapred.JobConf>`
Throws:
  `java.io.IOException`

#### copyConfig

    public <C> C copyConfig(C config)

Specified by:
  `copyConfig` in class `FlowProcess<org.apache.hadoop.mapred.JobConf>`

#### diffConfigIntoMap

    public <C> java.util.Map<java.lang.String,java.lang.String> diffConfigIntoMap(C defaultConfig,
                                                                         C updatedConfig)

Specified by:
  `diffConfigIntoMap` in class `FlowProcess<org.apache.hadoop.mapred.JobConf>`

#### mergeMapIntoConfig

    public org.apache.hadoop.mapred.JobConf mergeMapIntoConfig(org.apache.hadoop.mapred.JobConf defaultConfig,
                                                      java.util.Map<java.lang.String,java.lang.String> map)

Specified by:
  `mergeMapIntoConfig` in class `FlowProcess<org.apache.hadoop.mapred.JobConf>`