cascading.pipe

## Class Pipe

- java.lang.Object
- - cascading.pipe.Pipe

- All Implemented Interfaces:
    FlowElement, ScopedElement, Traceable, java.io.Serializable
  Direct Known Subclasses:
    Boundary, Checkpoint, Operator, Splice, SubAssembly

      public class Pipe
      extends java.lang.Object
      implements ScopedElement, FlowElement, java.io.Serializable, Traceable

  Class Pipe is used to name branches in pipe assemblies, and as a base class for core processing model types, specifically `Each`, `Every`, `GroupBy`, `CoGroup`, `Merge`, `HashJoin`, and `SubAssembly`.

  Pipes are chained together through their constructors.

  To effect a split in the pipe, simply pass a Pipe instance to two or more constructors of subsequent Pipe instances. A join can be achieved by passing two or more Pipe instances to a `CoGroup` or `HashJoin` pipe.

  A merge can be achieved by passing two or more Pipe instances to a `GroupBy` or `Merge` pipe.
  See Also:
    `Each`, `Every`, `GroupBy`, `Merge`, `CoGroup`, `HashJoin`, `SubAssembly`, Serialized Form

- - ### Field Summary

    Fields
    Modifier and Type | Field and Description
    --- | ---
    `protected ConfigDef` | `configDef`
    `protected java.lang.String` | `name` Field name
    `protected ConfigDef` | `nodeConfigDef`
    `protected Pipe` | `parent` Field parent
    `protected Pipe` | `previous` Field previous
    `protected ConfigDef` | `stepConfigDef`

  - ### Constructor Summary

    Constructors
    Modifier | Constructor and Description
    --- | ---
    `protected` | `Pipe()`
    `protected` | `Pipe(Pipe previous)`
    `` | `Pipe(java.lang.String name)` Constructor Pipe creates a new Pipe instance with the given name.
    `` | `Pipe(java.lang.String name, Pipe previous)` Constructor Pipe creates a new Pipe instance with the given name and previous Pipe instance.

  - ### Method Summary

    Methods
    Modifier and Type | Method and Description
    --- | ---
    `boolean` | `equals(java.lang.Object object)`
    `ConfigDef` | `getConfigDef()` Returns a `ConfigDef` instance that allows for local properties to be set and made available via a resulting `FlowProcess` instance when the pipe is invoked.
    `Pipe[]` | `getHeads()` Method getHeads returns the first Pipe instances in this pipe assembly.
    `java.lang.String` | `getName()` Get the name of this pipe.
    `ConfigDef` | `getNodeConfigDef()` Returns a `ConfigDef` instance that allows for process level properties to be set and made available via a resulting `FlowProcess` instance when the pipe is invoked.
    `Pipe` | `getParent()` Returns the enclosing parent Pipe instance, if any.
    `Pipe[]` | `getPrevious()` Get all the upstream pipes this pipe is connected to.
    `ConfigDef` | `getStepConfigDef()` Returns a `ConfigDef` instance that allows for process level properties to be set and made available via a resulting `FlowProcess` instance when the pipe is invoked.
    `java.lang.String` | `getTrace()` Method getTrace returns a String that pinpoints the caller that created this instance.
    `boolean` | `hasConfigDef()` Returns `true` if there are properties in the configDef instance.
    `int` | `hashCode()`
    `boolean` | `hasNodeConfigDef()` Returns `true` if there are properties in the nodeConfigDef instance.
    `boolean` | `hasStepConfigDef()` Returns `true` if there are properties in the stepConfigDef instance.
    `static java.lang.String` | `id(Pipe pipe)`
    `static Pipe[]` | `named(java.lang.String name, Pipe... tails)`
    `static java.lang.String[]` | `names(Pipe... tails)` Convenience method for finding all Pipe names in an assembly.
    `Scope` | `outgoingScopeFor(java.util.Set<Scope> incomingScopes)` Method outgoingScopeFor returns the Scope this FlowElement hands off to the next FlowElement.
    `static Pipe[]` | `pipes(Pipe... pipes)` Convenience method to create an array of Pipe instances.
    `java.lang.String` | `print(Scope scope)` Method print is used internally.
    `protected void` | `printInternal(java.lang.StringBuffer buffer, Scope scope)`
    `Fields` | `resolveIncomingOperationArgumentFields(Scope incomingScope)` Method resolveIncomingOperationArgumentFields returns the Fields outgoing from the previous FlowElement that are consumable by this FlowElement when preparing Operation arguments.
    `Fields` | `resolveIncomingOperationPassThroughFields(Scope incomingScope)` Method resolveIncomingOperationPassThroughFields returns the Fields outgoing from the previous FlowElement that are consumable by this FlowElement when preparing the Pipe outgoing tuple.
    `protected void` | `setParent(Pipe parent)`
    `java.lang.String` | `toString()`

    - ### Methods inherited from class java.lang.Object

      `clone, finalize, getClass, notify, notifyAll, wait, wait, wait`

- - ### Field Detail

    - #### name

          protected java.lang.String name

      Field name

    - #### previous

          protected Pipe previous

      Field previous

    - #### parent

          protected Pipe parent

      Field parent

    - #### configDef

          protected ConfigDef configDef

    - #### stepConfigDef

          protected ConfigDef stepConfigDef

    - #### nodeConfigDef

          protected ConfigDef nodeConfigDef

  - ### Constructor Detail

    - #### Pipe

          protected Pipe()

    - #### Pipe

          @ConstructorProperties(value="previous")
          protected Pipe(Pipe previous)

    - #### Pipe

          @ConstructorProperties(value="name")
          public Pipe(java.lang.String name)

      Constructor Pipe creates a new Pipe instance with the given name. This is useful as the 'start' or head of a pipe assembly.
      Parameters:
        `name` - name for this branch of Pipes

    - #### Pipe

          @ConstructorProperties(value={"name","previous"})
          public Pipe(java.lang.String name,
                                    Pipe previous)

      Constructor Pipe creates a new Pipe instance with the given name and previous Pipe instance. This is useful for naming a branch in a pipe assembly. Or renaming the branch mid-way down.
      Parameters:
        `name` - name for this branch of Pipes
        `previous` - previous Pipe to receive input Tuples from

  - ### Method Detail

    - #### id

          public static java.lang.String id(Pipe pipe)

    - #### pipes

          public static Pipe[] pipes(Pipe... pipes)

      Convenience method to create an array of Pipe instances.
      Parameters:
        `pipes` - vararg list of pipes
      Returns:
        array of pipes

    - #### names

          public static java.lang.String[] names(Pipe... tails)

      Convenience method for finding all Pipe names in an assembly.
      Parameters:
        `tails` - vararg list of all tails in given assembly
      Returns:
        array of Pipe names

    - #### named

          public static Pipe[] named(java.lang.String name,
                     Pipe... tails)

    - #### getName

          public java.lang.String getName()

      Get the name of this pipe. Guaranteed non-null.
      Returns:
        String the name of this pipe

    - #### getPrevious

          public Pipe[] getPrevious()

      Get all the upstream pipes this pipe is connected to. This method will return the Pipe instances passed on the constructors as inputs to this Pipe instance.
      Returns:
        all the upstream pipes this pipe is connected to.

    - #### setParent

          protected void setParent(Pipe parent)

    - #### getParent

          public Pipe getParent()

      Returns the enclosing parent Pipe instance, if any. A parent is typically a `SubAssembly` that wraps this instance.
      Returns:
        of type Pipe

    - #### getConfigDef

          public ConfigDef getConfigDef()

      Returns a `ConfigDef` instance that allows for local properties to be set and made available via a resulting `FlowProcess` instance when the pipe is invoked.

      Any properties set on the configDef will not show up in any `Flow` or `FlowStep` process level configuration, but will override any of those values as seen by the current Pipe instance.
      Specified by:
        `getConfigDef` in interface `ScopedElement`
      Returns:
        an instance of ConfigDef

    - #### hasConfigDef

          public boolean hasConfigDef()

      Returns `true` if there are properties in the configDef instance.
      Specified by:
        `hasConfigDef` in interface `ScopedElement`
      Returns:
        true if there are configDef properties

    - #### getNodeConfigDef

          public ConfigDef getNodeConfigDef()

      Returns a `ConfigDef` instance that allows for process level properties to be set and made available via a resulting `FlowProcess` instance when the pipe is invoked.

      Any properties set on the nodeConfigDef will not show up in any Flow configuration, but will show up in the current process `FlowNode` (in Apache Tez the Vertex configuration). Any value set in the nodeConfigDef will be overridden by the pipe local `#getConfigDef` instance. Use this method to tweak properties in the process node this pipe instance is planned into. In the case of the Apache Tez platform, when set on a `GroupBy` instance, the number of gather partitions can be modified.

      In the case of any Pipe that spans FlowNode boundaries, like GroupBy and CoGroup may on some platforms, any ConfigDef properties will be applied to the downstream FlowNode. That is, if a GroupBy is the source to a node, any node ConfigDef properties will be applied. If the GroupBy encountered when applying properties is on the sink side of a node, the properties will be ignored.
      Specified by:
        `getNodeConfigDef` in interface `ScopedElement`
      Returns:
        an instance of ConfigDef

    - #### hasNodeConfigDef

          public boolean hasNodeConfigDef()

      Returns `true` if there are properties in the nodeConfigDef instance.
      Specified by:
        `hasNodeConfigDef` in interface `ScopedElement`
      Returns:
        true if there are nodeConfigDef properties

    - #### getStepConfigDef

          public ConfigDef getStepConfigDef()

      Returns a `ConfigDef` instance that allows for process level properties to be set and made available via a resulting `FlowProcess` instance when the pipe is invoked.

      Any properties set on the stepConfigDef will not show up in any Flow configuration, but will show up in the current process `FlowStep` (in Hadoop the MapReduce jobconf). Any value set in the stepConfigDef will be overridden by the pipe local `#getConfigDef` instance. Use this method to tweak properties in the process step this pipe instance is planned into. In the case of the Hadoop platform, when set on a `GroupBy` instance, the number of reducers can be modified.
      Specified by:
        `getStepConfigDef` in interface `ScopedElement`
      Returns:
        an instance of ConfigDef

    - #### hasStepConfigDef

          public boolean hasStepConfigDef()

      Returns `true` if there are properties in the stepConfigDef instance.
      Specified by:
        `hasStepConfigDef` in interface `ScopedElement`
      Returns:
        true if there are stepConfigDef properties

    - #### getHeads

          public Pipe[] getHeads()

      Method getHeads returns the first Pipe instances in this pipe assembly.
      Returns:
        the first (type Pipe[]) of this Pipe object.

    - #### outgoingScopeFor

          public Scope outgoingScopeFor(java.util.Set<Scope> incomingScopes)

      Description copied from interface: `ScopedElement`
      Method outgoingScopeFor returns the Scope this FlowElement hands off to the next FlowElement.
      Specified by:
        `outgoingScopeFor` in interface `ScopedElement`
      Parameters:
        `incomingScopes` - of type Set
      Returns:
        Scope

    - #### resolveIncomingOperationArgumentFields

          public Fields resolveIncomingOperationArgumentFields(Scope incomingScope)

      Description copied from interface: `ScopedElement`
      Method resolveIncomingOperationArgumentFields returns the Fields outgoing from the previous FlowElement that are consumable by this FlowElement when preparing Operation arguments.
      Specified by:
        `resolveIncomingOperationArgumentFields` in interface `ScopedElement`
      Parameters:
        `incomingScope` - of type Scope
      Returns:
        Fields

    - #### resolveIncomingOperationPassThroughFields

          public Fields resolveIncomingOperationPassThroughFields(Scope incomingScope)

      Description copied from interface: `ScopedElement`
      Method resolveIncomingOperationPassThroughFields returns the Fields outgoing from the previous FlowElement that are consumable by this FlowElement when preparing the Pipe outgoing tuple.
      Specified by:
        `resolveIncomingOperationPassThroughFields` in interface `ScopedElement`
      Parameters:
        `incomingScope` - of type Scope
      Returns:
        Fields

    - #### getTrace

          public java.lang.String getTrace()

      Description copied from interface: `Traceable`
      Method getTrace returns a String that pinpoints the caller that created this instance.
      Specified by:
        `getTrace` in interface `Traceable`
      Returns:
        String

    - #### toString

          public java.lang.String toString()

      Overrides:
        `toString` in class `java.lang.Object`

    - #### equals

          public boolean equals(java.lang.Object object)

      Overrides:
        `equals` in class `java.lang.Object`

    - #### hashCode

          public int hashCode()

      Overrides:
        `hashCode` in class `java.lang.Object`

    - #### print

          public java.lang.String print(Scope scope)

      Method print is used internally.
      Parameters:
        `scope` - of type Scope
      Returns:
        String

    - #### printInternal

          protected void printInternal(java.lang.StringBuffer buffer,
                           Scope scope)

Copyright © 2007-2015 Xplenty, Inc. All Rights Reserved.