cascading.pipe.assembly

## Class Unique

- java.lang.Object
- - cascading.pipe.Pipe
  - - cascading.pipe.SubAssembly
    - - cascading.pipe.assembly.Unique

- All Implemented Interfaces:
    FlowElement, ScopedElement, Traceable, java.io.Serializable

      public class Unique
      extends SubAssembly

  Class Unique `SubAssembly` is used to filter all duplicates out of a tuple stream.

  Typically finding unique value in a tuple stream relies on a `GroupBy` and a `FirstNBuffer` `Buffer` operation.

  If the `include` value is set to `Unique.Include.NO_NULLS`, any tuple consisting of only `null` values will be removed from the stream.

  This SubAssembly uses the `Unique.FilterPartialDuplicates` `Filter` to remove as many observed duplicates before the GroupBy operator to reduce IO over the network.

  This strategy is similar to using `combiners`, except no sorting or serialization is invoked and results in a much simpler mechanism.

  Unique uses a `CascadingCache` or LRU to do the filtering. To tune the cache, set the `capacity` value to a high enough value to utilize available memory. Or set a default value via the `UniqueProps.UNIQUE_CACHE_CAPACITY` property. The current default is `10, 000` unique keys.

  The LRU cache is pluggable and defaults to `LRUHashMapCache`. It can be changed by setting `UniqueProps.UNIQUE_CACHE_FACTORY` property to the name of a sub-class of `BaseCacheFactory`.

  The `capacity` value tells the underlying FilterPartialDuplicates how many values to cache for duplicate comparison before dropping values from the LRU cache.
  See Also:
    `LRUHashMapCacheFactory`, `DirectMappedCacheFactory`, `LRUHashMapCache`, `DirectMappedCache`, Serialized Form

- - ### Nested Class Summary

    Nested Classes
    Modifier and Type | Class and Description
    --- | ---
    `static class` | `Unique.Cache`
    `static class` | `Unique.FilterPartialDuplicates` Class FilterPartialDuplicates is a `Filter` that is used to remove observed duplicates from the tuple stream.
    `static class` | `Unique.Include`

  - ### Field Summary

    - ### Fields inherited from class cascading.pipe.Pipe

      `configDef, name, nodeConfigDef, parent, stepConfigDef`

  - ### Constructor Summary

    Constructors
    Constructor and Description
    `Unique(Pipe[] pipes, Fields uniqueFields)` Constructor Unique creates a new Unique instance.
    `Unique(Pipe[] pipes, Fields uniqueFields, int capacity)` Constructor Unique creates a new Unique instance.
    `Unique(Pipe[] pipes, Fields uniqueFields, Unique.Include include)` Constructor Unique creates a new Unique instance.
    `Unique(Pipe[] pipes, Fields uniqueFields, Unique.Include include, int capacity)` Constructor Unique creates a new Unique instance.
    `Unique(Pipe pipe, Fields uniqueFields)` Constructor Unique creates a new Unique instance.
    `Unique(Pipe pipe, Fields uniqueFields, int capacity)` Constructor Unique creates a new Unique instance.
    `Unique(Pipe pipe, Fields uniqueFields, Unique.Include include)` Constructor Unique creates a new Unique instance.
    `Unique(Pipe pipe, Fields uniqueFields, Unique.Include include, int capacity)` Constructor Unique creates a new Unique instance.
    `Unique(java.lang.String name, Pipe[] pipes, Fields uniqueFields)` Constructor Unique creates a new Unique instance.
    `Unique(java.lang.String name, Pipe[] pipes, Fields uniqueFields, int capacity)` Constructor Unique creates a new Unique instance.
    `Unique(java.lang.String name, Pipe[] pipes, Fields uniqueFields, Unique.Include include)` Constructor Unique creates a new Unique instance.
    `Unique(java.lang.String name, Pipe[] pipes, Fields uniqueFields, Unique.Include include, int capacity)` Constructor Unique creates a new Unique instance.
    `Unique(java.lang.String name, Pipe pipe, Fields uniqueFields)` Constructor Unique creates a new Unique instance.
    `Unique(java.lang.String name, Pipe pipe, Fields uniqueFields, int capacity)` Constructor Unique creates a new Unique instance.
    `Unique(java.lang.String name, Pipe pipe, Fields uniqueFields, Unique.Include include)` Constructor Unique creates a new Unique instance.
    `Unique(java.lang.String name, Pipe pipe, Fields uniqueFields, Unique.Include include, int capacity)` Constructor Unique creates a new Unique instance.

  - ### Method Summary

    - ### Methods inherited from class cascading.pipe.SubAssembly

      `getName, getPrevious, getTailNames, getTails, setPrevious, setTails, unwind`

    - ### Methods inherited from class cascading.pipe.Pipe

      `equals, getConfigDef, getHeads, getNodeConfigDef, getParent, getStepConfigDef, getTrace, hasConfigDef, hashCode, hasNodeConfigDef, hasStepConfigDef, id, named, names, outgoingScopeFor, pipes, print, printInternal, resolveIncomingOperationArgumentFields, resolveIncomingOperationPassThroughFields, setParent, toString`

    - ### Methods inherited from class java.lang.Object

      `clone, finalize, getClass, notify, notifyAll, wait, wait, wait`

- - ### Constructor Detail

    - #### Unique

          @ConstructorProperties(value={"pipe","uniqueFields"})
          public Unique(Pipe pipe,
                                      Fields uniqueFields)

      Constructor Unique creates a new Unique instance.
      Parameters:
        `pipe` - of type Pipe
        `uniqueFields` - of type Fields

    - #### Unique

          @ConstructorProperties(value={"pipe","uniqueFields","include"})
          public Unique(Pipe pipe,
                                      Fields uniqueFields,
                                      Unique.Include include)

      Constructor Unique creates a new Unique instance.
      Parameters:
        `pipe` - of type Pipe
        `uniqueFields` - of type Fields
        `include` - of type Include

    - #### Unique

          @ConstructorProperties(value={"pipe","uniqueFields","capacity"})
          public Unique(Pipe pipe,
                                      Fields uniqueFields,
                                      int capacity)

      Constructor Unique creates a new Unique instance.
      Parameters:
        `pipe` - of type Pipe
        `uniqueFields` - of type Fields
        `capacity` - of type int

    - #### Unique

          @ConstructorProperties(value={"pipe","uniqueFields","include","capacity"})
          public Unique(Pipe pipe,
                                      Fields uniqueFields,
                                      Unique.Include include,
                                      int capacity)

      Constructor Unique creates a new Unique instance.
      Parameters:
        `pipe` - of type Pipe
        `uniqueFields` - of type Fields
        `include` - of type Include
        `capacity` - of type int

    - #### Unique

          @ConstructorProperties(value={"name","pipe","uniqueFields"})
          public Unique(java.lang.String name,
                                      Pipe pipe,
                                      Fields uniqueFields)

      Constructor Unique creates a new Unique instance.
      Parameters:
        `name` - of type String
        `pipe` - of type Pipe
        `uniqueFields` - of type Fields

    - #### Unique

          @ConstructorProperties(value={"name","pipe","uniqueFields","include"})
          public Unique(java.lang.String name,
                                      Pipe pipe,
                                      Fields uniqueFields,
                                      Unique.Include include)

      Constructor Unique creates a new Unique instance.
      Parameters:
        `name` - of type String
        `pipe` - of type Pipe
        `uniqueFields` - of type Fields
        `include` - of type Include

    - #### Unique

          @ConstructorProperties(value={"name","pipe","uniqueFields","capacity"})
          public Unique(java.lang.String name,
                                      Pipe pipe,
                                      Fields uniqueFields,
                                      int capacity)

      Constructor Unique creates a new Unique instance.
      Parameters:
        `name` - of type String
        `pipe` - of type Pipe
        `uniqueFields` - of type Fields
        `capacity` - of type int

    - #### Unique

          @ConstructorProperties(value={"name","pipe","uniqueFields","include","capacity"})
          public Unique(java.lang.String name,
                                      Pipe pipe,
                                      Fields uniqueFields,
                                      Unique.Include include,
                                      int capacity)

      Constructor Unique creates a new Unique instance.
      Parameters:
        `name` - of type String
        `pipe` - of type Pipe
        `uniqueFields` - of type Fields
        `include` - of type Include
        `capacity` - of type int

    - #### Unique

          @ConstructorProperties(value={"pipes","uniqueFields"})
          public Unique(Pipe[] pipes,
                                      Fields uniqueFields)

      Constructor Unique creates a new Unique instance.
      Parameters:
        `pipes` - of type Pipe[]
        `uniqueFields` - of type Fields

    - #### Unique

          @ConstructorProperties(value={"pipes","uniqueFields","include"})
          public Unique(Pipe[] pipes,
                                      Fields uniqueFields,
                                      Unique.Include include)

      Constructor Unique creates a new Unique instance.
      Parameters:
        `pipes` - of type Pipe[]
        `uniqueFields` - of type Fields
        `include` - of type Include

    - #### Unique

          @ConstructorProperties(value={"pipes","uniqueFields","capacity"})
          public Unique(Pipe[] pipes,
                                      Fields uniqueFields,
                                      int capacity)

      Constructor Unique creates a new Unique instance.
      Parameters:
        `pipes` - of type Pipe[]
        `uniqueFields` - of type Fields
        `capacity` - of type int

    - #### Unique

          @ConstructorProperties(value={"pipes","uniqueFields","include","capacity"})
          public Unique(Pipe[] pipes,
                                      Fields uniqueFields,
                                      Unique.Include include,
                                      int capacity)

      Constructor Unique creates a new Unique instance.
      Parameters:
        `pipes` - of type Pipe[]
        `uniqueFields` - of type Fields
        `include` - of type Include
        `capacity` - of type int

    - #### Unique

          @ConstructorProperties(value={"name","pipes","uniqueFields"})
          public Unique(java.lang.String name,
                                      Pipe[] pipes,
                                      Fields uniqueFields)

      Constructor Unique creates a new Unique instance.
      Parameters:
        `name` - of type String
        `pipes` - of type Pipe[]
        `uniqueFields` - of type Fields

    - #### Unique

          @ConstructorProperties(value={"name","pipes","uniqueFields","include"})
          public Unique(java.lang.String name,
                                      Pipe[] pipes,
                                      Fields uniqueFields,
                                      Unique.Include include)

      Constructor Unique creates a new Unique instance.
      Parameters:
        `name` - of type String
        `pipes` - of type Pipe[]
        `uniqueFields` - of type Fields
        `include` - of type Include

    - #### Unique

          @ConstructorProperties(value={"name","pipes","uniqueFields","capacity"})
          public Unique(java.lang.String name,
                                      Pipe[] pipes,
                                      Fields uniqueFields,
                                      int capacity)

      Constructor Unique creates a new Unique instance.
      Parameters:
        `name` - of type String
        `pipes` - of type Pipe[]
        `uniqueFields` - of type Fields
        `capacity` - of type int

    - #### Unique

          @ConstructorProperties(value={"name","pipes","uniqueFields","include","capacity"})
          public Unique(java.lang.String name,
                                      Pipe[] pipes,
                                      Fields uniqueFields,
                                      Unique.Include include,
                                      int capacity)

      Constructor Unique creates a new Unique instance.
      Parameters:
        `name` - of type String
        `pipes` - of type Pipe[]
        `uniqueFields` - of type Fields
        `capacity` - of type int

Copyright © 2007-2015 Xplenty, Inc. All Rights Reserved.