cascading.platform.hadoop

## Class BaseHadoopPlatform<Config extends Configuration>

- java.lang.Object
- - cascading.platform.TestPlatform
  - - cascading.platform.hadoop.BaseHadoopPlatform<Config>

- Direct Known Subclasses:
    Hadoop2MR1Platform

      public abstract class BaseHadoopPlatform<Config extends Configuration>
      extends cascading.platform.TestPlatform

- - ### Field Summary

    Fields
    Modifier and Type | Field and Description
    --- | ---
    `static Configuration` | `configuration`
    `static FileSystem` | `fileSys`
    `protected java.lang.String` | `logger`
    `static java.util.Map<java.lang.Object,java.lang.Object>` | `properties`

    - ### Fields inherited from class cascading.platform.TestPlatform

      `CLUSTER_TESTING_PROPERTY, numGatherPartitions, numMappers, numReducers, PLATFORM_PREFIX`

  - ### Constructor Summary

    Constructors
    Constructor and Description
    `BaseHadoopPlatform()`

  - ### Method Summary

    Methods
    Modifier and Type | Method and Description
    --- | ---
    `void` | `copyFromLocal(java.lang.String inputFile)`
    `void` | `copyToLocal(java.lang.String outputFile)`
    `abstract Config` | `getConfiguration()`
    `Tap` | `getDelimitedFile(Fields fields, boolean skipHeader, boolean writeHeader, java.lang.String delimiter, java.lang.String quote, java.lang.Class[] types, java.lang.String filename, SinkMode mode)`
    `Tap` | `getDelimitedFile(Fields fields, boolean hasHeader, java.lang.String delimiter, java.lang.String quote, java.lang.Class[] types, java.lang.String filename, SinkMode mode)`
    `Tap` | `getDelimitedFile(java.lang.String delimiter, java.lang.String quote, FieldTypeResolver fieldTypeResolver, java.lang.String filename, SinkMode mode)`
    `java.lang.String` | `getHiddenTemporaryPath()`
    `java.util.Comparator` | `getLongComparator(boolean reverseSort)`
    `Tap` | `getPartitionTap(Tap sink, Partition partition, int openThreshold)`
    `java.util.Map<java.lang.Object,java.lang.Object>` | `getProperties()`
    `java.util.Comparator` | `getStringComparator(boolean reverseSort)`
    `Tap` | `getTap(Scheme scheme, java.lang.String filename, SinkMode mode)`
    `Scheme` | `getTestConfigDefScheme()`
    `Scheme` | `getTestFailScheme()`
    `Tap` | `getTextFile(Fields sourceFields, Fields sinkFields, java.lang.String filename, SinkMode mode)`
    `boolean` | `isHDFSAvailable()`
    `boolean` | `isMapReduce()`
    `boolean` | `remoteExists(java.lang.String outputFile)`
    `boolean` | `remoteRemove(java.lang.String outputFile, boolean recursive)`
    `protected java.lang.String` | `safeFileName(java.lang.String filename)` Replaces characters, that are not allowed by HDFS with an "_".
    `void` | `setNumGatherPartitions(int numGatherPartitions)`
    `void` | `setNumMappers(int numMapTasks)`
    `void` | `setNumReducers(int numReduceTasks)`
    `void` | `tearDown()`

    - ### Methods inherited from class cascading.platform.TestPlatform

      `getApplicationJar, getDelimitedFile, getDelimitedFile, getDelimitedFile, getDelimitedFile, getDelimitedFile, getFlowConnector, getFlowConnector, getFlowProcess, getGlobalProperties, getName, getNumGatherPartitions, getNumGatherPartitionTasks, getNumMappers, getNumMapTasks, getNumReducers, getNumReduceTasks, getTabDelimitedFile, getTabDelimitedFile, getTextFile, getTextFile, getTextFile, getTextFile, isDAG, isUseCluster, setNumGatherPartitionTasks, setNumMapTasks, setNumReduceTasks, setUp, setUseCluster, supportsGroupByAfterMerge`

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- - ### Field Detail

    - #### fileSys

          public static transient FileSystem fileSys

    - #### configuration

          public static transient Configuration configuration

    - #### properties

          public static transient java.util.Map<java.lang.Object,java.lang.Object> properties

    - #### logger

          protected java.lang.String logger

  - ### Constructor Detail

    - #### BaseHadoopPlatform

          public BaseHadoopPlatform()

  - ### Method Detail

    - #### isMapReduce

          public boolean isMapReduce()

      Overrides:
        `isMapReduce` in class `cascading.platform.TestPlatform`

    - #### setNumMappers

          public void setNumMappers(int numMapTasks)

      Overrides:
        `setNumMappers` in class `cascading.platform.TestPlatform`

    - #### setNumReducers

          public void setNumReducers(int numReduceTasks)

      Overrides:
        `setNumReducers` in class `cascading.platform.TestPlatform`

    - #### setNumGatherPartitions

          public void setNumGatherPartitions(int numGatherPartitions)

      Overrides:
        `setNumGatherPartitions` in class `cascading.platform.TestPlatform`

    - #### getProperties

          public java.util.Map<java.lang.Object,java.lang.Object> getProperties()

      Specified by:
        `getProperties` in class `cascading.platform.TestPlatform`

    - #### tearDown

          public void tearDown()

      Specified by:
        `tearDown` in class `cascading.platform.TestPlatform`

    - #### getConfiguration

          public abstract Config getConfiguration()

    - #### isHDFSAvailable

          public boolean isHDFSAvailable()

    - #### copyFromLocal

          public void copyFromLocal(java.lang.String inputFile)
                             throws java.io.IOException

      Specified by:
        `copyFromLocal` in class `cascading.platform.TestPlatform`
      Throws:
        `java.io.IOException`

    - #### copyToLocal

          public void copyToLocal(java.lang.String outputFile)
                           throws java.io.IOException

      Specified by:
        `copyToLocal` in class `cascading.platform.TestPlatform`
      Throws:
        `java.io.IOException`

    - #### remoteExists

          public boolean remoteExists(java.lang.String outputFile)
                               throws java.io.IOException

      Specified by:
        `remoteExists` in class `cascading.platform.TestPlatform`
      Throws:
        `java.io.IOException`

    - #### remoteRemove

          public boolean remoteRemove(java.lang.String outputFile,
                             boolean recursive)
                               throws java.io.IOException

      Specified by:
        `remoteRemove` in class `cascading.platform.TestPlatform`
      Throws:
        `java.io.IOException`

    - #### getTap

          public Tap getTap(Scheme scheme,
                   java.lang.String filename,
                   SinkMode mode)

      Specified by:
        `getTap` in class `cascading.platform.TestPlatform`

    - #### getTextFile

          public Tap getTextFile(Fields sourceFields,
                        Fields sinkFields,
                        java.lang.String filename,
                        SinkMode mode)

      Specified by:
        `getTextFile` in class `cascading.platform.TestPlatform`

    - #### getDelimitedFile

          public Tap getDelimitedFile(Fields fields,
                             boolean hasHeader,
                             java.lang.String delimiter,
                             java.lang.String quote,
                             java.lang.Class[] types,
                             java.lang.String filename,
                             SinkMode mode)

      Specified by:
        `getDelimitedFile` in class `cascading.platform.TestPlatform`

    - #### getDelimitedFile

          public Tap getDelimitedFile(Fields fields,
                             boolean skipHeader,
                             boolean writeHeader,
                             java.lang.String delimiter,
                             java.lang.String quote,
                             java.lang.Class[] types,
                             java.lang.String filename,
                             SinkMode mode)

      Specified by:
        `getDelimitedFile` in class `cascading.platform.TestPlatform`

    - #### getDelimitedFile

          public Tap getDelimitedFile(java.lang.String delimiter,
                             java.lang.String quote,
                             FieldTypeResolver fieldTypeResolver,
                             java.lang.String filename,
                             SinkMode mode)

      Specified by:
        `getDelimitedFile` in class `cascading.platform.TestPlatform`

    - #### getPartitionTap

          public Tap getPartitionTap(Tap sink,
                            Partition partition,
                            int openThreshold)

      Specified by:
        `getPartitionTap` in class `cascading.platform.TestPlatform`

    - #### getTestConfigDefScheme

          public Scheme getTestConfigDefScheme()

      Specified by:
        `getTestConfigDefScheme` in class `cascading.platform.TestPlatform`

    - #### getTestFailScheme

          public Scheme getTestFailScheme()

      Specified by:
        `getTestFailScheme` in class `cascading.platform.TestPlatform`

    - #### getLongComparator

          public java.util.Comparator getLongComparator(boolean reverseSort)

      Specified by:
        `getLongComparator` in class `cascading.platform.TestPlatform`

    - #### getStringComparator

          public java.util.Comparator getStringComparator(boolean reverseSort)

      Specified by:
        `getStringComparator` in class `cascading.platform.TestPlatform`

    - #### getHiddenTemporaryPath

          public java.lang.String getHiddenTemporaryPath()

      Specified by:
        `getHiddenTemporaryPath` in class `cascading.platform.TestPlatform`

    - #### safeFileName

          protected java.lang.String safeFileName(java.lang.String filename)

      Replaces characters, that are not allowed by HDFS with an "_".
      Parameters:
        `filename` - The filename to make safe
      Returns:
        The filename with all non-supported characters removed.

Copyright © 2007-2015 Xplenty, Inc. All Rights Reserved.