cascading.tap.hadoop

## Class Hfs

- java.lang.Object
- - cascading.tap.Tap<org.apache.hadoop.conf.Configuration,org.apache.hadoop.mapred.RecordReader,org.apache.hadoop.mapred.OutputCollector>
  - - cascading.tap.hadoop.Hfs

- All Implemented Interfaces:
    FlowElement, ScopedElement, FileType<org.apache.hadoop.conf.Configuration>, Traceable, java.io.Serializable
  Direct Known Subclasses:
    Dfs, Lfs, TempHfs

      public class Hfs
      extends Tap<org.apache.hadoop.conf.Configuration,org.apache.hadoop.mapred.RecordReader,org.apache.hadoop.mapred.OutputCollector>
      implements FileType<org.apache.hadoop.conf.Configuration>

  Class Hfs is the base class for all Hadoop file system access. Hfs may only be used with the Hadoop `FlowConnector` sub-classes when creating Hadoop executable `Flow` instances.

  Paths typically should point to a directory, where in turn all the "part" files immediately in that directory will be included. This is the practice Hadoop expects. Sub-directories are not included and typically result in a failure.

  To include sub-directories, Hadoop supports "globing". Globing is a frustrating feature and is supported more robustly by `GlobHfs` and less so by Hfs.

  Hfs will accept `/*` (wildcard) paths, but not all convenience methods like `jobConf.getSize` will behave properly or reliably. Nor can the Hfs instance with a wildcard path be used as a sink to write data.

  In those cases use GlobHfs since it is a sub-class of `MultiSourceTap`.

  Optionally use `Dfs` or `Lfs` for resources specific to Hadoop Distributed file system or the Local file system, respectively. Using Hfs is the best practice when possible, Lfs and Dfs are conveniences.

  Use the Hfs class if the 'kind' of resource is unknown at design time. To use, prefix a scheme to the 'stringPath'. Where `hdfs://...` will denote Dfs, and `file://...` will denote Lfs.

  Call `HfsProps.setTemporaryDirectory(java.util.Map, String)` to use a different temporary file directory path other than the current Hadoop default path.

  By default Cascading on Hadoop will assume any source or sink Tap using the `file://` URI scheme intends to read files from the local client filesystem (for example when using the `Lfs` Tap) where the Hadoop job jar is started. Subsequently Cascading will force any MapReduce jobs reading or writing to `file://` resources to run in Hadoop "standalone mode" so that the file can be read.

  To change this behavior, `HfsProps.setLocalModeScheme(java.util.Map, String)` to set a different scheme value, or to "none" to disable entirely for the case the file to be read is available on every Hadoop processing node in the exact same path.

  When using a MapReduce planner, Hfs can optionally combine multiple small files (or a series of small "blocks") into larger "splits". This reduces the number of resulting map tasks created by Hadoop and can improve application performance.

  This is enabled by calling `HfsProps.setUseCombinedInput(boolean)` to `true`. By default, merging or combining splits into large ones is disabled.

  Apache Tez planner does not require this setting, it is supported by default and enabled by the application manager.
  See Also:
    Serialized Form

- - ### Field Summary

    Fields
    Modifier and Type | Field and Description
    --- | ---
    `protected java.lang.String` | `stringPath` Field stringPath

  - ### Constructor Summary

    Constructors
    Modifier | Constructor and Description
    --- | ---
    `protected` | `Hfs()`
    `protected` | `Hfs(Scheme<org.apache.hadoop.conf.Configuration,org.apache.hadoop.mapred.RecordReader,org.apache.hadoop.mapred.OutputCollector,?,?> scheme)`
    `` | `Hfs(Scheme<org.apache.hadoop.conf.Configuration,org.apache.hadoop.mapred.RecordReader,org.apache.hadoop.mapred.OutputCollector,?,?> scheme, java.lang.String stringPath)` Constructor Hfs creates a new Hfs instance.
    `` | `Hfs(Scheme<org.apache.hadoop.conf.Configuration,org.apache.hadoop.mapred.RecordReader,org.apache.hadoop.mapred.OutputCollector,?,?> scheme, java.lang.String stringPath, SinkMode sinkMode)` Constructor Hfs creates a new Hfs instance.

  - ### Method Summary

    Methods
    Modifier and Type | Method and Description
    --- | ---
    `protected void` | `applySourceConfInitIdentifiers(FlowProcess<? extends org.apache.hadoop.conf.Configuration> process, org.apache.hadoop.conf.Configuration conf, java.lang.String... fullIdentifiers)`
    `boolean` | `createResource(org.apache.hadoop.conf.Configuration conf)` Method createResource creates the underlying resource.
    `boolean` | `deleteChildResource(org.apache.hadoop.conf.Configuration conf, java.lang.String childIdentifier)`
    `boolean` | `deleteChildResource(FlowProcess<? extends org.apache.hadoop.conf.Configuration> flowProcess, java.lang.String childIdentifier)`
    `boolean` | `deleteResource(org.apache.hadoop.conf.Configuration conf)` Method deleteResource deletes the resource represented by this instance.
    `long` | `getBlockSize(org.apache.hadoop.conf.Configuration conf)` Method getBlockSize returns the `blocksize` specified by the underlying file system for this resource.
    `long` | `getBlockSize(FlowProcess<? extends org.apache.hadoop.conf.Configuration> flowProcess)` Method getBlockSize returns the `blocksize` specified by the underlying file system for this resource.
    `java.lang.String[]` | `getChildIdentifiers(org.apache.hadoop.conf.Configuration conf)` Method getChildIdentifiers returns an array of child identifiers if this resource is a directory.
    `java.lang.String[]` | `getChildIdentifiers(org.apache.hadoop.conf.Configuration conf, int depth, boolean fullyQualified)`
    `java.lang.String[]` | `getChildIdentifiers(FlowProcess<? extends org.apache.hadoop.conf.Configuration> flowProcess)` Method getChildIdentifiers returns an array of child identifiers if this resource is a directory.
    `java.lang.String[]` | `getChildIdentifiers(FlowProcess<? extends org.apache.hadoop.conf.Configuration> flowProcess, int depth, boolean fullyQualified)`
    `protected static boolean` | `getCombinedInputSafeMode(org.apache.hadoop.conf.Configuration conf)`
    `protected org.apache.hadoop.fs.FileSystem` | `getDefaultFileSystem(org.apache.hadoop.conf.Configuration configuration)`
    `java.net.URI` | `getDefaultFileSystemURIScheme(org.apache.hadoop.conf.Configuration configuration)` Method getDefaultFileSystemURIScheme returns the URI scheme for the default Hadoop FileSystem.
    `org.apache.hadoop.fs.FileStatus` | `getFileStatus(org.apache.hadoop.conf.Configuration conf)`
    `protected org.apache.hadoop.fs.FileSystem` | `getFileSystem(org.apache.hadoop.conf.Configuration configuration)`
    `java.lang.String` | `getFullIdentifier(org.apache.hadoop.conf.Configuration conf)` Method getFullIdentifier returns a fully qualified resource identifier.
    `java.lang.String` | `getIdentifier()` Method getIdentifier returns a String representing the resource this Tap instance represents.
    `protected static java.lang.String` | `getLocalModeScheme(org.apache.hadoop.conf.Configuration conf, java.lang.String defaultValue)`
    `long` | `getModifiedTime(org.apache.hadoop.conf.Configuration conf)` Method getModifiedTime returns the date this resource was last modified.
    `org.apache.hadoop.fs.Path` | `getPath()`
    `int` | `getReplication(org.apache.hadoop.conf.Configuration conf)` Method getReplication returns the `replication` specified by the underlying file system for this resource.
    `int` | `getReplication(FlowProcess<? extends org.apache.hadoop.conf.Configuration> flowProcess)` Method getReplication returns the `replication` specified by the underlying file system for this resource.
    `long` | `getSize(org.apache.hadoop.conf.Configuration conf)` Method getSize returns the size of the file referenced by this tap.
    `long` | `getSize(FlowProcess<? extends org.apache.hadoop.conf.Configuration> flowProcess)` Method getSize returns the size of the file referenced by this tap.
    `static org.apache.hadoop.fs.Path` | `getTempPath(org.apache.hadoop.conf.Configuration conf)`
    `java.net.URI` | `getURIScheme(org.apache.hadoop.conf.Configuration jobConf)`
    `protected static boolean` | `getUseCombinedInput(org.apache.hadoop.conf.Configuration conf)`
    `boolean` | `isDirectory(org.apache.hadoop.conf.Configuration conf)` Method isDirectory returns true if the underlying resource represents a directory or folder instead of an individual file.
    `boolean` | `isDirectory(FlowProcess<? extends org.apache.hadoop.conf.Configuration> flowProcess)` Method isDirectory returns true if the underlying resource represents a directory or folder instead of an individual file.
    `protected java.lang.String` | `makeTemporaryPathDirString(java.lang.String name)`
    `protected java.net.URI` | `makeURIScheme(org.apache.hadoop.conf.Configuration configuration)`
    `TupleEntryIterator` | `openForRead(FlowProcess<? extends org.apache.hadoop.conf.Configuration> flowProcess, org.apache.hadoop.mapred.RecordReader input)` Method openForRead opens the resource represented by this Tap instance for reading.
    `TupleEntryCollector` | `openForWrite(FlowProcess<? extends org.apache.hadoop.conf.Configuration> flowProcess, org.apache.hadoop.mapred.OutputCollector output)` Method openForWrite opens the resource represented by this Tap instance for writing.
    `void` | `resetFileStatuses()` Method resetFileStatuses removes the status cache, if any.
    `boolean` | `resourceExists(org.apache.hadoop.conf.Configuration conf)` Method resourceExists returns true if the path represented by this instance exists.
    `protected void` | `setStringPath(java.lang.String stringPath)`
    `protected void` | `setUriScheme(java.net.URI uriScheme)`
    `void` | `sinkConfInit(FlowProcess<? extends org.apache.hadoop.conf.Configuration> process, org.apache.hadoop.conf.Configuration conf)` Method sinkConfInit initializes this instance as a sink.
    `void` | `sourceConfInit(FlowProcess<? extends org.apache.hadoop.conf.Configuration> process, org.apache.hadoop.conf.Configuration conf)` Method sourceConfInit initializes this instance as a source.
    `protected void` | `sourceConfInitAddInputPath(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path qualifiedPath)` Deprecated.
    `protected void` | `sourceConfInitAddInputPaths(org.apache.hadoop.conf.Configuration conf, java.lang.Iterable<org.apache.hadoop.fs.Path> qualifiedPaths)`
    `protected void` | `sourceConfInitComplete(FlowProcess<? extends org.apache.hadoop.conf.Configuration> process, org.apache.hadoop.conf.Configuration conf)`
    `protected static void` | `verifyNoDuplicates(org.apache.hadoop.conf.Configuration conf)`

    - ### Methods inherited from class cascading.tap.Tap

      `commitResource, createResource, deleteResource, equals, flowConfInit, getConfigDef, getFullIdentifier, getModifiedTime, getNodeConfigDef, getScheme, getSinkFields, getSinkMode, getSourceFields, getStepConfigDef, getTrace, hasConfigDef, hashCode, hasNodeConfigDef, hasStepConfigDef, id, isKeep, isReplace, isSink, isSource, isTemporary, isUpdate, openForRead, openForWrite, outgoingScopeFor, prepareResourceForRead, prepareResourceForWrite, presentSinkFields, presentSourceFields, resolveIncomingOperationArgumentFields, resolveIncomingOperationPassThroughFields, resourceExists, retrieveSinkFields, retrieveSourceFields, rollbackResource, setScheme, taps, toString`

    - ### Methods inherited from class java.lang.Object

      `clone, finalize, getClass, notify, notifyAll, wait, wait, wait`

- - ### Field Detail

    - #### stringPath

          protected java.lang.String stringPath

      Field stringPath

  - ### Constructor Detail

    - #### Hfs

          protected Hfs()

    - #### Hfs

          @ConstructorProperties(value="scheme")
          protected Hfs(Scheme<org.apache.hadoop.conf.Configuration,org.apache.hadoop.mapred.RecordReader,org.apache.hadoop.mapred.OutputCollector,?,?> scheme)

    - #### Hfs

          @ConstructorProperties(value={"scheme","stringPath"})
          public Hfs(Scheme<org.apache.hadoop.conf.Configuration,org.apache.hadoop.mapred.RecordReader,org.apache.hadoop.mapred.OutputCollector,?,?> scheme,
                                   java.lang.String stringPath)

      Constructor Hfs creates a new Hfs instance.
      Parameters:
        `scheme` - of type Scheme
        `stringPath` - of type String

    - #### Hfs

          @ConstructorProperties(value={"scheme","stringPath","sinkMode"})
          public Hfs(Scheme<org.apache.hadoop.conf.Configuration,org.apache.hadoop.mapred.RecordReader,org.apache.hadoop.mapred.OutputCollector,?,?> scheme,
                                   java.lang.String stringPath,
                                   SinkMode sinkMode)

      Constructor Hfs creates a new Hfs instance.
      Parameters:
        `scheme` - of type Scheme
        `stringPath` - of type String
        `sinkMode` - of type SinkMode

  - ### Method Detail

    - #### getLocalModeScheme

          protected static java.lang.String getLocalModeScheme(org.apache.hadoop.conf.Configuration conf,
                                            java.lang.String defaultValue)

    - #### getUseCombinedInput

          protected static boolean getUseCombinedInput(org.apache.hadoop.conf.Configuration conf)

    - #### getCombinedInputSafeMode

          protected static boolean getCombinedInputSafeMode(org.apache.hadoop.conf.Configuration conf)

    - #### setStringPath

          protected void setStringPath(java.lang.String stringPath)

    - #### setUriScheme

          protected void setUriScheme(java.net.URI uriScheme)

    - #### getURIScheme

          public java.net.URI getURIScheme(org.apache.hadoop.conf.Configuration jobConf)

    - #### makeURIScheme

          protected java.net.URI makeURIScheme(org.apache.hadoop.conf.Configuration configuration)

    - #### getDefaultFileSystemURIScheme

          public java.net.URI getDefaultFileSystemURIScheme(org.apache.hadoop.conf.Configuration configuration)

      Method getDefaultFileSystemURIScheme returns the URI scheme for the default Hadoop FileSystem.
      Parameters:
        `configuration` - of type JobConf
      Returns:
        URI

    - #### getDefaultFileSystem

          protected org.apache.hadoop.fs.FileSystem getDefaultFileSystem(org.apache.hadoop.conf.Configuration configuration)

    - #### getFileSystem

          protected org.apache.hadoop.fs.FileSystem getFileSystem(org.apache.hadoop.conf.Configuration configuration)

    - #### getIdentifier

          public java.lang.String getIdentifier()

      Description copied from class: `Tap`
      Method getIdentifier returns a String representing the resource this Tap instance represents.

      Often, if the tap accesses a filesystem, the identifier is nothing more than the path to the file or directory. In other cases it may be a an URL or URI representing a connection string or remote resource.

      Any two Tap instances having the same value for the identifier are considered equal.
      Specified by:
        `getIdentifier` in class `Tap<org.apache.hadoop.conf.Configuration,org.apache.hadoop.mapred.RecordReader,org.apache.hadoop.mapred.OutputCollector>`
      Returns:
        String

    - #### getPath

          public org.apache.hadoop.fs.Path getPath()

    - #### getFullIdentifier

          public java.lang.String getFullIdentifier(org.apache.hadoop.conf.Configuration conf)

      Description copied from class: `Tap`
      Method getFullIdentifier returns a fully qualified resource identifier.
      Overrides:
        `getFullIdentifier` in class `Tap<org.apache.hadoop.conf.Configuration,org.apache.hadoop.mapred.RecordReader,org.apache.hadoop.mapred.OutputCollector>`
      Parameters:
        `conf` - of type Config
      Returns:
        String

    - #### sourceConfInit

          public void sourceConfInit(FlowProcess<? extends org.apache.hadoop.conf.Configuration> process,
                            org.apache.hadoop.conf.Configuration conf)

      Description copied from class: `Tap`
      Method sourceConfInit initializes this instance as a source.

      This method maybe called more than once if this Tap instance is used outside the scope of a `Flow` instance or if it participates in multiple times in a given Flow or across different Flows in a `Cascade`.

      In the context of a Flow, it will be called after `FlowListener.onStarting(cascading.flow.Flow)`

      Note that no resources or services should be modified by this method.
      Overrides:
        `sourceConfInit` in class `Tap<org.apache.hadoop.conf.Configuration,org.apache.hadoop.mapred.RecordReader,org.apache.hadoop.mapred.OutputCollector>`
      Parameters:
        `process` - of type FlowProcess
        `conf` - of type Config

    - #### verifyNoDuplicates

          protected static void verifyNoDuplicates(org.apache.hadoop.conf.Configuration conf)

    - #### applySourceConfInitIdentifiers

          protected void applySourceConfInitIdentifiers(FlowProcess<? extends org.apache.hadoop.conf.Configuration> process,
                                            org.apache.hadoop.conf.Configuration conf,
                                            java.lang.String... fullIdentifiers)

    - #### sourceConfInitAddInputPaths

          protected void sourceConfInitAddInputPaths(org.apache.hadoop.conf.Configuration conf,
                                         java.lang.Iterable<org.apache.hadoop.fs.Path> qualifiedPaths)

    - #### sourceConfInitAddInputPath

          @Deprecated
          protected void sourceConfInitAddInputPath(org.apache.hadoop.conf.Configuration conf,
                                                   org.apache.hadoop.fs.Path qualifiedPath)

      Deprecated.

    - #### sourceConfInitComplete

          protected void sourceConfInitComplete(FlowProcess<? extends org.apache.hadoop.conf.Configuration> process,
                                    org.apache.hadoop.conf.Configuration conf)

    - #### sinkConfInit

          public void sinkConfInit(FlowProcess<? extends org.apache.hadoop.conf.Configuration> process,
                          org.apache.hadoop.conf.Configuration conf)

      Description copied from class: `Tap`
      Method sinkConfInit initializes this instance as a sink.

      This method maybe called more than once if this Tap instance is used outside the scope of a `Flow` instance or if it participates in multiple times in a given Flow or across different Flows in a `Cascade`.

      Note this method will be called in context of this Tap being used as a traditional 'sink' and as a 'trap'.

      In the context of a Flow, it will be called after `FlowListener.onStarting(cascading.flow.Flow)`

      Note that no resources or services should be modified by this method. If this Tap instance returns true for `Tap.isReplace()`, then `Tap.deleteResource(Object)` will be called by the parent Flow.
      Overrides:
        `sinkConfInit` in class `Tap<org.apache.hadoop.conf.Configuration,org.apache.hadoop.mapred.RecordReader,org.apache.hadoop.mapred.OutputCollector>`
      Parameters:
        `process` - of type FlowProcess
        `conf` - of type Config

    - #### openForRead

          public TupleEntryIterator openForRead(FlowProcess<? extends org.apache.hadoop.conf.Configuration> flowProcess,
                                       org.apache.hadoop.mapred.RecordReader input)
                                         throws java.io.IOException

      Description copied from class: `Tap`
      Method openForRead opens the resource represented by this Tap instance for reading.

      `input` value may be null, if so, sub-classes must inquire with the underlying `Scheme` via `Scheme.sourceConfInit(cascading.flow.FlowProcess, Tap, Object)` to get the proper input type and instantiate it before calling `super.openForRead()`.

      Note the returned iterator will return the same instance of `TupleEntry` on every call, thus a copy must be made of either the TupleEntry or the underlying `Tuple` instance if they are to be stored in a Collection.
      Specified by:
        `openForRead` in class `Tap<org.apache.hadoop.conf.Configuration,org.apache.hadoop.mapred.RecordReader,org.apache.hadoop.mapred.OutputCollector>`
      Parameters:
        `flowProcess` - of type FlowProcess
        `input` - of type Input
      Returns:
        TupleEntryIterator
      Throws:
        `java.io.IOException` - when the resource cannot be opened

    - #### openForWrite

          public TupleEntryCollector openForWrite(FlowProcess<? extends org.apache.hadoop.conf.Configuration> flowProcess,
                                         org.apache.hadoop.mapred.OutputCollector output)
                                           throws java.io.IOException

      Description copied from class: `Tap`
      Method openForWrite opens the resource represented by this Tap instance for writing.

      This method is used internally and does not honor the `SinkMode` setting. If SinkMode is `SinkMode.REPLACE`, this call may fail. See `Tap.openForWrite(cascading.flow.FlowProcess)`.

      `output` value may be null, if so, sub-classes must inquire with the underlying `Scheme` via `Scheme.sinkConfInit(cascading.flow.FlowProcess, Tap, Object)` to get the proper output type and instantiate it before calling `super.openForWrite()`.
      Specified by:
        `openForWrite` in class `Tap<org.apache.hadoop.conf.Configuration,org.apache.hadoop.mapred.RecordReader,org.apache.hadoop.mapred.OutputCollector>`
      Parameters:
        `flowProcess` - of type FlowProcess
        `output` - of type Output
      Returns:
        TupleEntryCollector
      Throws:
        `java.io.IOException` - when the resource cannot be opened

    - #### createResource

          public boolean createResource(org.apache.hadoop.conf.Configuration conf)
                                 throws java.io.IOException

      Description copied from class: `Tap`
      Method createResource creates the underlying resource.
      Specified by:
        `createResource` in class `Tap<org.apache.hadoop.conf.Configuration,org.apache.hadoop.mapred.RecordReader,org.apache.hadoop.mapred.OutputCollector>`
      Parameters:
        `conf` - of type Config
      Returns:
        boolean
      Throws:
        `java.io.IOException` - when there is an error making directories

    - #### deleteResource

          public boolean deleteResource(org.apache.hadoop.conf.Configuration conf)
                                 throws java.io.IOException

      Description copied from class: `Tap`
      Method deleteResource deletes the resource represented by this instance.
      Specified by:
        `deleteResource` in class `Tap<org.apache.hadoop.conf.Configuration,org.apache.hadoop.mapred.RecordReader,org.apache.hadoop.mapred.OutputCollector>`
      Parameters:
        `conf` - of type Config
      Returns:
        boolean
      Throws:
        `java.io.IOException` - when the resource cannot be deleted

    - #### deleteChildResource

          public boolean deleteChildResource(FlowProcess<? extends org.apache.hadoop.conf.Configuration> flowProcess,
                                    java.lang.String childIdentifier)
                                      throws java.io.IOException

      Throws:
        `java.io.IOException`

    - #### deleteChildResource

          public boolean deleteChildResource(org.apache.hadoop.conf.Configuration conf,
                                    java.lang.String childIdentifier)
                                      throws java.io.IOException

      Throws:
        `java.io.IOException`

    - #### resourceExists

          public boolean resourceExists(org.apache.hadoop.conf.Configuration conf)
                                 throws java.io.IOException

      Description copied from class: `Tap`
      Method resourceExists returns true if the path represented by this instance exists.
      Specified by:
        `resourceExists` in class `Tap<org.apache.hadoop.conf.Configuration,org.apache.hadoop.mapred.RecordReader,org.apache.hadoop.mapred.OutputCollector>`
      Parameters:
        `conf` - of type Config
      Returns:
        true if the underlying resource already exists
      Throws:
        `java.io.IOException` - when the status cannot be determined

    - #### isDirectory

          public boolean isDirectory(FlowProcess<? extends org.apache.hadoop.conf.Configuration> flowProcess)
                              throws java.io.IOException

      Description copied from interface: `FileType`
      Method isDirectory returns true if the underlying resource represents a directory or folder instead of an individual file.
      Specified by:
        `isDirectory` in interface `FileType<org.apache.hadoop.conf.Configuration>`
      Returns:
        boolean
      Throws:
        `java.io.IOException`

    - #### isDirectory

          public boolean isDirectory(org.apache.hadoop.conf.Configuration conf)
                              throws java.io.IOException

      Description copied from interface: `FileType`
      Method isDirectory returns true if the underlying resource represents a directory or folder instead of an individual file.
      Specified by:
        `isDirectory` in interface `FileType<org.apache.hadoop.conf.Configuration>`
      Parameters:
        `conf` - of JobConf
      Returns:
        boolean
      Throws:
        `java.io.IOException`

    - #### getSize

          public long getSize(FlowProcess<? extends org.apache.hadoop.conf.Configuration> flowProcess)
                       throws java.io.IOException

      Description copied from interface: `FileType`
      Method getSize returns the size of the file referenced by this tap.
      Specified by:
        `getSize` in interface `FileType<org.apache.hadoop.conf.Configuration>`
      Returns:
        The size of the file reference by this tap.
      Throws:
        `java.io.IOException`

    - #### getSize

          public long getSize(org.apache.hadoop.conf.Configuration conf)
                       throws java.io.IOException

      Description copied from interface: `FileType`
      Method getSize returns the size of the file referenced by this tap.
      Specified by:
        `getSize` in interface `FileType<org.apache.hadoop.conf.Configuration>`
      Parameters:
        `conf` - of type Config
      Returns:
        The size of the file reference by this tap.
      Throws:
        `java.io.IOException`

    - #### getBlockSize

          public long getBlockSize(FlowProcess<? extends org.apache.hadoop.conf.Configuration> flowProcess)
                            throws java.io.IOException

      Method getBlockSize returns the `blocksize` specified by the underlying file system for this resource.
      Parameters:
        `flowProcess` -
      Returns:
        long
      Throws:
        `java.io.IOException` - when

    - #### getBlockSize

          public long getBlockSize(org.apache.hadoop.conf.Configuration conf)
                            throws java.io.IOException

      Method getBlockSize returns the `blocksize` specified by the underlying file system for this resource.
      Parameters:
        `conf` - of JobConf
      Returns:
        long
      Throws:
        `java.io.IOException` - when

    - #### getReplication

          public int getReplication(FlowProcess<? extends org.apache.hadoop.conf.Configuration> flowProcess)
                             throws java.io.IOException

      Method getReplication returns the `replication` specified by the underlying file system for this resource.
      Parameters:
        `flowProcess` -
      Returns:
        int
      Throws:
        `java.io.IOException` - when

    - #### getReplication

          public int getReplication(org.apache.hadoop.conf.Configuration conf)
                             throws java.io.IOException

      Method getReplication returns the `replication` specified by the underlying file system for this resource.
      Parameters:
        `conf` - of JobConf
      Returns:
        int
      Throws:
        `java.io.IOException` - when

    - #### getChildIdentifiers

          public java.lang.String[] getChildIdentifiers(FlowProcess<? extends org.apache.hadoop.conf.Configuration> flowProcess)
                                                 throws java.io.IOException

      Description copied from interface: `FileType`
      Method getChildIdentifiers returns an array of child identifiers if this resource is a directory.

      This method will skip Hadoop log directories (`_log`).
      Specified by:
        `getChildIdentifiers` in interface `FileType<org.apache.hadoop.conf.Configuration>`
      Returns:
        String[]
      Throws:
        `java.io.IOException`

    - #### getChildIdentifiers

          public java.lang.String[] getChildIdentifiers(org.apache.hadoop.conf.Configuration conf)
                                                 throws java.io.IOException

      Description copied from interface: `FileType`
      Method getChildIdentifiers returns an array of child identifiers if this resource is a directory.

      This method will skip Hadoop log directories (`_log`).
      Specified by:
        `getChildIdentifiers` in interface `FileType<org.apache.hadoop.conf.Configuration>`
      Parameters:
        `conf` - of JobConf
      Returns:
        String[]
      Throws:
        `java.io.IOException`

    - #### getChildIdentifiers

          public java.lang.String[] getChildIdentifiers(FlowProcess<? extends org.apache.hadoop.conf.Configuration> flowProcess,
                                               int depth,
                                               boolean fullyQualified)
                                                 throws java.io.IOException

      Specified by:
        `getChildIdentifiers` in interface `FileType<org.apache.hadoop.conf.Configuration>`
      Throws:
        `java.io.IOException`

    - #### getChildIdentifiers

          public java.lang.String[] getChildIdentifiers(org.apache.hadoop.conf.Configuration conf,
                                               int depth,
                                               boolean fullyQualified)
                                                 throws java.io.IOException

      Specified by:
        `getChildIdentifiers` in interface `FileType<org.apache.hadoop.conf.Configuration>`
      Throws:
        `java.io.IOException`

    - #### getModifiedTime

          public long getModifiedTime(org.apache.hadoop.conf.Configuration conf)
                               throws java.io.IOException

      Description copied from class: `Tap`
      Method getModifiedTime returns the date this resource was last modified.
      Specified by:
        `getModifiedTime` in class `Tap<org.apache.hadoop.conf.Configuration,org.apache.hadoop.mapred.RecordReader,org.apache.hadoop.mapred.OutputCollector>`
      Parameters:
        `conf` - of type Config
      Returns:
        The date this resource was last modified.
      Throws:
        `java.io.IOException`

    - #### getFileStatus

          public org.apache.hadoop.fs.FileStatus getFileStatus(org.apache.hadoop.conf.Configuration conf)
                                                        throws java.io.IOException

      Throws:
        `java.io.IOException`

    - #### getTempPath

          public static org.apache.hadoop.fs.Path getTempPath(org.apache.hadoop.conf.Configuration conf)

    - #### makeTemporaryPathDirString

          protected java.lang.String makeTemporaryPathDirString(java.lang.String name)

    - #### resetFileStatuses

          public void resetFileStatuses()

      Method resetFileStatuses removes the status cache, if any.

Copyright © 2007-2015 Xplenty, Inc. All Rights Reserved.