cascading.tap.hadoop

## Class Hfs

- java.lang.Object
- - cascading.tap.Tap<Configuration,RecordReader,OutputCollector>
  - - cascading.tap.hadoop.Hfs

All Implemented Interfaces:
  FlowElement, ScopedElement, FileType<Configuration>, Traceable, java.io.Serializable

Direct Known Subclasses:
  Dfs, Lfs, TempHfs

    public class Hfs
    extends Tap<Configuration,RecordReader,OutputCollector>
    implements FileType<Configuration>

Class Hfs is the base class for all Hadoop file system access. Hfs may only be used with the Hadoop `FlowConnector` sub-classes when creating Hadoop executable `Flow` instances.

Paths typically should point to a directory, where in turn all the "part" files immediately in that directory will be included. This is the practice Hadoop expects. Sub-directories are not included and typically result in a failure.

To include sub-directories, Hadoop supports "globing". Globing is a frustrating feature and is supported more robustly by `GlobHfs` and less so by Hfs.

Hfs will accept `/*` (wildcard) paths, but not all convenience methods like `jobConf.getSize` will behave properly or reliably. Nor can the Hfs instance with a wildcard path be used as a sink to write data.

In those cases use GlobHfs since it is a sub-class of `MultiSourceTap`.

Optionally use `Dfs` or `Lfs` for resources specific to Hadoop Distributed file system or the Local file system, respectively. Using Hfs is the best practice when possible, Lfs and Dfs are conveniences.

Use the Hfs class if the 'kind' of resource is unknown at design time. To use, prefix a scheme to the 'stringPath'. Where `hdfs://...` will denote Dfs, and `file://...` will denote Lfs.

Call `HfsProps.setTemporaryDirectory(java.util.Map, String)` to use a different temporary file directory path other than the current Hadoop default path.

By default Cascading on Hadoop will assume any source or sink Tap using the `file://` URI scheme intends to read files from the local client filesystem (for example when using the `Lfs` Tap) where the Hadoop job jar is started. Subsequently Cascading will force any MapReduce jobs reading or writing to `file://` resources to run in Hadoop "standalone mode" so that the file can be read.

To change this behavior, `HfsProps.setLocalModeScheme(java.util.Map, String)` to set a different scheme value, or to "none" to disable entirely for the case the file to be read is available on every Hadoop processing node in the exact same path.

When using a MapReduce planner, Hfs can optionally combine multiple small files (or a series of small "blocks") into larger "splits". This reduces the number of resulting map tasks created by Hadoop and can improve application performance.

This is enabled by calling `HfsProps.setUseCombinedInput(boolean)` to `true`. By default, merging or combining splits into large ones is disabled.

Apache Tez planner does not require this setting, it is supported by default and enabled by the application manager.

See Also:
  Serialized Form

### Field Summary

Fields
`protected java.lang.String` `stringPath` Field stringPath

### Constructor Summary

Constructors
`protected` `Hfs()`
`protected` `Hfs(Scheme<Configuration,RecordReader,OutputCollector,?,?> scheme)`
`` `Hfs(Scheme<Configuration,RecordReader,OutputCollector,?,?> scheme, java.lang.String stringPath)` Constructor Hfs creates a new Hfs instance.
`` `Hfs(Scheme<Configuration,RecordReader,OutputCollector,?,?> scheme, java.lang.String stringPath, SinkMode sinkMode)` Constructor Hfs creates a new Hfs instance.

### Method Summary

Methods
`protected void` `applySourceConfInitIdentifiers(FlowProcess<? extends Configuration> process, Configuration conf, java.lang.String... fullIdentifiers)`
`boolean` `createResource(Configuration conf)`
`boolean` `deleteChildResource(Configuration conf, java.lang.String childIdentifier)`
`boolean` `deleteChildResource(FlowProcess<? extends Configuration> flowProcess, java.lang.String childIdentifier)`
`boolean` `deleteResource(Configuration conf)`
`long` `getBlockSize(Configuration conf)` Method getBlockSize returns the `blocksize` specified by the underlying file system for this resource.
`long` `getBlockSize(FlowProcess<? extends Configuration> flowProcess)` Method getBlockSize returns the `blocksize` specified by the underlying file system for this resource.
`java.lang.String[]` `getChildIdentifiers(Configuration conf)`
`java.lang.String[]` `getChildIdentifiers(Configuration conf, int depth, boolean fullyQualified)`
`java.lang.String[]` `getChildIdentifiers(FlowProcess<? extends Configuration> flowProcess)`
`java.lang.String[]` `getChildIdentifiers(FlowProcess<? extends Configuration> flowProcess, int depth, boolean fullyQualified)`
`protected static boolean` `getCombinedInputSafeMode(Configuration conf)`
`protected FileSystem` `getDefaultFileSystem(Configuration configuration)`
`java.net.URI` `getDefaultFileSystemURIScheme(Configuration configuration)` Method getDefaultFileSystemURIScheme returns the URI scheme for the default Hadoop FileSystem.
`FileStatus` `getFileStatus(Configuration conf)`
`protected FileSystem` `getFileSystem(Configuration configuration)`
`java.lang.String` `getFullIdentifier(Configuration conf)`
`java.lang.String` `getIdentifier()`
`protected static java.lang.String` `getLocalModeScheme(Configuration conf, java.lang.String defaultValue)`
`long` `getModifiedTime(Configuration conf)`
`Path` `getPath()`
`int` `getReplication(Configuration conf)` Method getReplication returns the `replication` specified by the underlying file system for this resource.
`int` `getReplication(FlowProcess<? extends Configuration> flowProcess)` Method getReplication returns the `replication` specified by the underlying file system for this resource.
`long` `getSize(Configuration conf)`
`long` `getSize(FlowProcess<? extends Configuration> flowProcess)`
`static Path` `getTempPath(Configuration conf)`
`java.net.URI` `getURIScheme(Configuration jobConf)`
`protected static boolean` `getUseCombinedInput(Configuration conf)`
`boolean` `isDirectory(Configuration conf)`
`boolean` `isDirectory(FlowProcess<? extends Configuration> flowProcess)`
`protected java.lang.String` `makeTemporaryPathDirString(java.lang.String name)`
`protected java.net.URI` `makeURIScheme(Configuration configuration)`
`TupleEntryIterator` `openForRead(FlowProcess<? extends Configuration> flowProcess, RecordReader input)`
`TupleEntryCollector` `openForWrite(FlowProcess<? extends Configuration> flowProcess, OutputCollector output)`
`void` `resetFileStatuses()` Method resetFileStatuses removes the status cache, if any.
`boolean` `resourceExists(Configuration conf)`
`protected void` `setStringPath(java.lang.String stringPath)`
`protected void` `setUriScheme(java.net.URI uriScheme)`
`void` `sinkConfInit(FlowProcess<? extends Configuration> process, Configuration conf)`
`void` `sourceConfInit(FlowProcess<? extends Configuration> process, Configuration conf)`
`protected void` `sourceConfInitAddInputPath(Configuration conf, Path qualifiedPath)` Deprecated.
`protected void` `sourceConfInitAddInputPaths(Configuration conf, java.lang.Iterable<Path> qualifiedPaths)`
`protected void` `sourceConfInitComplete(FlowProcess<? extends Configuration> process, Configuration conf)`
`protected static void` `verifyNoDuplicates(Configuration conf)`

### Methods inherited from class cascading.tap.Tap

`commitResource, createResource, deleteResource, equals, flowConfInit, getConfigDef, getFullIdentifier, getModifiedTime, getNodeConfigDef, getScheme, getSinkFields, getSinkMode, getSourceFields, getStepConfigDef, getTrace, hasConfigDef, hashCode, hasNodeConfigDef, hasStepConfigDef, id, isKeep, isReplace, isSink, isSource, isTemporary, isUpdate, openForRead, openForWrite, outgoingScopeFor, prepareResourceForRead, prepareResourceForWrite, presentSinkFields, presentSourceFields, resolveIncomingOperationArgumentFields, resolveIncomingOperationPassThroughFields, resourceExists, retrieveSinkFields, retrieveSourceFields, rollbackResource, setScheme, taps, toString`

### Methods inherited from class java.lang.Object

`clone, finalize, getClass, notify, notifyAll, wait, wait, wait`

### Field Detail

#### stringPath

    protected java.lang.String stringPath

Field stringPath

### Constructor Detail

#### Hfs

    protected Hfs()

#### Hfs

    @ConstructorProperties(value="scheme")
    protected Hfs(Scheme<Configuration,RecordReader,OutputCollector,?,?> scheme)

#### Hfs

    @ConstructorProperties(value={"scheme","stringPath"})
    public Hfs(Scheme<Configuration,RecordReader,OutputCollector,?,?> scheme,
                             java.lang.String stringPath)

Constructor Hfs creates a new Hfs instance.

Parameters:
  `scheme` - of type Scheme
  `stringPath` - of type String

#### Hfs

    @ConstructorProperties(value={"scheme","stringPath","sinkMode"})
    public Hfs(Scheme<Configuration,RecordReader,OutputCollector,?,?> scheme,
                             java.lang.String stringPath,
                             SinkMode sinkMode)

Constructor Hfs creates a new Hfs instance.

Parameters:
  `scheme` - of type Scheme
  `stringPath` - of type String
  `sinkMode` - of type SinkMode

### Method Detail

#### getLocalModeScheme

    protected static java.lang.String getLocalModeScheme(Configuration conf,
                                      java.lang.String defaultValue)

#### getUseCombinedInput

    protected static boolean getUseCombinedInput(Configuration conf)

#### getCombinedInputSafeMode

    protected static boolean getCombinedInputSafeMode(Configuration conf)

#### setStringPath

    protected void setStringPath(java.lang.String stringPath)

#### setUriScheme

    protected void setUriScheme(java.net.URI uriScheme)

#### getURIScheme

    public java.net.URI getURIScheme(Configuration jobConf)

#### makeURIScheme

    protected java.net.URI makeURIScheme(Configuration configuration)

#### getDefaultFileSystemURIScheme

    public java.net.URI getDefaultFileSystemURIScheme(Configuration configuration)

Method getDefaultFileSystemURIScheme returns the URI scheme for the default Hadoop FileSystem.

Parameters:
  `configuration` - of type JobConf
Returns:
  URI

#### getDefaultFileSystem

    protected FileSystem getDefaultFileSystem(Configuration configuration)

#### getFileSystem

    protected FileSystem getFileSystem(Configuration configuration)

#### getIdentifier

    public java.lang.String getIdentifier()

Specified by:
  `getIdentifier` in class `Tap<Configuration,RecordReader,OutputCollector>`

#### getPath

    public Path getPath()

#### getFullIdentifier

    public java.lang.String getFullIdentifier(Configuration conf)

Overrides:
  `getFullIdentifier` in class `Tap<Configuration,RecordReader,OutputCollector>`

#### sourceConfInit

    public void sourceConfInit(FlowProcess<? extends Configuration> process,
                      Configuration conf)

Overrides:
  `sourceConfInit` in class `Tap<Configuration,RecordReader,OutputCollector>`

#### verifyNoDuplicates

    protected static void verifyNoDuplicates(Configuration conf)

#### applySourceConfInitIdentifiers

    protected void applySourceConfInitIdentifiers(FlowProcess<? extends Configuration> process,
                                      Configuration conf,
                                      java.lang.String... fullIdentifiers)

#### sourceConfInitAddInputPaths

    protected void sourceConfInitAddInputPaths(Configuration conf,
                                   java.lang.Iterable<Path> qualifiedPaths)

#### sourceConfInitAddInputPath

    @Deprecated
    protected void sourceConfInitAddInputPath(Configuration conf,
                                             Path qualifiedPath)

Deprecated.

#### sourceConfInitComplete

    protected void sourceConfInitComplete(FlowProcess<? extends Configuration> process,
                              Configuration conf)

#### sinkConfInit

    public void sinkConfInit(FlowProcess<? extends Configuration> process,
                    Configuration conf)

Overrides:
  `sinkConfInit` in class `Tap<Configuration,RecordReader,OutputCollector>`

#### openForRead

    public TupleEntryIterator openForRead(FlowProcess<? extends Configuration> flowProcess,
                                 RecordReader input)
                                   throws java.io.IOException

Specified by:
  `openForRead` in class `Tap<Configuration,RecordReader,OutputCollector>`
Throws:
  `java.io.IOException`

#### openForWrite

    public TupleEntryCollector openForWrite(FlowProcess<? extends Configuration> flowProcess,
                                   OutputCollector output)
                                     throws java.io.IOException

Specified by:
  `openForWrite` in class `Tap<Configuration,RecordReader,OutputCollector>`
Throws:
  `java.io.IOException`

#### createResource

    public boolean createResource(Configuration conf)
                           throws java.io.IOException

Specified by:
  `createResource` in class `Tap<Configuration,RecordReader,OutputCollector>`
Throws:
  `java.io.IOException`

#### deleteResource

    public boolean deleteResource(Configuration conf)
                           throws java.io.IOException

Specified by:
  `deleteResource` in class `Tap<Configuration,RecordReader,OutputCollector>`
Throws:
  `java.io.IOException`

#### deleteChildResource

    public boolean deleteChildResource(FlowProcess<? extends Configuration> flowProcess,
                              java.lang.String childIdentifier)
                                throws java.io.IOException

Throws:
  `java.io.IOException`

#### deleteChildResource

    public boolean deleteChildResource(Configuration conf,
                              java.lang.String childIdentifier)
                                throws java.io.IOException

Throws:
  `java.io.IOException`

#### resourceExists

    public boolean resourceExists(Configuration conf)
                           throws java.io.IOException

Specified by:
  `resourceExists` in class `Tap<Configuration,RecordReader,OutputCollector>`
Throws:
  `java.io.IOException`

#### isDirectory

    public boolean isDirectory(FlowProcess<? extends Configuration> flowProcess)
                        throws java.io.IOException

Specified by:
  `isDirectory` in interface `FileType<Configuration>`
Throws:
  `java.io.IOException`

#### isDirectory

    public boolean isDirectory(Configuration conf)
                        throws java.io.IOException

Specified by:
  `isDirectory` in interface `FileType<Configuration>`
Throws:
  `java.io.IOException`

#### getSize

    public long getSize(FlowProcess<? extends Configuration> flowProcess)
                 throws java.io.IOException

Specified by:
  `getSize` in interface `FileType<Configuration>`
Throws:
  `java.io.IOException`

#### getSize

    public long getSize(Configuration conf)
                 throws java.io.IOException

Specified by:
  `getSize` in interface `FileType<Configuration>`
Throws:
  `java.io.IOException`

#### getBlockSize

    public long getBlockSize(FlowProcess<? extends Configuration> flowProcess)
                      throws java.io.IOException

Method getBlockSize returns the `blocksize` specified by the underlying file system for this resource.

Parameters:
  `flowProcess` -
Returns:
  long
Throws:
  `java.io.IOException` - when

#### getBlockSize

    public long getBlockSize(Configuration conf)
                      throws java.io.IOException

Method getBlockSize returns the `blocksize` specified by the underlying file system for this resource.

Parameters:
  `conf` - of JobConf
Returns:
  long
Throws:
  `java.io.IOException` - when

#### getReplication

    public int getReplication(FlowProcess<? extends Configuration> flowProcess)
                       throws java.io.IOException

Method getReplication returns the `replication` specified by the underlying file system for this resource.

Parameters:
  `flowProcess` -
Returns:
  int
Throws:
  `java.io.IOException` - when

#### getReplication

    public int getReplication(Configuration conf)
                       throws java.io.IOException

Method getReplication returns the `replication` specified by the underlying file system for this resource.

Parameters:
  `conf` - of JobConf
Returns:
  int
Throws:
  `java.io.IOException` - when

#### getChildIdentifiers

    public java.lang.String[] getChildIdentifiers(FlowProcess<? extends Configuration> flowProcess)
                                           throws java.io.IOException

Specified by:
  `getChildIdentifiers` in interface `FileType<Configuration>`
Throws:
  `java.io.IOException`

#### getChildIdentifiers

    public java.lang.String[] getChildIdentifiers(Configuration conf)
                                           throws java.io.IOException

Specified by:
  `getChildIdentifiers` in interface `FileType<Configuration>`
Throws:
  `java.io.IOException`

#### getChildIdentifiers

    public java.lang.String[] getChildIdentifiers(FlowProcess<? extends Configuration> flowProcess,
                                         int depth,
                                         boolean fullyQualified)
                                           throws java.io.IOException

Specified by:
  `getChildIdentifiers` in interface `FileType<Configuration>`
Throws:
  `java.io.IOException`

#### getChildIdentifiers

    public java.lang.String[] getChildIdentifiers(Configuration conf,
                                         int depth,
                                         boolean fullyQualified)
                                           throws java.io.IOException

Specified by:
  `getChildIdentifiers` in interface `FileType<Configuration>`
Throws:
  `java.io.IOException`

#### getModifiedTime

    public long getModifiedTime(Configuration conf)
                         throws java.io.IOException

Specified by:
  `getModifiedTime` in class `Tap<Configuration,RecordReader,OutputCollector>`
Throws:
  `java.io.IOException`

#### getFileStatus

    public FileStatus getFileStatus(Configuration conf)
                             throws java.io.IOException

Throws:
  `java.io.IOException`

#### getTempPath

    public static Path getTempPath(Configuration conf)

#### makeTemporaryPathDirString

    protected java.lang.String makeTemporaryPathDirString(java.lang.String name)

#### resetFileStatuses

    public void resetFileStatuses()

Method resetFileStatuses removes the status cache, if any.