public abstract class BaseDistCacheTap extends DecoratorTap<java.lang.Void,org.apache.hadoop.conf.Configuration,org.apache.hadoop.mapred.RecordReader,org.apache.hadoop.mapred.OutputCollector>
metaInfo, original
Constructor and Description |
---|
BaseDistCacheTap(Tap<org.apache.hadoop.conf.Configuration,org.apache.hadoop.mapred.RecordReader,org.apache.hadoop.mapred.OutputCollector> original) |
Modifier and Type | Method and Description |
---|---|
protected abstract void |
addLocalCacheFiles(org.apache.hadoop.conf.Configuration conf,
java.net.URI uri) |
protected abstract org.apache.hadoop.fs.Path[] |
getLocalCacheFiles(FlowProcess<? extends org.apache.hadoop.conf.Configuration> flowProcess) |
TupleEntryIterator |
openForRead(FlowProcess<? extends org.apache.hadoop.conf.Configuration> flowProcess,
org.apache.hadoop.mapred.RecordReader input)
Method openForRead opens the resource represented by this Tap instance for reading.
|
void |
sourceConfInit(FlowProcess<? extends org.apache.hadoop.conf.Configuration> process,
org.apache.hadoop.conf.Configuration conf)
Method sourceConfInit initializes this instance as a source.
|
commitResource, createResource, createResource, deleteResource, deleteResource, flowConfInit, getConfigDef, getFullIdentifier, getFullIdentifier, getIdentifier, getMetaInfo, getModifiedTime, getModifiedTime, getNodeConfigDef, getOriginal, getScheme, getSinkFields, getSinkMode, getSourceFields, getStepConfigDef, getTrace, hasConfigDef, hasNodeConfigDef, hasStepConfigDef, isKeep, isReplace, isSink, isSource, isTemporary, isUpdate, openForRead, openForWrite, openForWrite, outgoingScopeFor, prepareResourceForRead, prepareResourceForWrite, presentSinkFields, presentSourceFields, resolveIncomingOperationArgumentFields, resolveIncomingOperationPassThroughFields, resourceExists, resourceExists, retrieveSinkFields, retrieveSourceFields, rollbackResource, setMetaInfo, setOriginal, sinkConfInit, toString
entryStream, entryStream, entryStreamCopy, entryStreamCopy, equals, hashCode, id, openForReadUnchecked, setScheme, spliterator, splititerator, taps, tupleStream, tupleStream, tupleStreamCopy, tupleStreamCopy
public BaseDistCacheTap(Tap<org.apache.hadoop.conf.Configuration,org.apache.hadoop.mapred.RecordReader,org.apache.hadoop.mapred.OutputCollector> original)
public void sourceConfInit(FlowProcess<? extends org.apache.hadoop.conf.Configuration> process, org.apache.hadoop.conf.Configuration conf)
Tap
This method maybe called more than once if this Tap instance is used outside the scope of a Flow
instance or if it participates in multiple times in a given Flow or across different Flows in
a Cascade
.
In the context of a Flow, it will be called after
FlowListener.onStarting(cascading.flow.Flow)
Note that no resources or services should be modified by this method.
sourceConfInit
in class DecoratorTap<java.lang.Void,org.apache.hadoop.conf.Configuration,org.apache.hadoop.mapred.RecordReader,org.apache.hadoop.mapred.OutputCollector>
process
- of type FlowProcessconf
- of type Configpublic TupleEntryIterator openForRead(FlowProcess<? extends org.apache.hadoop.conf.Configuration> flowProcess, org.apache.hadoop.mapred.RecordReader input) throws java.io.IOException
Tap
input
value may be null, if so, sub-classes must inquire with the underlying Scheme
via Scheme.sourceConfInit(cascading.flow.FlowProcess, Tap, Object)
to get the proper
input type and instantiate it before calling super.openForRead()
.
Note the returned iterator will return the same instance of TupleEntry
on every call,
thus a copy must be made of either the TupleEntry or the underlying Tuple
instance if they are to be
stored in a Collection.
openForRead
in class DecoratorTap<java.lang.Void,org.apache.hadoop.conf.Configuration,org.apache.hadoop.mapred.RecordReader,org.apache.hadoop.mapred.OutputCollector>
flowProcess
- of type FlowProcessinput
- of type Inputjava.io.IOException
- when the resource cannot be openedprotected abstract org.apache.hadoop.fs.Path[] getLocalCacheFiles(FlowProcess<? extends org.apache.hadoop.conf.Configuration> flowProcess) throws java.io.IOException
java.io.IOException
protected abstract void addLocalCacheFiles(org.apache.hadoop.conf.Configuration conf, java.net.URI uri)
Copyright © 2007-2017 Cascading Maintainers. All Rights Reserved.