cascading.flow.planner
Class FlowPlanner

java.lang.Object
  extended by cascading.flow.planner.FlowPlanner
Direct Known Subclasses:
HadoopPlanner, LocalPlanner

public abstract class FlowPlanner
extends java.lang.Object

Class FlowPlanner is the base class for all planner implementations.


Field Summary
protected  AssertionLevel assertionLevel
          Field assertionLevel
protected  DebugLevel debugLevel
          Field debugLevel
protected  java.util.Map<java.lang.Object,java.lang.Object> properties
          Field properties
 
Constructor Summary
FlowPlanner()
           
 
Method Summary
abstract  Flow buildFlow(FlowDef flowDef)
          Method buildFlow renders the actual Flow instance.
protected  ElementGraph createElementGraph(FlowDef flowDef)
           
protected  void failOnGroupEverySplit(ElementGraph elementGraph)
           
protected  void failOnLoneGroupAssertion(ElementGraph elementGraph)
          Verifies that there are not only GroupAssertions following any given Group instance.
protected  void failOnMissingGroup(ElementGraph elementGraph)
           
protected  void failOnMisusedBuffer(ElementGraph elementGraph)
           
protected  PlannerException handleExceptionDuringPlanning(java.lang.Exception exception, ElementGraph elementGraph)
           
protected  void handleJobPartitioning(ElementGraph elementGraph)
          Inserts a temporary Tap between logical MR jobs.
protected  void handleJoins(ElementGraph elementGraph)
          Prevent leftmost sources from sourcing a downstream join on the rightmost side intra-task by inserting a temp tap between the left-sourced join and right-sourced join.
protected  void handleNonSafeOperations(ElementGraph elementGraph)
           
 void initialize(FlowConnector flowConnector, java.util.Map<java.lang.Object,java.lang.Object> properties)
           
protected  void insertTempTapAfter(ElementGraph graph, Pipe pipe)
          Method insertTapAfter ...
protected abstract  Tap makeTempTap(java.lang.String name)
          Method makeTemp ...
protected  void verifyAssembly(FlowDef flowDef)
          Must be called to determine if all elements of the base pipe assembly are available
protected  void verifyPipeAssemblyEndPoints(FlowDef flowDef)
          Method verifyEndPoints verifies

there aren't dupe names in heads or tails.

protected  void verifySourceNotSinks(java.util.Map<java.lang.String,Tap> sources, java.util.Map<java.lang.String,Tap> sinks)
           
protected  void verifyTaps(java.util.Map<java.lang.String,Tap> taps, boolean areSources, boolean mayNotBeEmpty)
          Method verifyTaps ...
protected  void verifyTraps(FlowDef flowDef)
           
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

properties

protected java.util.Map<java.lang.Object,java.lang.Object> properties
Field properties


assertionLevel

protected AssertionLevel assertionLevel
Field assertionLevel


debugLevel

protected DebugLevel debugLevel
Field debugLevel

Constructor Detail

FlowPlanner

public FlowPlanner()
Method Detail

initialize

public void initialize(FlowConnector flowConnector,
                       java.util.Map<java.lang.Object,java.lang.Object> properties)

buildFlow

public abstract Flow buildFlow(FlowDef flowDef)
Method buildFlow renders the actual Flow instance.

Parameters:
flowDef@return - Flow

verifyAssembly

protected void verifyAssembly(FlowDef flowDef)
Must be called to determine if all elements of the base pipe assembly are available


createElementGraph

protected ElementGraph createElementGraph(FlowDef flowDef)

verifySourceNotSinks

protected void verifySourceNotSinks(java.util.Map<java.lang.String,Tap> sources,
                                    java.util.Map<java.lang.String,Tap> sinks)

verifyTaps

protected void verifyTaps(java.util.Map<java.lang.String,Tap> taps,
                          boolean areSources,
                          boolean mayNotBeEmpty)
Method verifyTaps ...

Parameters:
taps - of type Map
areSources - of type boolean
mayNotBeEmpty - of type boolean

verifyPipeAssemblyEndPoints

protected void verifyPipeAssemblyEndPoints(FlowDef flowDef)
Method verifyEndPoints verifies

there aren't dupe names in heads or tails. all the sink and source tap names match up with tail and head pipes


verifyTraps

protected void verifyTraps(FlowDef flowDef)

failOnLoneGroupAssertion

protected void failOnLoneGroupAssertion(ElementGraph elementGraph)
Verifies that there are not only GroupAssertions following any given Group instance. This will adversely affect the stream entering any subsequent Tap of Each instances.


failOnMissingGroup

protected void failOnMissingGroup(ElementGraph elementGraph)

failOnMisusedBuffer

protected void failOnMisusedBuffer(ElementGraph elementGraph)

failOnGroupEverySplit

protected void failOnGroupEverySplit(ElementGraph elementGraph)

handleExceptionDuringPlanning

protected PlannerException handleExceptionDuringPlanning(java.lang.Exception exception,
                                                         ElementGraph elementGraph)

handleNonSafeOperations

protected void handleNonSafeOperations(ElementGraph elementGraph)

insertTempTapAfter

protected void insertTempTapAfter(ElementGraph graph,
                                  Pipe pipe)
Method insertTapAfter ...

Parameters:
graph - of type PipeGraph
pipe - of type Pipe

makeTempTap

protected abstract Tap makeTempTap(java.lang.String name)
Method makeTemp ...

Parameters:
name -
Returns:
Tap

handleJobPartitioning

protected void handleJobPartitioning(ElementGraph elementGraph)
Inserts a temporary Tap between logical MR jobs.

Since all joins are at groups or splices, depth first search is safe

todo: refactor so that rules are applied to path segments bounded by taps todo: this would allow balancing of operations within paths instead of pushing todo: all operations up. may allow for consolidation of rules

Parameters:
elementGraph - of type PipeGraph

handleJoins

protected void handleJoins(ElementGraph elementGraph)
Prevent leftmost sources from sourcing a downstream join on the rightmost side intra-task by inserting a temp tap between the left-sourced join and right-sourced join.

Parameters:
elementGraph -


Copyright © 2007-2011 Concurrent, Inc. All Rights Reserved.