cascading

## Class JoinFieldedPipesPlatformTest

- java.lang.Object
- - junit.framework.Assert
  - - junit.framework.TestCase
    - - cascading.CascadingTestCase
      - - cascading.PlatformTestCase
        - - cascading.JoinFieldedPipesPlatformTest

- All Implemented Interfaces:
    java.io.Serializable, junit.framework.Test

      public class JoinFieldedPipesPlatformTest
      extends PlatformTestCase

  See Also:
    Serialized Form

- - ### Nested Class Summary

    Nested Classes
    Modifier and Type | Class and Description
    --- | ---
    `static class` | `JoinFieldedPipesPlatformTest.AllComparator`

  - ### Field Summary

    - ### Fields inherited from class cascading.CascadingTestCase

      `name, ROOT_OUTPUT_PATH, ROOT_PLAN_PATH, TEST_TRACEPLAN_ENABLED`

  - ### Constructor Summary

    Constructors
    Constructor and Description
    `JoinFieldedPipesPlatformTest()`

  - ### Method Summary

    Methods
    Modifier and Type | Method and Description
    --- | ---
    `void` | `testCross()`
    `void` | `testForkCoGroupThenHashJoin()` This test checks for a deadlock when the same input is forked, adapted on one edge, then hashjoined back together.
    `void` | `testForkCoGroupThenHashJoinCoGroupAgain()` This test checks for a deadlock when the same input is forked, adapted on one edge, cogroup with something, then hashjoined back together.
    `void` | `testForkThenJoin()` This test checks for a deadlock when the same input is forked, adapted on one edge, then hashjoined back together.
    `void` | `testGroupBySplitAroundSplitGroupByJoin()`
    `void` | `testGroupBySplitGroupByJoin()`
    `void` | `testGroupBySplitJoins()`
    `void` | `testGroupBySplitSplitGroupByJoin()`
    `void` | `testJoin()`
    `void` | `testJoinAfterEvery()` Method testCoGroupAfterEvery tests that a tmp tap is inserted after the Every in the cogroup join
    `void` | `testJoinAroundJoinLeftMost()` Same source as leftmost
    `void` | `testJoinAroundJoinRightMost()` Same source as rightmost should be a single job as the same file accumulates into the joins
    `void` | `testJoinAroundJoinRightMostSwapped()` Upper as leftmost and rightmost forcing two jobs
    `void` | `testJoinDiffFields()`
    `void` | `testJoinFilteredBranch()` this test intentionally filters out all values so the intermediate tap is empty.
    `void` | `testJoinGroupBy()`
    `void` | `testJoinGroupByJoin()`
    `void` | `testJoinInner()` 1 a1 1 a2 1 a3 2 b1 3 c1 4 d1 4 d2 4 d3 5 e1 5 e2 5 e3 7 g1 7 g2 7 g3 7 g4 7 g5 null h1 1 A1 1 A2 1 A3 2 B1 2 B2 2 B3 4 D1 6 F1 6 F2 null H1 1 a1 1 A1 1 a1 1 A2 1 a1 1 A3 1 a2 1 A1 1 a2 1 A2 1 a2 1 A3 1 a3 1 A1 1 a3 1 A2 1 a3 1 A3 2 b1 2 B1 2 b1 2 B2 2 b1 2 B3 4 d1 4 D1 4 d2 4 D1 4 d3 4 D1 null h1 null H1
    `void` | `testJoinInnerOuter()` 1 a1 1 a2 1 a3 2 b1 3 c1 4 d1 4 d2 4 d3 5 e1 5 e2 5 e3 7 g1 7 g2 7 g3 7 g4 7 g5 null h1 1 A1 1 A2 1 A3 2 B1 2 B2 2 B3 4 D1 6 F1 6 F2 null H1 1 a1 1 A1 1 a1 1 A2 1 a1 1 A3 1 a2 1 A1 1 a2 1 A2 1 a2 1 A3 1 a3 1 A1 1 a3 1 A2 1 a3 1 A3 2 b1 2 B1 2 b1 2 B2 2 b1 2 B3 3 c1 null null 4 d1 4 D1 4 d2 4 D1 4 d3 4 D1 5 e1 null null 5 e2 null null 5 e3 null null 7 g1 null null 7 g2 null null 7 g3 null null 7 g4 null null 7 g5 null null null h1 null H1
    `void` | `testJoinInnerSingleField()`
    `void` | `testJoinMergeGroupBy()` When run against a cluster a Merge before a GroupBy can hide the streamed/accumulated nature of a branch.
    `void` | `testJoinMixed()` 1 a 5 b 6 c 5 b 5 e 1 A 2 B 3 C 4 D 5 E 1 a 2 b 3 c 4 d 5 e 1 a 1 A 1 a - - 2 B 2 b - - 3 C 3 c - - 4 D 4 d 5 b 5 E 5 e 5 e 5 E 5 e
    `void` | `testJoinNone()`
    `void` | `testJoinOuter()` /** 1 a1 1 a2 1 a3 2 b1 3 c1 4 d1 4 d2 4 d3 5 e1 5 e2 5 e3 7 g1 7 g2 7 g3 7 g4 7 g5 null h1 1 A1 1 A2 1 A3 2 B1 2 B2 2 B3 4 D1 6 F1 6 F2 null H1 1 a1 1 A1 1 a1 1 A2 1 a1 1 A3 1 a2 1 A1 1 a2 1 A2 1 a2 1 A3 1 a3 1 A1 1 a3 1 A2 1 a3 1 A3 2 b1 2 B1 2 b1 2 B2 2 b1 2 B3 3 c1 null null 4 d1 4 D1 4 d2 4 D1 4 d3 4 D1 5 e1 null null 5 e2 null null 5 e3 null null null null 6 F1 null null 6 F2 7 g1 null null 7 g2 null null 7 g3 null null 7 g4 null null 7 g5 null null null h1 null H1
    `void` | `testJoinOuterInner()` 1 a1 1 a2 1 a3 2 b1 3 c1 4 d1 4 d2 4 d3 5 e1 5 e2 5 e3 7 g1 7 g2 7 g3 7 g4 7 g5 null h1 1 A1 1 A2 1 A3 2 B1 2 B2 2 B3 4 D1 6 F1 6 F2 null H1 1 a1 1 A1 1 a1 1 A2 1 a1 1 A3 1 a2 1 A1 1 a2 1 A2 1 a2 1 A3 1 a3 1 A1 1 a3 1 A2 1 a3 1 A3 2 b1 2 B1 2 b1 2 B2 2 b1 2 B3 4 d1 4 D1 4 d2 4 D1 4 d3 4 D1 null null 6 F1 null null 6 F2 null h1 null H1
    `void` | `testJoinSamePipe()`
    `void` | `testJoinSamePipe2()`
    `void` | `testJoinSamePipe3()`
    `void` | `testJoinSamePipeAroundGroupBy()`
    `void` | `testJoinSamePipeName()`
    `void` | `testJoinSameSourceIntoJoin()` here the same file is fed into the same HashJoin.
    `void` | `testJoinSameSourceIntoJoinSimple()`
    `void` | `testJoinSameSourceOverGroupByIntoJoinSimple()` Loosely tests for a deadlock when BlockingHashJoinAnnotator rule doesn't excluce the GroupBy from the blocking annotation.
    `void` | `testJoinSelf()`
    `void` | `testJoinsIntoCoGroup()`
    `void` | `testJoinsIntoCoGroupLhs()` This test results in two MR jobs because one join feeds into the accumulated side of the second.
    `void` | `testJoinsIntoCoGroupLhsSwappedJoin()` This test results in one MR jobs because one join feeds into the streamed side of the second.
    `void` | `testJoinsIntoCoGroupRhs()`
    `void` | `testJoinsIntoGroupBy()` Tests that two independent streamed sources with loadable tributaries properly plan into a GroupBy without loading unused sources
    `void` | `testJoinSplit()` Under tez, this can result in the HashJoin being duplicated across nodes for each split after the HashJoin BoundaryBalanceJoinSplitTransformer inserts a Boundary at the split, preventing duplication of the path
    `void` | `testJoinSplitBeforeJoin()` checks that a split after a HashJoin does not result in the HashJoin execution being duplicated across multiple nodes, one for each branch in the split.
    `void` | `testJoinWithHasher()` Tests Hasher being honored even if default comparator is null.
    `void` | `testJoinWithUnknowns()`
    `void` | `testSameSourceJoin()`
    `void` | `testSameSourceJoinSplitIntoJoin()` catches a situation where BottomUpJoinedBoundariesNodePartitioner may capture an invalid HashJoin sub-graph if the in-bound Boundary is split upon.

    - ### Methods inherited from class cascading.PlatformTestCase

      `copyFromLocal, disableDebug, getOutputPath, getOutputPathElements, getPlanPathElements, getPlatform, getPlatformName, getProperties, installPlatform, makeOutputPath, setUp, tearDown`

    - ### Methods inherited from class cascading.CascadingTestCase

      `asCollection, asCollection, asCollection, asCollection, asList, asList, asSet, asSet, getOutputPath, getPlanPath, getSinkAsList, getSourceAsList, getTestCaseName, getTestName, getTestOutputRoot, getTestPlanRoot, invokeAggregator, invokeAggregator, invokeAggregator, invokeAggregator, invokeAggregator, invokeAggregator, invokeBuffer, invokeBuffer, invokeBuffer, invokeBuffer, invokeBuffer, invokeBuffer, invokeFilter, invokeFilter, invokeFilter, invokeFilter, invokeFilter, invokeFilter, invokeFilter, invokeFilter, invokeFunction, invokeFunction, invokeFunction, invokeFunction, invokeFunction, invokeFunction, invokeFunction, invokeFunction, validateLength, validateLength, validateLength, validateLength, validateLength, validateLength, validateLength, validateLength, validateLength, validateLength`

    - ### Methods inherited from class junit.framework.TestCase

      `assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertFalse, assertFalse, assertNotNull, assertNotNull, assertNotSame, assertNotSame, assertNull, assertNull, assertSame, assertSame, assertTrue, assertTrue, countTestCases, createResult, fail, fail, failNotEquals, failNotSame, failSame, format, getName, run, run, runBare, runTest, setName, toString`

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait`

- - ### Constructor Detail

    - #### JoinFieldedPipesPlatformTest

          public JoinFieldedPipesPlatformTest()

  - ### Method Detail

    - #### testCross

          public void testCross()
                         throws java.lang.Exception

      Throws:
        `java.lang.Exception`

    - #### testJoin

          public void testJoin()
                        throws java.lang.Exception

      Throws:
        `java.lang.Exception`

    - #### testJoinSamePipeName

          public void testJoinSamePipeName()
                                    throws java.lang.Exception

      Throws:
        `java.lang.Exception`

    - #### testJoinWithUnknowns

          public void testJoinWithUnknowns()
                                    throws java.lang.Exception

      Throws:
        `java.lang.Exception`

    - #### testJoinFilteredBranch

          public void testJoinFilteredBranch()
                                      throws java.lang.Exception

      this test intentionally filters out all values so the intermediate tap is empty. this tap is cogrouped with a new stream using an outerjoin.
      Throws:
        `java.lang.Exception`

    - #### testJoinSelf

          public void testJoinSelf()
                            throws java.lang.Exception

      Throws:
        `java.lang.Exception`

    - #### testSameSourceJoin

          public void testSameSourceJoin()
                                  throws java.lang.Exception

      Throws:
        `java.lang.Exception`

    - #### testJoinAfterEvery

          public void testJoinAfterEvery()
                                  throws java.lang.Exception

      Method testCoGroupAfterEvery tests that a tmp tap is inserted after the Every in the cogroup join
      Throws:
        `java.lang.Exception` - when

    - #### testJoinInnerSingleField

          public void testJoinInnerSingleField()
                                        throws java.lang.Exception

      Throws:
        `java.lang.Exception`

    - #### testJoinInner

          public void testJoinInner()
                             throws java.lang.Exception

      1 a1 1 a2 1 a3 2 b1 3 c1 4 d1 4 d2 4 d3 5 e1 5 e2 5 e3 7 g1 7 g2 7 g3 7 g4 7 g5 null h1

      1 A1 1 A2 1 A3 2 B1 2 B2 2 B3 4 D1 6 F1 6 F2 null H1

      1 a1 1 A1 1 a1 1 A2 1 a1 1 A3 1 a2 1 A1 1 a2 1 A2 1 a2 1 A3 1 a3 1 A1 1 a3 1 A2 1 a3 1 A3 2 b1 2 B1 2 b1 2 B2 2 b1 2 B3 4 d1 4 D1 4 d2 4 D1 4 d3 4 D1 null h1 null H1
      Throws:
        `java.lang.Exception`

    - #### testJoinOuter

          public void testJoinOuter()
                             throws java.lang.Exception

      /** 1 a1 1 a2 1 a3 2 b1 3 c1 4 d1 4 d2 4 d3 5 e1 5 e2 5 e3 7 g1 7 g2 7 g3 7 g4 7 g5 null h1

      1 A1 1 A2 1 A3 2 B1 2 B2 2 B3 4 D1 6 F1 6 F2 null H1

      1 a1 1 A1 1 a1 1 A2 1 a1 1 A3 1 a2 1 A1 1 a2 1 A2 1 a2 1 A3 1 a3 1 A1 1 a3 1 A2 1 a3 1 A3 2 b1 2 B1 2 b1 2 B2 2 b1 2 B3 3 c1 null null 4 d1 4 D1 4 d2 4 D1 4 d3 4 D1 5 e1 null null 5 e2 null null 5 e3 null null null null 6 F1 null null 6 F2 7 g1 null null 7 g2 null null 7 g3 null null 7 g4 null null 7 g5 null null null h1 null H1
      Throws:
        `java.lang.Exception`

    - #### testJoinInnerOuter

          public void testJoinInnerOuter()
                                  throws java.lang.Exception

      1 a1 1 a2 1 a3 2 b1 3 c1 4 d1 4 d2 4 d3 5 e1 5 e2 5 e3 7 g1 7 g2 7 g3 7 g4 7 g5 null h1

      1 A1 1 A2 1 A3 2 B1 2 B2 2 B3 4 D1 6 F1 6 F2 null H1

      1 a1 1 A1 1 a1 1 A2 1 a1 1 A3 1 a2 1 A1 1 a2 1 A2 1 a2 1 A3 1 a3 1 A1 1 a3 1 A2 1 a3 1 A3 2 b1 2 B1 2 b1 2 B2 2 b1 2 B3 3 c1 null null 4 d1 4 D1 4 d2 4 D1 4 d3 4 D1 5 e1 null null 5 e2 null null 5 e3 null null 7 g1 null null 7 g2 null null 7 g3 null null 7 g4 null null 7 g5 null null null h1 null H1
      Throws:
        `java.lang.Exception`

    - #### testJoinOuterInner

          public void testJoinOuterInner()
                                  throws java.lang.Exception

      1 a1 1 a2 1 a3 2 b1 3 c1 4 d1 4 d2 4 d3 5 e1 5 e2 5 e3 7 g1 7 g2 7 g3 7 g4 7 g5 null h1

      1 A1 1 A2 1 A3 2 B1 2 B2 2 B3 4 D1 6 F1 6 F2 null H1

      1 a1 1 A1 1 a1 1 A2 1 a1 1 A3 1 a2 1 A1 1 a2 1 A2 1 a2 1 A3 1 a3 1 A1 1 a3 1 A2 1 a3 1 A3 2 b1 2 B1 2 b1 2 B2 2 b1 2 B3 4 d1 4 D1 4 d2 4 D1 4 d3 4 D1 null null 6 F1 null null 6 F2 null h1 null H1
      Throws:
        `java.lang.Exception`

    - #### testJoinMixed

          public void testJoinMixed()
                             throws java.lang.Exception

      1 a 5 b 6 c 5 b 5 e

      1 A 2 B 3 C 4 D 5 E

      1 a 2 b 3 c 4 d 5 e

      1 a 1 A 1 a - - 2 B 2 b - - 3 C 3 c - - 4 D 4 d 5 b 5 E 5 e 5 e 5 E 5 e
      Throws:
        `java.lang.Exception`

    - #### testJoinDiffFields

          public void testJoinDiffFields()
                                  throws java.lang.Exception

      Throws:
        `java.lang.Exception`

    - #### testJoinGroupBy

          public void testJoinGroupBy()
                               throws java.lang.Exception

      Throws:
        `java.lang.Exception`

    - #### testJoinSamePipe

          public void testJoinSamePipe()
                                throws java.lang.Exception

      Throws:
        `java.lang.Exception`

    - #### testJoinSamePipe2

          public void testJoinSamePipe2()
                                 throws java.lang.Exception

      Throws:
        `java.lang.Exception`

    - #### testJoinSamePipe3

          public void testJoinSamePipe3()
                                 throws java.lang.Exception

      Throws:
        `java.lang.Exception`

    - #### testJoinAroundJoinRightMost

          public void testJoinAroundJoinRightMost()
                                           throws java.lang.Exception

      Same source as rightmost

      should be a single job as the same file accumulates into the joins
      Throws:
        `java.lang.Exception`

    - #### testJoinAroundJoinLeftMost

          public void testJoinAroundJoinLeftMost()
                                          throws java.lang.Exception

      Same source as leftmost
      Throws:
        `java.lang.Exception`

    - #### testJoinAroundJoinRightMostSwapped

          public void testJoinAroundJoinRightMostSwapped()
                                                  throws java.lang.Exception

      Upper as leftmost and rightmost forcing two jobs
      Throws:
        `java.lang.Exception`

    - #### testJoinGroupByJoin

          public void testJoinGroupByJoin()
                                   throws java.lang.Exception

      Throws:
        `java.lang.Exception`

    - #### testJoinSameSourceIntoJoin

          public void testJoinSameSourceIntoJoin()
                                          throws java.lang.Exception

      here the same file is fed into the same HashJoin.

      This is three jobs.

      a temp tap is inserted before the accumulated branch for two reasons on the common HashJoin

      it is assumed the accumulated side is filtered down, so pushing to disk will preserve io if accumulated side was streamed instead via a fork, only part of the file will accumulate into the HashJoin

      /-T-\ <-- accumulated T HJ \---/ <-- streamed
      Throws:
        `java.lang.Exception`

    - #### testJoinSameSourceIntoJoinSimple

          public void testJoinSameSourceIntoJoinSimple()
                                                throws java.lang.Exception

      Throws:
        `java.lang.Exception`

    - #### testJoinSameSourceOverGroupByIntoJoinSimple

          public void testJoinSameSourceOverGroupByIntoJoinSimple()
                                                           throws java.lang.Exception

      Loosely tests for a deadlock when BlockingHashJoinAnnotator rule doesn't excluce the GroupBy from the blocking annotation.

      the deadlock is random on the order of the paths traversed from the Source Tap + fork.
      Throws:
        `java.lang.Exception`

    - #### testJoinsIntoGroupBy

          public void testJoinsIntoGroupBy()
                                    throws java.lang.Exception

      Tests that two independent streamed sources with loadable tributaries properly plan into a GroupBy without loading unused sources
      Throws:
        `java.lang.Exception`

    - #### testJoinSamePipeAroundGroupBy

          public void testJoinSamePipeAroundGroupBy()
                                             throws java.lang.Exception

      Throws:
        `java.lang.Exception`

    - #### testJoinsIntoCoGroupLhs

          public void testJoinsIntoCoGroupLhs()
                                       throws java.lang.Exception

      This test results in two MR jobs because one join feeds into the accumulated side of the second. A mapper can only stream on branch at a time forcing a temp file between the mappers. see next test for swapped join
      Throws:
        `java.lang.Exception`

    - #### testJoinsIntoCoGroupLhsSwappedJoin

          public void testJoinsIntoCoGroupLhsSwappedJoin()
                                                  throws java.lang.Exception

      This test results in one MR jobs because one join feeds into the streamed side of the second.
      Throws:
        `java.lang.Exception`

    - #### testJoinsIntoCoGroupRhs

          public void testJoinsIntoCoGroupRhs()
                                       throws java.lang.Exception

      Throws:
        `java.lang.Exception`

    - #### testJoinsIntoCoGroup

          public void testJoinsIntoCoGroup()
                                    throws java.lang.Exception

      Throws:
        `java.lang.Exception`

    - #### testJoinWithHasher

          public void testJoinWithHasher()
                                  throws java.lang.Exception

      Tests Hasher being honored even if default comparator is null.
      Throws:
        `java.lang.Exception`

    - #### testJoinNone

          public void testJoinNone()
                            throws java.lang.Exception

      Throws:
        `java.lang.Exception`

    - #### testGroupBySplitJoins

          public void testGroupBySplitJoins()
                                     throws java.lang.Exception

      Throws:
        `java.lang.Exception`

    - #### testJoinMergeGroupBy

          public void testJoinMergeGroupBy()
                                    throws java.lang.Exception

      When run against a cluster a Merge before a GroupBy can hide the streamed/accumulated nature of a branch.

      The planner nw

      commented code is for troubleshooting.
      Throws:
        `java.lang.Exception`

    - #### testJoinSplit

          public void testJoinSplit()
                             throws java.lang.Exception

      Under tez, this can result in the HashJoin being duplicated across nodes for each split after the HashJoin BoundaryBalanceJoinSplitTransformer inserts a Boundary at the split, preventing duplication of the path
      Throws:
        `java.lang.Exception`

    - #### testSameSourceJoinSplitIntoJoin

          public void testSameSourceJoinSplitIntoJoin()
                                               throws java.lang.Exception

      catches a situation where BottomUpJoinedBoundariesNodePartitioner may capture an invalid HashJoin sub-graph if the in-bound Boundary is split upon.
      Throws:
        `java.lang.Exception`

    - #### testJoinSplitBeforeJoin

          public void testJoinSplitBeforeJoin()
                                       throws java.lang.Exception

      checks that a split after a HashJoin does not result in the HashJoin execution being duplicated across multiple nodes, one for each branch in the split.
      Throws:
        `java.lang.Exception`

    - #### testGroupBySplitGroupByJoin

          public void testGroupBySplitGroupByJoin()
                                           throws java.lang.Exception

      Throws:
        `java.lang.Exception`

    - #### testGroupBySplitSplitGroupByJoin

          public void testGroupBySplitSplitGroupByJoin()
                                                throws java.lang.Exception

      Throws:
        `java.lang.Exception`

    - #### testGroupBySplitAroundSplitGroupByJoin

          public void testGroupBySplitAroundSplitGroupByJoin()
                                                      throws java.lang.Exception

      Throws:
        `java.lang.Exception`

    - #### testForkThenJoin

          public void testForkThenJoin()
                                throws java.lang.Exception

      This test checks for a deadlock when the same input is forked, adapted on one edge, then hashjoined back together.
      Throws:
        `java.lang.Exception`

    - #### testForkCoGroupThenHashJoin

          public void testForkCoGroupThenHashJoin()
                                           throws java.lang.Exception

      This test checks for a deadlock when the same input is forked, adapted on one edge, then hashjoined back together.
      Throws:
        `java.lang.Exception`

    - #### testForkCoGroupThenHashJoinCoGroupAgain

          public void testForkCoGroupThenHashJoinCoGroupAgain()
                                                       throws java.lang.Exception

      This test checks for a deadlock when the same input is forked, adapted on one edge, cogroup with something, then hashjoined back together.
      Throws:
        `java.lang.Exception`

Copyright © 2007-2015 Xplenty, Inc. All Rights Reserved.