cascading

Class JoinFieldedPipesPlatformTest

java.lang.Object

junit.framework.Assert

junit.framework.TestCase

cascading.CascadingTestCase

cascading.PlatformTestCase

cascading.JoinFieldedPipesPlatformTest

All Implemented Interfaces:

java.io.Serializable, junit.framework.Test

public class JoinFieldedPipesPlatformTest
extends PlatformTestCase

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	JoinFieldedPipesPlatformTest.AllComparator

Field Summary

Fields inherited from class cascading.CascadingTestCase

name, ROOT_OUTPUT_PATH, ROOT_PLAN_PATH, TEST_TRACEPLAN_ENABLED

Constructor Summary

Constructors

Constructor and Description
JoinFieldedPipesPlatformTest()

Method Summary

Modifier and Type	Method and Description
void	testCross()
void	testForkCoGroupThenHashJoin() This test checks for a deadlock when the same input is forked, adapted on one edge, then hashjoined back together.
void	testForkCoGroupThenHashJoinCoGroupAgain() This test checks for a deadlock when the same input is forked, adapted on one edge, cogroup with something, then hashjoined back together.
void	testForkThenJoin() This test checks for a deadlock when the same input is forked, adapted on one edge, then hashjoined back together.
void	testGroupBySplitAroundSplitGroupByJoin()
void	testGroupBySplitGroupByJoin()
void	testGroupBySplitJoins()
void	testGroupBySplitSplitGroupByJoin()
void	testJoin()
void	testJoinAfterEvery() Method testCoGroupAfterEvery tests that a tmp tap is inserted after the Every in the cogroup join
void	testJoinAroundJoinLeftMost() Same source as leftmost
void	testJoinAroundJoinRightMost() Same source as rightmost
void	testJoinAroundJoinRightMostSwapped() Upper as leftmost and rightmost forcing two jobs
void	testJoinDiffFields()
void	testJoinFilteredBranch() this test intentionally filters out all values so the intermediate tap is empty.
void	testJoinGroupBy()
void	testJoinGroupByJoin()
void	testJoinInner() 1 a1 1 a2 1 a3 2 b1 3 c1 4 d1 4 d2 4 d3 5 e1 5 e2 5 e3 7 g1 7 g2 7 g3 7 g4 7 g5 null h1
void	testJoinInnerOuter() 1 a1 1 a2 1 a3 2 b1 3 c1 4 d1 4 d2 4 d3 5 e1 5 e2 5 e3 7 g1 7 g2 7 g3 7 g4 7 g5 null h1
void	testJoinInnerSingleField()
void	testJoinMergeGroupBy() When run against a cluster a Merge before a GroupBy can hide the streamed/accumulated nature of a branch.
void	testJoinMixed() 1 a 5 b 6 c 5 b 5 e
void	testJoinNone()
void	testJoinOuter() /** 1 a1 1 a2 1 a3 2 b1 3 c1 4 d1 4 d2 4 d3 5 e1 5 e2 5 e3 7 g1 7 g2 7 g3 7 g4 7 g5 null h1
void	testJoinOuterInner() 1 a1 1 a2 1 a3 2 b1 3 c1 4 d1 4 d2 4 d3 5 e1 5 e2 5 e3 7 g1 7 g2 7 g3 7 g4 7 g5 null h1
void	testJoinSamePipe()
void	testJoinSamePipe2()
void	testJoinSamePipe3()
void	testJoinSamePipeAroundGroupBy()
void	testJoinSamePipeName()
void	testJoinSameSourceIntoJoin() here the same file is fed into the same HashJoin.
void	testJoinSameSourceIntoJoinSimple()
void	testJoinSameSourceOverGroupByIntoJoinSimple() Loosely tests for a deadlock when BlockingHashJoinAnnotator rule doesn't excluce the GroupBy from the blocking annotation.
void	testJoinSelf()
void	testJoinsIntoCoGroup()
void	testJoinsIntoCoGroupLhs() This test results in two MR jobs because one join feeds into the accumulated side of the second.
void	testJoinsIntoCoGroupLhsSwappedJoin() This test results in one MR jobs because one join feeds into the streamed side of the second.
void	testJoinsIntoCoGroupRhs()
void	testJoinsIntoGroupBy() Tests that two independent streamed sources with loadable tributaries properly plan into a GroupBy without loading unused sources
void	testJoinSplit() Under tez, this can result in the HashJoin being duplicated across nodes for each split after the HashJoin BoundaryBalanceJoinSplitTransformer inserts a Boundary at the split, preventing duplication of the path
void	testJoinSplitBeforeJoin() checks that a split after a HashJoin does not result in the HashJoin execution being duplicated across multiple nodes, one for each branch in the split.
void	testJoinWithHasher() Tests Hasher being honored even if default comparator is null.
void	testJoinWithUnknowns()
void	testSameSourceJoin()
void	testSameSourceJoinSplitIntoJoin() catches a situation where BottomUpJoinedBoundariesNodePartitioner may capture an invalid HashJoin sub-graph if the in-bound Boundary is split upon.

Methods inherited from class cascading.PlatformTestCase

copyFromLocal, disableDebug, getOutputPath, getOutputPathElements, getPlanPathElements, getPlatform, getPlatformName, getProperties, installPlatform, makeOutputPath, setUp, tearDown

Methods inherited from class cascading.CascadingTestCase

asCollection, asCollection, asCollection, asCollection, asList, asList, asSet, asSet, getOutputPath, getPlanPath, getSinkAsList, getSourceAsList, getTestCaseName, getTestName, getTestOutputRoot, getTestPlanRoot, invokeAggregator, invokeAggregator, invokeAggregator, invokeAggregator, invokeAggregator, invokeAggregator, invokeBuffer, invokeBuffer, invokeBuffer, invokeBuffer, invokeBuffer, invokeBuffer, invokeFilter, invokeFilter, invokeFilter, invokeFilter, invokeFilter, invokeFilter, invokeFilter, invokeFilter, invokeFunction, invokeFunction, invokeFunction, invokeFunction, invokeFunction, invokeFunction, invokeFunction, invokeFunction, validateLength, validateLength, validateLength, validateLength, validateLength, validateLength, validateLength, validateLength, validateLength, validateLength

Methods inherited from class junit.framework.TestCase

assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertFalse, assertFalse, assertNotNull, assertNotNull, assertNotSame, assertNotSame, assertNull, assertNull, assertSame, assertSame, assertTrue, assertTrue, countTestCases, createResult, fail, fail, failNotEquals, failNotSame, failSame, format, getName, run, run, runBare, runTest, setName, toString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

JoinFieldedPipesPlatformTest

public JoinFieldedPipesPlatformTest()

Method Detail

testCross

public void testCross()
               throws java.lang.Exception

Throws:

java.lang.Exception

testJoin

public void testJoin()
              throws java.lang.Exception

Throws:

java.lang.Exception

testJoinSamePipeName

public void testJoinSamePipeName()
                          throws java.lang.Exception

Throws:

java.lang.Exception

testJoinWithUnknowns

public void testJoinWithUnknowns()
                          throws java.lang.Exception

Throws:

java.lang.Exception

testJoinFilteredBranch

public void testJoinFilteredBranch()
                            throws java.lang.Exception

this test intentionally filters out all values so the intermediate tap is empty. this tap is cogrouped with a new stream using an outerjoin.

Throws:

java.lang.Exception

testJoinSelf

public void testJoinSelf()
                  throws java.lang.Exception

Throws:

java.lang.Exception

testSameSourceJoin

public void testSameSourceJoin()
                        throws java.lang.Exception

Throws:

java.lang.Exception

testJoinAfterEvery

public void testJoinAfterEvery()
                        throws java.lang.Exception

Method testCoGroupAfterEvery tests that a tmp tap is inserted after the Every in the cogroup join

Throws:

java.lang.Exception - when

testJoinInnerSingleField

public void testJoinInnerSingleField()
                              throws java.lang.Exception

Throws:

java.lang.Exception

testJoinInner

public void testJoinInner()
                   throws java.lang.Exception

1 a1 1 a2 1 a3 2 b1 3 c1 4 d1 4 d2 4 d3 5 e1 5 e2 5 e3 7 g1 7 g2 7 g3 7 g4 7 g5 null h1

1 A1 1 A2 1 A3 2 B1 2 B2 2 B3 4 D1 6 F1 6 F2 null H1

1 a1 1 A1 1 a1 1 A2 1 a1 1 A3 1 a2 1 A1 1 a2 1 A2 1 a2 1 A3 1 a3 1 A1 1 a3 1 A2 1 a3 1 A3 2 b1 2 B1 2 b1 2 B2 2 b1 2 B3 4 d1 4 D1 4 d2 4 D1 4 d3 4 D1 null h1 null H1

Throws:

java.lang.Exception

testJoinOuter

public void testJoinOuter()
                   throws java.lang.Exception

/** 1 a1 1 a2 1 a3 2 b1 3 c1 4 d1 4 d2 4 d3 5 e1 5 e2 5 e3 7 g1 7 g2 7 g3 7 g4 7 g5 null h1

1 A1 1 A2 1 A3 2 B1 2 B2 2 B3 4 D1 6 F1 6 F2 null H1

1 a1 1 A1 1 a1 1 A2 1 a1 1 A3 1 a2 1 A1 1 a2 1 A2 1 a2 1 A3 1 a3 1 A1 1 a3 1 A2 1 a3 1 A3 2 b1 2 B1 2 b1 2 B2 2 b1 2 B3 3 c1 null null 4 d1 4 D1 4 d2 4 D1 4 d3 4 D1 5 e1 null null 5 e2 null null 5 e3 null null null null 6 F1 null null 6 F2 7 g1 null null 7 g2 null null 7 g3 null null 7 g4 null null 7 g5 null null null h1 null H1

Throws:

java.lang.Exception

testJoinInnerOuter

public void testJoinInnerOuter()
                        throws java.lang.Exception

1 a1 1 a2 1 a3 2 b1 3 c1 4 d1 4 d2 4 d3 5 e1 5 e2 5 e3 7 g1 7 g2 7 g3 7 g4 7 g5 null h1

1 A1 1 A2 1 A3 2 B1 2 B2 2 B3 4 D1 6 F1 6 F2 null H1

1 a1 1 A1 1 a1 1 A2 1 a1 1 A3 1 a2 1 A1 1 a2 1 A2 1 a2 1 A3 1 a3 1 A1 1 a3 1 A2 1 a3 1 A3 2 b1 2 B1 2 b1 2 B2 2 b1 2 B3 3 c1 null null 4 d1 4 D1 4 d2 4 D1 4 d3 4 D1 5 e1 null null 5 e2 null null 5 e3 null null 7 g1 null null 7 g2 null null 7 g3 null null 7 g4 null null 7 g5 null null null h1 null H1

Throws:

java.lang.Exception

testJoinOuterInner

public void testJoinOuterInner()
                        throws java.lang.Exception

1 a1 1 a2 1 a3 2 b1 3 c1 4 d1 4 d2 4 d3 5 e1 5 e2 5 e3 7 g1 7 g2 7 g3 7 g4 7 g5 null h1

1 A1 1 A2 1 A3 2 B1 2 B2 2 B3 4 D1 6 F1 6 F2 null H1

1 a1 1 A1 1 a1 1 A2 1 a1 1 A3 1 a2 1 A1 1 a2 1 A2 1 a2 1 A3 1 a3 1 A1 1 a3 1 A2 1 a3 1 A3 2 b1 2 B1 2 b1 2 B2 2 b1 2 B3 4 d1 4 D1 4 d2 4 D1 4 d3 4 D1 null null 6 F1 null null 6 F2 null h1 null H1

Throws:

java.lang.Exception

testJoinMixed

public void testJoinMixed()
                   throws java.lang.Exception

1 a 5 b 6 c 5 b 5 e

1 A 2 B 3 C 4 D 5 E

1 a 2 b 3 c 4 d 5 e

1 a 1 A 1 a - - 2 B 2 b - - 3 C 3 c - - 4 D 4 d 5 b 5 E 5 e 5 e 5 E 5 e

Throws:

java.lang.Exception

testJoinDiffFields

public void testJoinDiffFields()
                        throws java.lang.Exception

Throws:

java.lang.Exception

testJoinGroupBy

public void testJoinGroupBy()
                     throws java.lang.Exception

Throws:

java.lang.Exception

testJoinSamePipe

public void testJoinSamePipe()
                      throws java.lang.Exception

Throws:

java.lang.Exception

testJoinSamePipe2

public void testJoinSamePipe2()
                       throws java.lang.Exception

Throws:

java.lang.Exception

testJoinSamePipe3

public void testJoinSamePipe3()
                       throws java.lang.Exception

Throws:

java.lang.Exception

testJoinAroundJoinRightMost

public void testJoinAroundJoinRightMost()
                                 throws java.lang.Exception

Same source as rightmost

should be a single job as the same file accumulates into the joins

Throws:

java.lang.Exception

testJoinAroundJoinLeftMost

public void testJoinAroundJoinLeftMost()
                                throws java.lang.Exception

Same source as leftmost

Throws:

java.lang.Exception

testJoinAroundJoinRightMostSwapped

public void testJoinAroundJoinRightMostSwapped()
                                        throws java.lang.Exception

Upper as leftmost and rightmost forcing two jobs

Throws:

java.lang.Exception

testJoinGroupByJoin

public void testJoinGroupByJoin()
                         throws java.lang.Exception

Throws:

java.lang.Exception

testJoinSameSourceIntoJoin

public void testJoinSameSourceIntoJoin()
                                throws java.lang.Exception

here the same file is fed into the same HashJoin.

This is three jobs.

a temp tap is inserted before the accumulated branch for two reasons on the common HashJoin

it is assumed the accumulated side is filtered down, so pushing to disk will preserve io if accumulated side was streamed instead via a fork, only part of the file will accumulate into the HashJoin

/-T-\ <-- accumulated T HJ \---/ <-- streamed

Throws:

java.lang.Exception

testJoinSameSourceIntoJoinSimple

public void testJoinSameSourceIntoJoinSimple()
                                      throws java.lang.Exception

Throws:

java.lang.Exception

testJoinSameSourceOverGroupByIntoJoinSimple

public void testJoinSameSourceOverGroupByIntoJoinSimple()
                                                 throws java.lang.Exception

Loosely tests for a deadlock when BlockingHashJoinAnnotator rule doesn't excluce the GroupBy from the blocking annotation.

the deadlock is random on the order of the paths traversed from the Source Tap + fork.

Throws:

java.lang.Exception

testJoinsIntoGroupBy

public void testJoinsIntoGroupBy()
                          throws java.lang.Exception

Tests that two independent streamed sources with loadable tributaries properly plan into a GroupBy without loading unused sources

Throws:

java.lang.Exception

testJoinSamePipeAroundGroupBy

public void testJoinSamePipeAroundGroupBy()
                                   throws java.lang.Exception

Throws:

java.lang.Exception

testJoinsIntoCoGroupLhs

public void testJoinsIntoCoGroupLhs()
                             throws java.lang.Exception

This test results in two MR jobs because one join feeds into the accumulated side of the second. A mapper can only stream on branch at a time forcing a temp file between the mappers. see next test for swapped join

Throws:

java.lang.Exception

testJoinsIntoCoGroupLhsSwappedJoin

public void testJoinsIntoCoGroupLhsSwappedJoin()
                                        throws java.lang.Exception

This test results in one MR jobs because one join feeds into the streamed side of the second.

Throws:

java.lang.Exception

testJoinsIntoCoGroupRhs

public void testJoinsIntoCoGroupRhs()
                             throws java.lang.Exception

Throws:

java.lang.Exception

testJoinsIntoCoGroup

public void testJoinsIntoCoGroup()
                          throws java.lang.Exception

Throws:

java.lang.Exception

testJoinWithHasher

public void testJoinWithHasher()
                        throws java.lang.Exception

Tests Hasher being honored even if default comparator is null.

Throws:

java.lang.Exception

testJoinNone

public void testJoinNone()
                  throws java.lang.Exception

Throws:

java.lang.Exception

testGroupBySplitJoins

public void testGroupBySplitJoins()
                           throws java.lang.Exception

Throws:

java.lang.Exception

testJoinMergeGroupBy

public void testJoinMergeGroupBy()
                          throws java.lang.Exception

When run against a cluster a Merge before a GroupBy can hide the streamed/accumulated nature of a branch.

The planner nw

commented code is for troubleshooting.

Throws:

java.lang.Exception

testJoinSplit

public void testJoinSplit()
                   throws java.lang.Exception

Under tez, this can result in the HashJoin being duplicated across nodes for each split after the HashJoin BoundaryBalanceJoinSplitTransformer inserts a Boundary at the split, preventing duplication of the path

Throws:

java.lang.Exception

testSameSourceJoinSplitIntoJoin

public void testSameSourceJoinSplitIntoJoin()
                                     throws java.lang.Exception

catches a situation where BottomUpJoinedBoundariesNodePartitioner may capture an invalid HashJoin sub-graph if the in-bound Boundary is split upon.

Throws:

java.lang.Exception

testJoinSplitBeforeJoin

public void testJoinSplitBeforeJoin()
                             throws java.lang.Exception

checks that a split after a HashJoin does not result in the HashJoin execution being duplicated across multiple nodes, one for each branch in the split.

Throws:

java.lang.Exception

testGroupBySplitGroupByJoin

public void testGroupBySplitGroupByJoin()
                                 throws java.lang.Exception

Throws:

java.lang.Exception

testGroupBySplitSplitGroupByJoin

public void testGroupBySplitSplitGroupByJoin()
                                      throws java.lang.Exception

Throws:

java.lang.Exception

testGroupBySplitAroundSplitGroupByJoin

public void testGroupBySplitAroundSplitGroupByJoin()
                                            throws java.lang.Exception

Throws:

java.lang.Exception

testForkThenJoin

public void testForkThenJoin()
                      throws java.lang.Exception

This test checks for a deadlock when the same input is forked, adapted on one edge, then hashjoined back together.

Throws:

java.lang.Exception

testForkCoGroupThenHashJoin

public void testForkCoGroupThenHashJoin()
                                 throws java.lang.Exception

This test checks for a deadlock when the same input is forked, adapted on one edge, then hashjoined back together.

Throws:

java.lang.Exception

testForkCoGroupThenHashJoinCoGroupAgain

public void testForkCoGroupThenHashJoinCoGroupAgain()
                                             throws java.lang.Exception

This test checks for a deadlock when the same input is forked, adapted on one edge, cogroup with something, then hashjoined back together.

Throws:

java.lang.Exception