Skip to content

Commit d701292

Browse files
jaceklaskowskijaceklaskowski
committed
Except Logical Operator
1 parent e0c7e55 commit d701292

File tree

1 file changed

+37
-48
lines changed

1 file changed

+37
-48
lines changed

‎docs/logical-operators/Except.md

Lines changed: 37 additions & 48 deletions
Original file line numberDiff line numberDiff line change
@@ -4,61 +4,48 @@ title: Except
44

55
# Except Logical Operator
66

7-
`Except` is a spark-sql-LogicalPlan.md#BinaryNode[binary logical operator] that represents the following high-level operators in a logical plan:
7+
`Except` is a `SetOperation` binary logical operator that represents the following high-level operators (in a logical plan):
88

9-
* `EXCEPT [ DISTINCT | ALL ]` and `MINUS [ DISTINCT | ALL ]` SQL statements (cf. sql/AstBuilder.md#visitSetOperation[AstBuilder])
9+
* `EXCEPT [ DISTINCT | ALL ]` and `MINUS [ DISTINCT | ALL ]` SQL statements (cf. [AstBuilder](../sql/AstBuilder.md#visitSetOperation))
10+
* [Dataset.except](../Dataset.md#except) and [Dataset.exceptAll](../Dataset.md#exceptAll)
1011

11-
* spark-sql-dataset-operators.md#except[Dataset.except] and spark-sql-dataset-operators.md#exceptAll[Dataset.exceptAll]
12+
## Creating Instance
1213

13-
`Except` is supposed to be resolved (_optimized_) to <<logical-conversions, other logical commands>> at logical optimization phase (i.e. `Except` should not be part of a logical plan after logical optimization). [BasicOperators](../execution-planning-strategies/BasicOperators.md) execution planning strategy throws an `IllegalStateException` if conversions did not happen.
14-
15-
[[logical-conversions]]
16-
.Except's Logical Resolutions (Conversions)
17-
[cols="30,70",options="header",width="100%"]
18-
|===
19-
| Target Logical Operators
20-
| Optimization Rules and Demos
21-
22-
| Left-Anti Join.md[Join]
23-
| `Except` (DISTINCT) in ReplaceExceptWithAntiJoin.md[ReplaceExceptWithAntiJoin] logical optimization rule
24-
25-
Consult <<demo-left-anti-join, Demo: Except Operator Replaced with Left-Anti Join>>
26-
27-
| `Filter`
28-
| `Except` (DISTINCT) in [ReplaceExceptWithFilter](../logical-optimizations/ReplaceExceptWithFilter.md) logical optimization rule
29-
30-
Consult <<demo-except-filter, Demo: Except Operator Replaced with Filter Operator>>
14+
`Except` takes the following to be created:
3115

32-
| `Union`, Aggregate.md[Aggregate] and Generate.md[Generate]
33-
| `Except` (ALL) in RewriteExceptAll.md[RewriteExceptAll] logical optimization rule
16+
* <span id="left"> Left [logical operator](LogicalPlan.md)
17+
* <span id="right"> Right [logical operator](LogicalPlan.md)
18+
* <span id="isAll"> `isAll` flag for `DISTINCT` (`false`) or `ALL` (`true`)
3419

35-
Consult <<demo-except-all, Demo: Except (All) Operator Replaced with Union, Aggregate and Generate Operators>>
20+
`Except` is created when:
3621

37-
|===
22+
* `AstBuilder` is requested to [visit a SetOperation](../sql/AstBuilder.md#visitSetOperation) (`EXCEPT` and `MINUS` operators)
23+
* [Dataset.except](../Dataset.md#except) and [Dataset.exceptAll](../Dataset.md#exceptAll) operators are used
24+
* Catalyst DSL's [except](../catalyst-dsl/DslLogicalPlan.md#except) operator is used
3825

39-
The types of the <<left, left>> and <<right, right>> logical (sub)operators can be widen in `WidenSetOperationTypes` logical analysis type-coercion rule.
26+
## Logical Optimization
4027

41-
=== [[creating-instance]] Creating Except Instance
28+
`Except` is supposed to be resolved (_optimized_) to other logical commands at logical optimization phase (i.e. `Except` should not be part of a logical plan after logical optimization).
4229

43-
`Except` takes the following to be created:
30+
[BasicOperators](../execution-planning-strategies/BasicOperators.md) execution planning strategy throws an `IllegalStateException` if conversions did not happen.
4431

45-
* [[left]] Left spark-sql-LogicalPlan.md[logical operator]
46-
* [[right]] Right spark-sql-LogicalPlan.md[logical operator]
47-
* [[isAll]] `isAll` flag for `DISTINCT` (`false`) or `ALL` (`true`)
32+
Target Logical Operators | Optimization Rules and Demos
33+
-------------------------|-----------------------------
34+
Left-Anti [Join](Join.md) | `Except` (DISTINCT) in [ReplaceExceptWithAntiJoin](../logical-optimizations/ReplaceExceptWithAntiJoin.md) logical optimization rule<p>Demo: [Except Operator Replaced with Left-Anti Join](#demo-left-anti-join)
35+
`Filter` | `Except` (DISTINCT) in [ReplaceExceptWithFilter](../logical-optimizations/ReplaceExceptWithFilter.md) logical optimization rule<p>Demo: [Except Operator Replaced with Filter Operator](#demo-except-filter)
36+
`Union`, [Aggregate](Aggregate.md) and [Generate](Generate.md) | `Except` (ALL) in [RewriteExceptAll](../logical-optimizations/RewriteExceptAll.md) logical optimization rule<p>Demo: [Except (All) Operator Replaced with Union, Aggregate and Generate Operators](#demo-except-all)
4837

49-
=== [[catalyst-dsl]] Catalyst DSL -- `except` Operator
38+
## Catalyst DSL
5039

51-
[source, scala]
52-
----
40+
```scala
5341
except(
5442
otherPlan: LogicalPlan,
5543
isAll: Boolean): LogicalPlan
56-
----
44+
```
5745

58-
[Catalyst DSL](../catalyst-dsl/index.md) defines [except](../catalyst-dsl/index.md#except) extension method to create an `Except` logical operator, e.g. for testing or Spark SQL internals exploration.
46+
[Catalyst DSL](../catalyst-dsl/index.md) defines [except](../catalyst-dsl/index.md#except) extension method to create an `Except` logical operator (e.g. for testing or Spark SQL internals exploration).
5947

60-
[source, plaintext]
61-
----
48+
```text
6249
import org.apache.spark.sql.catalyst.dsl.plans._
6350
val plan = table("a").except(table("b"), isAll = false)
6451
scala> println(plan.numberedTreeString)
@@ -69,13 +56,13 @@ scala> println(plan.numberedTreeString)
6956
import org.apache.spark.sql.catalyst.plans.logical.Except
7057
val op = plan.p(0)
7158
assert(op.isInstanceOf[Except])
72-
----
59+
```
7360

74-
=== [[CheckAnalysis]] Except Only on Relations with Same Number of Columns
61+
## Except Only on Relations with Same Number of Columns { #CheckAnalysis }
7562

76-
`Except` logical operator can only be performed on CheckAnalysis.md#checkAnalysis[tables with the same number of columns].
63+
`Except` logical operator can only be performed on [tables with the same number of columns](../CheckAnalysis.md#checkAnalysis).
7764

78-
```
65+
```text
7966
scala> left.except(right)
8067
org.apache.spark.sql.AnalysisException: Except can only be performed on tables with the same number of columns, but the first table has 3 columns and the second table has 4 columns;;
8168
'Except false
@@ -91,9 +78,11 @@ org.apache.spark.sql.AnalysisException: Except can only be performed on tables w
9178
...
9279
```
9380

94-
=== [[demo-left-anti-join]] Demo: Except Operator Replaced with Left-Anti Join
81+
## Demo
9582

96-
```
83+
### Except Operator Replaced with Left-Anti Join { #demo-left-anti-join }
84+
85+
```text
9786
Seq((0, "zero", "000"), (1, "one", "111"))
9887
.toDF("id", "name", "triple")
9988
.write
@@ -121,9 +110,9 @@ scala> println(q.queryExecution.optimizedPlan.numberedTreeString)
121110
03 +- Relation[id#209,name#210,triple#211] parquet
122111
```
123112

124-
=== [[demo-except-filter]] Demo: Except Operator Replaced with Filter Operator
113+
### Except Operator Replaced with Filter Operator { #demo-except-filter }
125114

126-
```
115+
```text
127116
Seq((0, "zero", "000"), (1, "one", "111"))
128117
.toDF("id", "name", "triple")
129118
.write
@@ -142,9 +131,9 @@ scala> println(q.queryExecution.optimizedPlan.numberedTreeString)
142131
02 +- Relation[id#16,name#17,triple#18] parquet
143132
```
144133

145-
=== [[demo-except-all]] Demo: Except (All) Operator Replaced with Union, Aggregate and Generate Operators
134+
### Except (All) Operator Replaced with Union, Aggregate and Generate Operators { #demo-except-all }
146135

147-
```
136+
```text
148137
Seq((0, "zero", "000"), (1, "one", "111"))
149138
.toDF("id", "name", "triple")
150139
.write

0 commit comments

Comments
 (0)