Skip to content

Commit 1017be7

Browse files
committed
Move the data parts of SourceDefinition to dbt/artifacts
Something interesting here is that we had to override the `freshness` property. We had to do this because if we didn't we wouldn't get the functional parts of `FreshnessThreshold`, we'd only get the data parts. Also of note, the `SourceDefintion` has a lot of `@property` methods that on other classes would be actual attribute properties of the node. There is an argument to be made that these should be moved as well, but thats perhaps a separate discussion. Finally, we have not (yet) moved `NodeInfoMixin`. It is an open discussion whether we do or not. It seems primarily functional, as a means to update the source freshness information. As the artifacts primarily deal with the shape of the data, not how it should be set, it seems for now that `NodeInfoMixin` should stay in core / not move to artifacts. This thinking may change though.
1 parent 36fd4c7 commit 1017be7

File tree

4 files changed

+39
-18
lines changed

4 files changed

+39
-18
lines changed
Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
kind: Under the Hood
2+
body: Move data parts of `SourceDefinition` class to dbt/artifacts
3+
time: 2024-02-08T12:06:20.696709-08:00
4+
custom:
5+
Author: QMalcolm
6+
Issue: "9384"

core/dbt/artifacts/resources/__init__.py

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -62,6 +62,7 @@
6262
from dbt.artifacts.resources.v1.source_definition import (
6363
ExternalPartition,
6464
ExternalTable,
65+
SourceDefinition,
6566
ParsedSourceMandatory,
6667
SourceConfig,
6768
)

core/dbt/artifacts/resources/v1/source_definition.py

Lines changed: 26 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,14 @@
1+
import time
2+
13
from dataclasses import dataclass, field
24
from dbt.artifacts.resources.base import GraphResource
35
from dbt.artifacts.resources.types import NodeType
4-
from dbt.artifacts.resources.v1.components import HasRelationMetadata
6+
from dbt.artifacts.resources.v1.components import (
7+
ColumnInfo,
8+
FreshnessThreshold,
9+
HasRelationMetadata,
10+
Quoting,
11+
)
512
from dbt_common.contracts.config.base import BaseConfig
613
from dbt_common.contracts.config.properties import AdditionalPropertiesAllowed
714
from dbt_common.contracts.util import Mergeable, Replaceable
@@ -45,3 +52,21 @@ class ParsedSourceMandatory(GraphResource, HasRelationMetadata):
4552
loader: str
4653
identifier: str
4754
resource_type: Literal[NodeType.Source]
55+
56+
57+
@dataclass
58+
class SourceDefinition(ParsedSourceMandatory):
59+
quoting: Quoting = field(default_factory=Quoting)
60+
loaded_at_field: Optional[str] = None
61+
freshness: Optional[FreshnessThreshold] = None
62+
external: Optional[ExternalTable] = None
63+
description: str = ""
64+
columns: Dict[str, ColumnInfo] = field(default_factory=dict)
65+
meta: Dict[str, Any] = field(default_factory=dict)
66+
source_meta: Dict[str, Any] = field(default_factory=dict)
67+
tags: List[str] = field(default_factory=list)
68+
config: SourceConfig = field(default_factory=SourceConfig)
69+
patch_path: Optional[str] = None
70+
unrendered_config: Dict[str, Any] = field(default_factory=dict)
71+
relation_name: Optional[str] = None
72+
created_at: float = field(default_factory=lambda: time.time())

core/dbt/contracts/graph/nodes.py

Lines changed: 6 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -78,7 +78,6 @@
7878
DependsOn,
7979
Docs,
8080
Exposure as ExposureResource,
81-
ExternalTable as ExternalTableResource,
8281
MacroDependsOn,
8382
MacroArgument,
8483
Documentation as DocumentationResource,
@@ -87,12 +86,11 @@
8786
NodeVersion,
8887
Group as GroupResource,
8988
GraphResource,
90-
ParsedSourceMandatory as ParsedSourceMandatoryResource,
9189
Quoting as QuotingResource,
9290
RefArgs as RefArgsResource,
9391
SavedQuery as SavedQueryResource,
9492
SemanticModel as SemanticModelResource,
95-
SourceConfig,
93+
SourceDefinition as SourceDefinitionResource,
9694
)
9795

9896
# =====================================================================
@@ -1212,22 +1210,13 @@ def tests(self) -> List[TestDef]:
12121210

12131211
@dataclass
12141212
class SourceDefinition(
1215-
NodeInfoMixin, GraphNode[GraphResource], HasRelationMetadata, ParsedSourceMandatoryResource
1213+
NodeInfoMixin,
1214+
GraphNode[SourceDefinitionResource],
1215+
SourceDefinitionResource,
1216+
HasRelationMetadata,
12161217
):
1217-
quoting: QuotingResource = field(default_factory=QuotingResource)
1218-
loaded_at_field: Optional[str] = None
1218+
# Overriding the `freshness` property to use the `FreshnessThreshold` instead of the `FreshnessThresholdResource`
12191219
freshness: Optional[FreshnessThreshold] = None
1220-
external: Optional[ExternalTableResource] = None
1221-
description: str = ""
1222-
columns: Dict[str, ColumnInfoResource] = field(default_factory=dict)
1223-
meta: Dict[str, Any] = field(default_factory=dict)
1224-
source_meta: Dict[str, Any] = field(default_factory=dict)
1225-
tags: List[str] = field(default_factory=list)
1226-
config: SourceConfig = field(default_factory=SourceConfig)
1227-
patch_path: Optional[str] = None
1228-
unrendered_config: Dict[str, Any] = field(default_factory=dict)
1229-
relation_name: Optional[str] = None
1230-
created_at: float = field(default_factory=lambda: time.time())
12311220

12321221
def __post_serialize__(self, dct):
12331222
if "_event_status" in dct:

0 commit comments

Comments
 (0)