Skip to content

Commit df27403

Browse files
authored
Add dataloader explaination to book (#518)
1 parent dea15f4 commit df27403

File tree

3 files changed

+189
-0
lines changed

3 files changed

+189
-0
lines changed

docs/book/content/SUMMARY.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,7 @@
3131
- [Non-struct objects](advanced/non_struct_objects.md)
3232
- [Objects and generics](advanced/objects_and_generics.md)
3333
- [Multiple operations per request](advanced/multiple_ops_per_request.md)
34+
- [Dataloaders](advanced/dataloaders.md)
3435

3536
# - [Context switching]
3637

Lines changed: 187 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,187 @@
1+
# Avoiding the N+1 Problem With Dataloaders
2+
3+
A common issue with graphql servers is how the resolvers query their datasource.
4+
his issue results in a large number of unneccessary database queries or http requests.
5+
Say you were wanting to list a bunch of cults people were in
6+
7+
```graphql
8+
query {
9+
persons {
10+
id
11+
name
12+
cult {
13+
id
14+
name
15+
}
16+
}
17+
}
18+
```
19+
20+
What would be executed by a SQL database would be:
21+
22+
```sql
23+
SELECT id, name, cult_id FROM persons;
24+
SELECT id, name FROM cults WHERE id = 1;
25+
SELECT id, name FROM cults WHERE id = 1;
26+
SELECT id, name FROM cults WHERE id = 1;
27+
SELECT id, name FROM cults WHERE id = 1;
28+
SELECT id, name FROM cults WHERE id = 2;
29+
SELECT id, name FROM cults WHERE id = 2;
30+
SELECT id, name FROM cults WHERE id = 2;
31+
# ...
32+
```
33+
34+
Once the list of users has been returned, a separate query is run to find the cult of each user.
35+
You can see how this could quickly become a problem.
36+
37+
A common solution to this is to introduce a **dataloader**.
38+
This can be done with Juniper using the crate [cksac/dataloader-rs](https://github.com/cksac/dataloader-rs), which has two types of dataloaders; cached and non-cached. This example will explore the non-cached option.
39+
40+
41+
### What does it look like?
42+
43+
!FILENAME Cargo.toml
44+
45+
```toml
46+
[dependencies]
47+
actix-identity = "0.2"
48+
actix-rt = "1.0"
49+
actix-web = {version = "2.0", features = []}
50+
juniper = { git = "https://github.com/graphql-rust/juniper", branch = "async-await", features = ["async"] }
51+
futures = "0.3"
52+
postgres = "0.15.2"
53+
dataloader = "0.6.0"
54+
```
55+
56+
```rust, ignore
57+
use dataloader::Loader;
58+
use dataloader::{BatchFn, BatchFuture};
59+
use futures::{future, FutureExt as _};
60+
use std::collections::HashMap;
61+
use postgres::{Connection, TlsMode};
62+
use std::env;
63+
64+
pub fn get_db_conn() -> Connection {
65+
let pg_connection_string = env::var("DATABASE_URI").expect("need a db uri");
66+
println!("Connecting to {}", pg_connection_string);
67+
let conn = Connection::connect(&pg_connection_string[..], TlsMode::None).unwrap();
68+
println!("Connection is fine");
69+
conn
70+
}
71+
72+
#[derive(Debug, Clone)]
73+
pub struct Cult {
74+
pub id: i32,
75+
pub name: String,
76+
}
77+
78+
pub fn get_cult_by_ids(hashmap: &mut HashMap<i32, Cult>, ids: Vec<i32>) {
79+
let conn = get_db_conn();
80+
for row in &conn
81+
.query("SELECT id, name FROM cults WHERE id = ANY($1)", &[&ids])
82+
.unwrap()
83+
{
84+
let cult = Cult {
85+
id: row.get(0),
86+
name: row.get(1),
87+
};
88+
hashmap.insert(cult.id, cult);
89+
}
90+
}
91+
92+
pub struct CultBatcher;
93+
94+
impl BatchFn<i32, Cult> for CultBatcher {
95+
type Error = ();
96+
97+
fn load(&self, keys: &[i32]) -> BatchFuture<Cult, Self::Error> {
98+
println!("load batch {:?}", keys);
99+
// A hashmap is used, as we need to return an array which maps each original key to a Cult.
100+
let mut cult_hashmap = HashMap::new();
101+
get_cult_by_ids(&mut cult_hashmap, keys.to_vec());
102+
103+
future::ready(keys.iter().map(|key| cult_hashmap[key].clone()).collect())
104+
.unit_error()
105+
.boxed()
106+
}
107+
}
108+
109+
pub type CultLoader = Loader<i32, Cult, (), CultBatcher>;
110+
111+
// To create a new loader
112+
pub fn get_loader() -> CultLoader {
113+
Loader::new(CultBatcher)
114+
}
115+
116+
#[juniper::graphql_object(Context = Context)]
117+
impl Cult {
118+
// your resolvers
119+
120+
// To call the dataloader
121+
pub async fn cult_by_id(ctx: &Context, id: i32) -> Cult {
122+
ctx.cult_loader.load(id).await.unwrap()
123+
}
124+
}
125+
126+
```
127+
128+
### How do I call them?
129+
130+
Once created, a dataloader has the functions `.load()` and `.load_many()`.
131+
When called these return a Future.
132+
In the above example `cult_loader.load(id: i32)` returns `Future<Cult>`. If we had used `cult_loader.load_may(Vec<i32>)` it would have returned `Future<Vec<Cult>>`.
133+
134+
135+
### Where do I create my dataloaders?
136+
137+
**Dataloaders** should be created per-request to avoid risk of bugs where one user is able to load cached/batched data from another user/ outside of its authenticated scope.
138+
Creating dataloaders within individual resolvers will prevent batching from occurring and will nullify the benefits of the dataloader.
139+
140+
For example:
141+
142+
_When you declare your context_
143+
```rust, ignore
144+
use juniper;
145+
146+
#[derive(Clone)]
147+
pub struct Context {
148+
pub cult_loader: CultLoader,
149+
}
150+
151+
impl juniper::Context for Context {}
152+
153+
impl Context {
154+
pub fn new(cult_loader: CultLoader) -> Self {
155+
Self {
156+
cult_loader
157+
}
158+
}
159+
}
160+
```
161+
162+
_Your handler for GraphQL (Note: instantiating context here keeps it per-request)_
163+
```rust, ignore
164+
pub async fn graphql(
165+
st: web::Data<Arc<Schema>>,
166+
data: web::Json<GraphQLRequest>,
167+
) -> Result<HttpResponse, Error> {
168+
let mut rt = futures::executor::LocalPool::new();
169+
170+
// Context setup
171+
let cult_loader = get_loader();
172+
let ctx = Context::new(cult_loader);
173+
174+
// Execute
175+
let future_execute = data.execute_async(&st, &ctx);
176+
let res = rt.run_until(future_execute);
177+
let json = serde_json::to_string(&res).map_err(error::ErrorInternalServerError)?;
178+
179+
Ok(HttpResponse::Ok()
180+
.content_type("application/json")
181+
.body(json))
182+
}
183+
```
184+
185+
### Further Example:
186+
187+
For a full example using Dataloaders and Context check out [jayy-lmao/rust-graphql-docker](https://github.com/jayy-lmao/rust-graphql-docker).

docs/book/content/advanced/index.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,3 +6,4 @@ The chapters below cover some more advanced scenarios.
66
- [Non-struct objects](non_struct_objects.md)
77
- [Objects and generics](objects_and_generics.md)
88
- [Multiple operations per request](multiple_ops_per_request.md)
9+
- [Dataloaders](dataloaders.md)

0 commit comments

Comments
 (0)