Skip to content

Commit 035965a

Browse files
mp911dechristophstrobl
mp911de
authored and
christophstrobl
committed
Introduce Scroll API.
See: #2151 Original Pull Request: #2787
1 parent a5de842 commit 035965a

27 files changed

+1280
-134
lines changed

Diff for: ‎src/main/asciidoc/index.adoc

+1
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22
Oliver Gierke; Thomas Darimont; Christoph Strobl; Mark Pollack; Thomas Risberg; Mark Paluch; Jay Bryant
33
:revnumber: {version}
44
:revdate: {localdate}
5+
:feature-scroll: true
56
ifdef::backend-epub3[:front-cover-image: image:epub-cover.png[Front Cover,1050,1600]]
67

78
(C) 2008-2022 The original authors.

Diff for: ‎src/main/asciidoc/repositories-paging-sorting.adoc

+136
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,136 @@
1+
[[repositories.special-parameters]]
2+
=== Special parameter handling
3+
4+
To handle parameters in your query, define method parameters as already seen in the preceding examples.
5+
Besides that, the infrastructure recognizes certain specific types like `Pageable` and `Sort`, to apply pagination and sorting to your queries dynamically.
6+
The following example demonstrates these features:
7+
8+
ifdef::feature-scroll[]
9+
.Using `Pageable`, `Slice`, `ScrollPosition`, and `Sort` in query methods
10+
====
11+
[source,java]
12+
----
13+
Page<User> findByLastname(String lastname, Pageable pageable);
14+
15+
Slice<User> findByLastname(String lastname, Pageable pageable);
16+
17+
Scroll<User> findTop10ByLastname(String lastname, ScrollPosition position, Sort sort);
18+
19+
List<User> findByLastname(String lastname, Sort sort);
20+
21+
List<User> findByLastname(String lastname, Pageable pageable);
22+
----
23+
====
24+
endif::[]
25+
26+
ifndef::feature-scroll[]
27+
.Using `Pageable`, `Slice`, and `Sort` in query methods
28+
====
29+
[source,java]
30+
----
31+
Page<User> findByLastname(String lastname, Pageable pageable);
32+
33+
Slice<User> findByLastname(String lastname, Pageable pageable);
34+
35+
List<User> findByLastname(String lastname, Sort sort);
36+
37+
List<User> findByLastname(String lastname, Pageable pageable);
38+
----
39+
====
40+
endif::[]
41+
42+
IMPORTANT: APIs taking `Sort` and `Pageable` expect non-`null` values to be handed into methods.
43+
If you do not want to apply any sorting or pagination, use `Sort.unsorted()` and `Pageable.unpaged()`.
44+
45+
The first method lets you pass an `org.springframework.data.domain.Pageable` instance to the query method to dynamically add paging to your statically defined query.
46+
A `Page` knows about the total number of elements and pages available.
47+
It does so by the infrastructure triggering a count query to calculate the overall number.
48+
As this might be expensive (depending on the store used), you can instead return a `Slice`.
49+
A `Slice` knows only about whether a next `Slice` is available, which might be sufficient when walking through a larger result set.
50+
51+
Sorting options are handled through the `Pageable` instance, too.
52+
If you need only sorting, add an `org.springframework.data.domain.Sort` parameter to your method.
53+
As you can see, returning a `List` is also possible.
54+
In this case, the additional metadata required to build the actual `Page` instance is not created (which, in turn, means that the additional count query that would have been necessary is not issued).
55+
Rather, it restricts the query to look up only the given range of entities.
56+
57+
NOTE: To find out how many pages you get for an entire query, you have to trigger an additional count query.
58+
By default, this query is derived from the query you actually trigger.
59+
60+
[[repositories.paging-and-sorting]]
61+
==== Paging and Sorting
62+
63+
You can define simple sorting expressions by using property names.
64+
You can concatenate expressions to collect multiple criteria into one expression.
65+
66+
.Defining sort expressions
67+
====
68+
[source,java]
69+
----
70+
Sort sort = Sort.by("firstname").ascending()
71+
.and(Sort.by("lastname").descending());
72+
----
73+
====
74+
75+
For a more type-safe way to define sort expressions, start with the type for which to define the sort expression and use method references to define the properties on which to sort.
76+
77+
.Defining sort expressions by using the type-safe API
78+
====
79+
[source,java]
80+
----
81+
TypedSort<Person> person = Sort.sort(Person.class);
82+
83+
Sort sort = person.by(Person::getFirstname).ascending()
84+
.and(person.by(Person::getLastname).descending());
85+
----
86+
====
87+
88+
NOTE: `TypedSort.by(…)` makes use of runtime proxies by (typically) using CGlib, which may interfere with native image compilation when using tools such as Graal VM Native.
89+
90+
If your store implementation supports Querydsl, you can also use the generated metamodel types to define sort expressions:
91+
92+
.Defining sort expressions by using the Querydsl API
93+
====
94+
[source,java]
95+
----
96+
QSort sort = QSort.by(QPerson.firstname.asc())
97+
.and(QSort.by(QPerson.lastname.desc()));
98+
----
99+
====
100+
101+
ifdef::feature-scroll[]
102+
include::repositories-scrolling.adoc[]
103+
endif::[]
104+
105+
[[repositories.limit-query-result]]
106+
=== Limiting Query Results
107+
108+
You can limit the results of query methods by using the `first` or `top` keywords, which you can use interchangeably.
109+
You can append an optional numeric value to `top` or `first` to specify the maximum result size to be returned.
110+
If the number is left out, a result size of 1 is assumed.
111+
The following example shows how to limit the query size:
112+
113+
.Limiting the result size of a query with `Top` and `First`
114+
====
115+
[source,java]
116+
----
117+
User findFirstByOrderByLastnameAsc();
118+
119+
User findTopByOrderByAgeDesc();
120+
121+
Page<User> queryFirst10ByLastname(String lastname, Pageable pageable);
122+
123+
Slice<User> findTop3ByLastname(String lastname, Pageable pageable);
124+
125+
List<User> findFirst10ByLastname(String lastname, Sort sort);
126+
127+
List<User> findTop10ByLastname(String lastname, Pageable pageable);
128+
----
129+
====
130+
131+
The limiting expressions also support the `Distinct` keyword for datastores that support distinct queries.
132+
Also, for the queries that limit the result set to one instance, wrapping the result into with the `Optional` keyword is supported.
133+
134+
If pagination or slicing is applied to a limiting query pagination (and the calculation of the number of available pages), it is applied within the limited result.
135+
136+
NOTE: Limiting the results in combination with dynamic sorting by using a `Sort` parameter lets you express query methods for the 'K' smallest as well as for the 'K' biggest elements.

Diff for: ‎src/main/asciidoc/repositories-scrolling.adoc

+83
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,83 @@
1+
[[repositories.scrolling]]
2+
==== Scrolling
3+
4+
Scrolling is a more fine-grained approach to iterate through larger results set chunks.
5+
Scrolling consists of a stable sort, a scroll type (Offset- or Keyset-based scrolling) and result limiting.
6+
You can define simple sorting expressions by using property names and define static result limiting using the <<repositories.limit-query-result,`Top` or `First` keyword>> through query derivation.
7+
You can concatenate expressions to collect multiple criteria into one expression.
8+
9+
Scroll queries return a `Scroll<T>` that allows obtaining the scroll position to resume to obtain the next `Scroll<T>` until your application has consumed the entire query result.
10+
Similar to consuming a Java `Iterator<List<…>>` by obtaining the next batch of results, query result scrolling lets you access the next `ScrollPosition` through `Scroll.lastScrollPosition()`.
11+
12+
[[repositories.scrolling.offset]]
13+
===== Scrolling using Offset
14+
15+
Offset scrolling uses similar to pagination, an Offset counter to skip a number of results and let the data source only return results beginning at the given Offset.
16+
This simple mechanism avoids large results being sent to the client application.
17+
However, most databases require materializing the full query result before your server can return the results.
18+
19+
.Using `OffsetScrollPosition` with Repository Query Methods
20+
====
21+
[source,java]
22+
----
23+
interface UserRepository extends Repository<User, Long> {
24+
25+
Scroll<User> findFirst10ByLastnameOrderByFirstname(String lastname, OffsetScrollPosition position);
26+
}
27+
28+
Scroll<User> users = repository.findFirst10ByLastnameOrderByFirstname("Doe", OffsetScrollPosition.initial());
29+
30+
do {
31+
32+
for (User u : users) {
33+
// consume the user
34+
}
35+
36+
// obtain the next Scroll
37+
users = repository.findFirst10ByLastnameOrderByFirstname("Doe", users.lastScrollPosition());
38+
} while (!users.isEmpty() && !users.isLast());
39+
----
40+
====
41+
42+
[[repositories.scrolling.keyset]]
43+
===== Scrolling using Keyset-Filtering
44+
45+
Offset-based requires most databases require materializing the entire result before your server can return the results.
46+
So while the client only sees the portion of the requested results, your server needs to build the full result, which causes additional load.
47+
48+
Keyset-Filtering approaches result subset retrieval by leveraging built-in capabilities of your database aiming to reduce the computation and I/O requirements for individual queries.
49+
This approach maintains a set of keys to resume scrolling by passing keys into the query, effectively amending your filter criteria.
50+
51+
The core idea of Keyset-Filtering is to start retrieving results using a stable sorting order.
52+
Once you want to scroll to the next chunk, you obtain a `ScrollPosition` that is used to reconstruct the position within the sorted result.
53+
The `ScrollPosition` captures the keyset of the last entity within the current `Scroll`.
54+
To run the query, reconstruction rewrites the criteria clause to include all sort fields and the primary key so that the database can leverage potential indexes to run the query.
55+
The database needs only constructing a much smaller result from the given keyset position without the need to fully materialize a large result and then skipping results until reaching a particular offset.
56+
57+
.Using `KeysetScrollPosition` with Repository Query Methods
58+
====
59+
[source,java]
60+
----
61+
interface UserRepository extends Repository<User, Long> {
62+
63+
Scroll<User> findFirst10ByLastnameOrderByFirstname(String lastname, KeysetScrollPosition position);
64+
}
65+
66+
Scroll<User> users = repository.findFirst10ByLastnameOrderByFirstname("Doe", KeysetScrollPosition.initial());
67+
68+
do {
69+
70+
for (User u : users) {
71+
// consume the user
72+
}
73+
74+
// obtain the next Scroll
75+
users = repository.findFirst10ByLastnameOrderByFirstname("Doe", users.lastScrollPosition());
76+
} while (!users.isEmpty() && !users.isLast());
77+
----
78+
====
79+
80+
Keyset-Filtering works best when your database contains an index that matches the sort fields, hence a static sort works well.
81+
Scroll queries applying Keyset-Filtering require to the properties used in the sort order to be returned by the query, and these must be mapped in the returned entity.
82+
83+
You can use interface and DTO projections, however make sure to include all properties that you've sorted by to avoid keyset extraction failures.

Diff for: ‎src/main/asciidoc/repositories.adoc

+5-112
Original file line numberDiff line numberDiff line change
@@ -92,6 +92,10 @@ Page<User> users = repository.findAll(PageRequest.of(1, 20));
9292
----
9393
====
9494

95+
ifdef::feature-scroll[]
96+
In addition to pagination, scrolling provides a more fine-grained access to iterate through chunks of larger result sets.
97+
endif::[]
98+
9599
In addition to query methods, query derivation for both count and delete queries is available.
96100
The following list shows the interface definition for a derived count query:
97101

@@ -517,118 +521,7 @@ List<Person> findByAddress_ZipCode(ZipCode zipCode);
517521

518522
Because we treat the underscore character as a reserved character, we strongly advise following standard Java naming conventions (that is, not using underscores in property names but using camel case instead).
519523

520-
[[repositories.special-parameters]]
521-
=== Special parameter handling
522-
523-
To handle parameters in your query, define method parameters as already seen in the preceding examples.
524-
Besides that, the infrastructure recognizes certain specific types like `Pageable` and `Sort`, to apply pagination and sorting to your queries dynamically.
525-
The following example demonstrates these features:
526-
527-
.Using `Pageable`, `Slice`, and `Sort` in query methods
528-
====
529-
[source,java]
530-
----
531-
Page<User> findByLastname(String lastname, Pageable pageable);
532-
533-
Slice<User> findByLastname(String lastname, Pageable pageable);
534-
535-
List<User> findByLastname(String lastname, Sort sort);
536-
537-
List<User> findByLastname(String lastname, Pageable pageable);
538-
----
539-
====
540-
541-
IMPORTANT: APIs taking `Sort` and `Pageable` expect non-`null` values to be handed into methods.
542-
If you do not want to apply any sorting or pagination, use `Sort.unsorted()` and `Pageable.unpaged()`.
543-
544-
The first method lets you pass an `org.springframework.data.domain.Pageable` instance to the query method to dynamically add paging to your statically defined query.
545-
A `Page` knows about the total number of elements and pages available.
546-
It does so by the infrastructure triggering a count query to calculate the overall number.
547-
As this might be expensive (depending on the store used), you can instead return a `Slice`.
548-
A `Slice` knows only about whether a next `Slice` is available, which might be sufficient when walking through a larger result set.
549-
550-
Sorting options are handled through the `Pageable` instance, too.
551-
If you need only sorting, add an `org.springframework.data.domain.Sort` parameter to your method.
552-
As you can see, returning a `List` is also possible.
553-
In this case, the additional metadata required to build the actual `Page` instance is not created (which, in turn, means that the additional count query that would have been necessary is not issued).
554-
Rather, it restricts the query to look up only the given range of entities.
555-
556-
NOTE: To find out how many pages you get for an entire query, you have to trigger an additional count query.
557-
By default, this query is derived from the query you actually trigger.
558-
559-
[[repositories.paging-and-sorting]]
560-
==== Paging and Sorting
561-
562-
You can define simple sorting expressions by using property names.
563-
You can concatenate expressions to collect multiple criteria into one expression.
564-
565-
.Defining sort expressions
566-
====
567-
[source,java]
568-
----
569-
Sort sort = Sort.by("firstname").ascending()
570-
.and(Sort.by("lastname").descending());
571-
----
572-
====
573-
574-
For a more type-safe way to define sort expressions, start with the type for which to define the sort expression and use method references to define the properties on which to sort.
575-
576-
.Defining sort expressions by using the type-safe API
577-
====
578-
[source,java]
579-
----
580-
TypedSort<Person> person = Sort.sort(Person.class);
581-
582-
Sort sort = person.by(Person::getFirstname).ascending()
583-
.and(person.by(Person::getLastname).descending());
584-
----
585-
====
586-
587-
NOTE: `TypedSort.by(…)` makes use of runtime proxies by (typically) using CGlib, which may interfere with native image compilation when using tools such as Graal VM Native.
588-
589-
If your store implementation supports Querydsl, you can also use the generated metamodel types to define sort expressions:
590-
591-
.Defining sort expressions by using the Querydsl API
592-
====
593-
[source,java]
594-
----
595-
QSort sort = QSort.by(QPerson.firstname.asc())
596-
.and(QSort.by(QPerson.lastname.desc()));
597-
----
598-
====
599-
600-
[[repositories.limit-query-result]]
601-
=== Limiting Query Results
602-
603-
You can limit the results of query methods by using the `first` or `top` keywords, which you can use interchangeably.
604-
You can append an optional numeric value to `top` or `first` to specify the maximum result size to be returned.
605-
If the number is left out, a result size of 1 is assumed.
606-
The following example shows how to limit the query size:
607-
608-
.Limiting the result size of a query with `Top` and `First`
609-
====
610-
[source,java]
611-
----
612-
User findFirstByOrderByLastnameAsc();
613-
614-
User findTopByOrderByAgeDesc();
615-
616-
Page<User> queryFirst10ByLastname(String lastname, Pageable pageable);
617-
618-
Slice<User> findTop3ByLastname(String lastname, Pageable pageable);
619-
620-
List<User> findFirst10ByLastname(String lastname, Sort sort);
621-
622-
List<User> findTop10ByLastname(String lastname, Pageable pageable);
623-
----
624-
====
625-
626-
The limiting expressions also support the `Distinct` keyword for datastores that support distinct queries.
627-
Also, for the queries that limit the result set to one instance, wrapping the result into with the `Optional` keyword is supported.
628-
629-
If pagination or slicing is applied to a limiting query pagination (and the calculation of the number of available pages), it is applied within the limited result.
630-
631-
NOTE: Limiting the results in combination with dynamic sorting by using a `Sort` parameter lets you express query methods for the 'K' smallest as well as for the 'K' biggest elements.
524+
include::repositories-paging-sorting.adoc[]
632525

633526
[[repositories.collections-and-iterables]]
634527
=== Repository Methods Returning Collections or Iterables

0 commit comments

Comments
 (0)