Skip to content

Added support for bulk/batch requests (atomic:operations) #930

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 126 commits into from
Feb 10, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
126 commits
Select commit Hold shift + click to select a range
da055ae
Resurrected original operations code
Nov 23, 2020
91d4429
Getting the code to build and tests to succeed.
Nov 23, 2020
f21e317
Removed option to enable the atomic-operations feature. It is implici…
Nov 24, 2020
6d8aa36
Renames and namespace moves
Nov 24, 2020
88f5eee
Change patch to post
Nov 25, 2020
0d616c6
Separate controller layers, align naming in deserializer, move reques…
Nov 25, 2020
4dc7e4a
Updated json document structure, based on atomic:operations spec
Nov 25, 2020
ee52719
Simplified processor resolver
Nov 25, 2020
fc00620
Converted atomic:operations Add tests
Nov 26, 2020
9d534e4
Converted atomic:operations Remove tests
Nov 26, 2020
de2c3d3
Converted atomic:operations Update tests
Nov 26, 2020
07cb792
Converted rollback test and removed separate projects
Nov 26, 2020
f191bc0
Rename plus overridable ToString
Nov 26, 2020
8fcedb6
Added local ID tracking
Nov 26, 2020
b4be29c
Added support for atomic:operations extension in ContentType header
Nov 27, 2020
9b6e6b8
Added support for atomic:operations extension in Accept headers
Nov 27, 2020
7b5a0d2
Moved existing local-id tests
Nov 30, 2020
a7c06c6
Improvements on local ID usage (see tests).
Nov 30, 2020
31c3efa
Block query string parameters on atomic:operations request
Dec 1, 2020
21c4d30
Fixed broken tests
Dec 1, 2020
aa5e454
Return NoContent when data for all operations is null
Dec 1, 2020
01e113c
Corrected example
Dec 1, 2020
6b74ded
Fail on local ID usage outside of operations
Dec 1, 2020
e0f6482
Added ModelState validation
Dec 1, 2020
0aafb84
Fixes in handling add-to and remove-from to-many relationships
Dec 2, 2020
4d4e5d7
Run model state validation for all operations up-front
Dec 2, 2020
e5237df
Fix broken test after rebase
Dec 15, 2020
56b8a84
Added tests for delete resource
Dec 15, 2020
cd51071
File rename
Dec 16, 2020
2efc6f9
added tests for Remove from to-many
Dec 16, 2020
ce0cd4c
added tests for Add to to-many
Dec 17, 2020
204b320
added tests for Replace to-many
Dec 17, 2020
36c4023
added tests for Replace to-one
Dec 17, 2020
f72d7b2
reorder tests
Dec 17, 2020
4be8af6
fail on href usage
Dec 17, 2020
87d2bda
Fail when using both 'id' and 'lid'
Dec 17, 2020
fca7192
Tests for create resource
Dec 18, 2020
0dfb323
Pushed down all operation serialization/deserialization into the Seri…
Jan 5, 2021
175215c
Fixed skipped tests
Jan 7, 2021
de2acf2
Added test for read-only attribute
Jan 7, 2021
626d58b
Added operation tests for Create resource with client-generated ID
Jan 7, 2021
f4a3447
Reordered tests
Jan 7, 2021
0b824e3
Added operation tests for Create resource with ToOne relationship
Jan 7, 2021
6e1b9d5
Added operation tests for Create resource with ToMany relationship
Jan 7, 2021
a090abb
Added tests for unknown ID in ref
Jan 7, 2021
fc63438
Reordered tests
Jan 7, 2021
5167fdc
Fixed: exclude attributes without AttrCapabilities.AllowView flag in …
Jan 12, 2021
10b8c1b
Added operation tests for Update resource
Jan 13, 2021
7bbe3aa
Cleanup tests
Jan 13, 2021
5ce2218
Added tests for error cases
Jan 13, 2021
1b00d34
Fixed: Both id and lid in create resource operation should not be all…
Jan 13, 2021
6dc7e43
Added option: MaximumOperationsPerRequest
Jan 13, 2021
de58379
Added operation tests for Update resource with to-one/to-many relatio…
Jan 14, 2021
b43fe8e
Fixed links rendering in operation responses
Jan 15, 2021
cf01dba
Separated transaction support from operations processing.
Jan 15, 2021
82d3dee
Automatic routing setup for operations controller
Jan 18, 2021
e6baf42
Fixed: top-level meta in atomic:operation responses
Jan 18, 2021
1d6065b
Fixed: resource-level meta in atomic:operation responses
Jan 19, 2021
79af8f1
Renames
Jan 19, 2021
bb7c7c7
Fixed: duplicate calls to ResourceDefinition.GetMeta
Jan 19, 2021
24a4ccc
Fixes resource definition callbacks for sparse fieldsets
Jan 19, 2021
25ba724
Fixed broken build after rebase master
Jan 20, 2021
ca14023
Reduce diff with master branch
Jan 20, 2021
4291a90
Renames
Jan 20, 2021
a5f191e
Simplified processors
Jan 20, 2021
b95cc88
Removed dependency on GenericServiceFactory
Jan 20, 2021
9119575
Corrected error message
Jan 20, 2021
b8fec24
API review
Jan 20, 2021
06b3604
More API review
Jan 21, 2021
aa8084c
Cleanup RequestDeserializer
Jan 21, 2021
7954882
Small name tweaks
Jan 22, 2021
3f3667c
API shape tweaks
Jan 22, 2021
42c1ede
Validate local ID usage upfront
Jan 22, 2021
f451fc7
Added documentation
Jan 22, 2021
f91b25b
Small tweaks
Jan 23, 2021
355083c
Refactored tests for HttpReadOnly/NoHttpPost/Patch/Delete controller …
Jan 26, 2021
90274c6
Refactored tests for casing convention
Jan 26, 2021
4bdb3e8
Refactored tests for custom exception handler
Jan 26, 2021
cbe01c2
Refactored tests for nulls/default query string parameters; removed s…
Jan 26, 2021
0c5d5e8
Refactored tests for DisableQueryStringAttribute
Jan 27, 2021
7057c89
Refactored tests for resource injection
Jan 28, 2021
8756f25
Fixed assertions on DateTime/DateTimeOffset in tests
Jan 28, 2021
44653f8
Refactored tests for non-json:api controllers
Jan 28, 2021
22738b6
Refactored tests for ActionResult usage
Jan 28, 2021
e12d27f
Refactored tests for custom routes
Feb 2, 2021
ddcd4ac
Refactored tests for links rendering
Feb 2, 2021
8b443d8
Refactored tests for exception handling in serializer
Feb 2, 2021
64e09db
Refactored tests for serialization
Feb 2, 2021
8bf0989
Refactored tests for resource hooks
Feb 2, 2021
255ed36
General cleanup
Feb 3, 2021
c7000b8
Extracted test building blocks in shared project
Feb 4, 2021
db9778e
Auto-adjust namespaces
Feb 4, 2021
d9a2ac7
Refactored tests for service discovery
Feb 4, 2021
53f8c82
Refactored tests for no EF Core
Feb 4, 2021
86bac20
Renamed IntegrationTestContext to ExampleIntegrationTestContext becau…
Feb 4, 2021
d94235d
Adjusted test names for ModelStateValidationTests
Feb 4, 2021
c8a6012
Adjusted test names for pagination
Feb 4, 2021
cb983b9
Adjusted test names for meta
Feb 4, 2021
5ad5c76
Fixed broken tests; simpler bootstrap setup
Feb 4, 2021
838a158
Adjusted test names for hooks
Feb 4, 2021
d3ff09a
Enable concurrent testruns (makes running all tests 60% faster)
Feb 4, 2021
64c5b4a
Removed unused using
Feb 4, 2021
e2a2bb0
Merge branch 'master' into refactor-tests
Feb 4, 2021
bcb7320
Cleanup tests for hooks
Feb 4, 2021
60ac856
Cleanup example models
Feb 4, 2021
de40fb7
Revert "Enable concurrent testruns (makes running all tests 60% faste…
Feb 4, 2021
bd11b52
fixed failing testrunner
Feb 4, 2021
b0ac024
Enable running integration tests in parallel
Feb 4, 2021
f343d08
test
Feb 4, 2021
38c127a
Disable duplicate builds
Feb 4, 2021
9ee3fc5
Revert "Disable duplicate builds"
Feb 4, 2021
59fccd8
Addressed review feedback
Feb 5, 2021
21b94c5
Moved integration tests for filter/include/page/sort/fields into Quer…
Feb 5, 2021
a044cf0
Move types into separate files
Feb 5, 2021
78bc0c3
Moved unit tests for query string parameters into ExampleTests projec…
Feb 5, 2021
c2da909
Refactored query string integration tests to use private models. Chan…
Feb 5, 2021
e594391
Refactored query string unit tests to use private models.
Feb 5, 2021
f2fd733
Cleanup models in example project
Feb 6, 2021
f97b81f
Addressed review feedback
Feb 8, 2021
7c7fbe4
Added option to set transaction isolation level
Feb 8, 2021
2516d2e
Rename
Feb 8, 2021
114dc07
Refactored tests for logging
Feb 8, 2021
37487cf
Refactored tests for meta
Feb 8, 2021
dd9f1fb
Merge branch 'refactor-tests' into atomic-operations-merge
Feb 8, 2021
d9df27d
Post-merge fixes
Feb 8, 2021
2d3771d
Merge branch 'master' into atomic-operations-merge
Feb 9, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion benchmarks/Serialization/JsonApiDeserializerBenchmarks.cs
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ public JsonApiDeserializerBenchmarks()
IResourceGraph resourceGraph = DependencyFactory.CreateResourceGraph(options);
var targetedFields = new TargetedFields();
var request = new JsonApiRequest();
_jsonApiDeserializer = new RequestDeserializer(resourceGraph, new ResourceFactory(new ServiceContainer()), targetedFields, new HttpContextAccessor(), request);
_jsonApiDeserializer = new RequestDeserializer(resourceGraph, new ResourceFactory(new ServiceContainer()), targetedFields, new HttpContextAccessor(), request, options);
}

[Benchmark]
Expand Down
1 change: 1 addition & 0 deletions docs/usage/toc.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
## [Creating](writing/creating.md)
## [Updating](writing/updating.md)
## [Deleting](writing/deleting.md)
## [Bulk/batch](writing/bulk-batch-operations.md)

# [Resource Graph](resource-graph.md)
# [Options](options.md)
Expand Down
116 changes: 116 additions & 0 deletions docs/usage/writing/bulk-batch-operations.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,116 @@
# Bulk/batch

_since v4.1_

The [Atomic Operations](https://jsonapi.org/ext/atomic/) JSON:API extension defines
how to perform multiple write operations in a linear and atomic manner.

Clients can send an array of operations in a single request. JsonApiDotNetCore guarantees that those
operations will be processed in order and will either completely succeed or fail together.

On failure, the zero-based index of the failing operation is returned in the `error.source.pointer` field of the error response.

## Usage

To enable operations, add a controller to your project that inherits from `JsonApiOperationsController` or `BaseJsonApiOperationsController`:
```c#
public sealed class OperationsController : JsonApiOperationsController
{
public OperationsController(IJsonApiOptions options, ILoggerFactory loggerFactory,
IOperationsProcessor processor, IJsonApiRequest request,
ITargetedFields targetedFields)
: base(options, loggerFactory, processor, request, targetedFields)
{
}
}
```

You'll need to send the next Content-Type in a POST request for operations:
```
application/vnd.api+json; ext="https://jsonapi.org/ext/atomic"
```

### Local IDs

Local IDs (lid) can be used to associate resources that have not yet been assigned an ID.
The next example creates two resources and sets a relationship between them:

```json
POST http://localhost/api/operations HTTP/1.1
Content-Type: application/vnd.api+json;ext="https://jsonapi.org/ext/atomic"

{
"atomic:operations": [
{
"op": "add",
"data": {
"type": "musicTracks",
"lid": "id-for-i-will-survive",
"attributes": {
"title": "I will survive"
}
}
},
{
"op": "add",
"data": {
"type": "performers",
"lid": "id-for-gloria-gaynor",
"attributes": {
"artistName": "Gloria Gaynor"
}
}
},
{
"op": "update",
"ref": {
"type": "musicTracks",
"lid": "id-for-i-will-survive",
"relationship": "performers"
},
"data": [
{
"type": "performers",
"lid": "id-for-gloria-gaynor"
}
]
}
]
}
```

For example requests, see our suite of tests in JsonApiDotNetCoreExampleTests.IntegrationTests.AtomicOperations.

## Configuration

The maximum number of operations per request defaults to 10, which you can change from Startup.cs:
```c#
services.AddJsonApi(options => options.MaximumOperationsPerRequest = 250);
```
Or, if you want to allow unconstrained, set it to `null` instead.

### Multiple controllers

You can register multiple operations controllers using custom routes, for example:
```c#
[DisableRoutingConvention, Route("/operations/musicTracks/create")]
public sealed class CreateMusicTrackOperationsController : JsonApiOperationsController
{
public override async Task<IActionResult> PostOperationsAsync(
IList<OperationContainer> operations, CancellationToken cancellationToken)
{
AssertOnlyCreatingMusicTracks(operations);

return await base.PostOperationsAsync(operations, cancellationToken);
}
}
```

## Limitations

For our atomic:operations implementation, the next limitations apply:

- The `ref.href` field cannot be used. Use type/id or type/lid instead.
- You cannot both assign and reference the same local ID in a single operation.
- All repositories used in an operations request must implement `IRepositorySupportsTransaction` and participate in the same transaction.
- If you're not using Entity Framework Core, you'll need to implement and register `IOperationsTransactionFactory` yourself.
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
using JsonApiDotNetCore.AtomicOperations;
using JsonApiDotNetCore.Configuration;
using JsonApiDotNetCore.Controllers;
using JsonApiDotNetCore.Middleware;
using JsonApiDotNetCore.Resources;
using Microsoft.Extensions.Logging;

namespace JsonApiDotNetCoreExample.Controllers
{
public sealed class OperationsController : JsonApiOperationsController
{
public OperationsController(IJsonApiOptions options, ILoggerFactory loggerFactory,
IOperationsProcessor processor, IJsonApiRequest request, ITargetedFields targetedFields)
: base(options, loggerFactory, processor, request, targetedFields)
{
}
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
using System;
using System.Threading;
using System.Threading.Tasks;
using JsonApiDotNetCore.Repositories;
using Microsoft.EntityFrameworkCore;
using Microsoft.EntityFrameworkCore.Storage;

namespace JsonApiDotNetCore.AtomicOperations
{
/// <summary>
/// Represents an Entity Framework Core transaction in an atomic:operations request.
/// </summary>
public sealed class EntityFrameworkCoreTransaction : IOperationsTransaction
{
private readonly IDbContextTransaction _transaction;
private readonly DbContext _dbContext;

/// <inheritdoc />
public Guid TransactionId => _transaction.TransactionId;

public EntityFrameworkCoreTransaction(IDbContextTransaction transaction, DbContext dbContext)
{
_transaction = transaction ?? throw new ArgumentNullException(nameof(transaction));
_dbContext = dbContext ?? throw new ArgumentNullException(nameof(dbContext));
}

/// <summary>
/// Detaches all entities from the Entity Framework Core change tracker.
/// </summary>
public Task BeforeProcessOperationAsync(CancellationToken cancellationToken)
{
_dbContext.ResetChangeTracker();
return Task.CompletedTask;
}

/// <summary>
/// Does nothing.
/// </summary>
public Task AfterProcessOperationAsync(CancellationToken cancellationToken)
{
return Task.CompletedTask;
}

/// <inheritdoc />
public Task CommitAsync(CancellationToken cancellationToken)
{
return _transaction.CommitAsync(cancellationToken);
}

/// <inheritdoc />
public ValueTask DisposeAsync()
{
return _transaction.DisposeAsync();
}
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
using System;
using System.Threading;
using System.Threading.Tasks;
using JsonApiDotNetCore.Configuration;
using JsonApiDotNetCore.Repositories;
using Microsoft.EntityFrameworkCore;

namespace JsonApiDotNetCore.AtomicOperations
{
/// <summary>
/// Provides transaction support for atomic:operation requests using Entity Framework Core.
/// </summary>
public sealed class EntityFrameworkCoreTransactionFactory : IOperationsTransactionFactory
{
private readonly IDbContextResolver _dbContextResolver;
private readonly IJsonApiOptions _options;

public EntityFrameworkCoreTransactionFactory(IDbContextResolver dbContextResolver, IJsonApiOptions options)
{
_dbContextResolver = dbContextResolver ?? throw new ArgumentNullException(nameof(dbContextResolver));
_options = options ?? throw new ArgumentNullException(nameof(options));
}

/// <inheritdoc />
public async Task<IOperationsTransaction> BeginTransactionAsync(CancellationToken cancellationToken)
{
var dbContext = _dbContextResolver.GetContext();

var transaction = _options.TransactionIsolationLevel != null
? await dbContext.Database.BeginTransactionAsync(_options.TransactionIsolationLevel.Value,
cancellationToken)
: await dbContext.Database.BeginTransactionAsync(cancellationToken);

return new EntityFrameworkCoreTransaction(transaction, dbContext);
}
}
}
28 changes: 28 additions & 0 deletions src/JsonApiDotNetCore/AtomicOperations/ILocalIdTracker.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
namespace JsonApiDotNetCore.AtomicOperations
{
/// <summary>
/// Used to track declarations, assignments and references to local IDs an in atomic:operations request.
/// </summary>
public interface ILocalIdTracker
{
/// <summary>
/// Removes all declared and assigned values.
/// </summary>
void Reset();

/// <summary>
/// Declares a local ID without assigning a server-generated value.
/// </summary>
void Declare(string localId, string resourceType);

/// <summary>
/// Assigns a server-generated ID value to a previously declared local ID.
/// </summary>
void Assign(string localId, string resourceType, string stringId);

/// <summary>
/// Gets the server-assigned ID for the specified local ID.
/// </summary>
string GetValue(string localId, string resourceType);
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
using System.Threading;
using System.Threading.Tasks;
using JsonApiDotNetCore.AtomicOperations.Processors;
using JsonApiDotNetCore.Resources;

namespace JsonApiDotNetCore.AtomicOperations
{
/// <summary>
/// Retrieves a <see cref="IOperationProcessor"/> instance from the D/I container and invokes a method on it.
/// </summary>
public interface IOperationProcessorAccessor
{
/// <summary>
/// Invokes <see cref="IOperationProcessor.ProcessAsync"/> on a processor compatible with the operation kind.
/// </summary>
Task<OperationContainer> ProcessAsync(OperationContainer operation, CancellationToken cancellationToken);
}
}
18 changes: 18 additions & 0 deletions src/JsonApiDotNetCore/AtomicOperations/IOperationsProcessor.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
using System.Collections.Generic;
using System.Threading;
using System.Threading.Tasks;
using JsonApiDotNetCore.Resources;

namespace JsonApiDotNetCore.AtomicOperations
{
/// <summary>
/// Atomically processes a request that contains a list of operations.
/// </summary>
public interface IOperationsProcessor
{
/// <summary>
/// Processes the list of specified operations.
/// </summary>
Task<IList<OperationContainer>> ProcessAsync(IList<OperationContainer> operations, CancellationToken cancellationToken);
}
}
32 changes: 32 additions & 0 deletions src/JsonApiDotNetCore/AtomicOperations/IOperationsTransaction.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
using System;
using System.Threading;
using System.Threading.Tasks;

namespace JsonApiDotNetCore.AtomicOperations
{
/// <summary>
/// Represents the overarching transaction in an atomic:operations request.
/// </summary>
public interface IOperationsTransaction : IAsyncDisposable
{
/// <summary>
/// Identifies the active transaction.
/// </summary>
Guid TransactionId { get; }

/// <summary>
/// Enables to execute custom logic before processing of an operation starts.
/// </summary>
Task BeforeProcessOperationAsync(CancellationToken cancellationToken);

/// <summary>
/// Enables to execute custom logic after processing of an operation succeeds.
/// </summary>
Task AfterProcessOperationAsync(CancellationToken cancellationToken);

/// <summary>
/// Commits all changes made to the underlying data store.
/// </summary>
Task CommitAsync(CancellationToken cancellationToken);
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
using System.Threading;
using System.Threading.Tasks;

namespace JsonApiDotNetCore.AtomicOperations
{
/// <summary>
/// Provides a method to start the overarching transaction for an atomic:operations request.
/// </summary>
public interface IOperationsTransactionFactory
{
/// <summary>
/// Starts a new transaction.
/// </summary>
Task<IOperationsTransaction> BeginTransactionAsync(CancellationToken cancellationToken);
}
}
Loading