Support for NoSql datastores (Redis)

[mikesaurus]

Expected Behavior
Some infrastructures do not include SQL-based data stores and instead make extensive use of NoSql data stores such as Redis, making the introduction of a SQL data store to support OAuth capabilities incredibly burdensome. spring-authorization-server data objects that are persisted as part of the process flow (such as OAuth2Authorization) should be easily (de)serializable using something like a JSON mapper in order to allow for either Spring provided/packaged or app developer created Redis implementations of OAuth2AuthorizationConsentService and OAuth2AuthorizationService.

Current Behavior
Serializing/deserializing spring-auth-server data objects to/from JSON requires custom data mapping that can create a maintenance nightmare in terms spring-auth-server compatibility (especially in the early/pre-v1.x stages of the spring-auth-server project).

Context
We currently use Redis as the backing data store for our customized spring-security-oauth2-based authorization server. We would like to upgrade to a spring-authorization-server-based authorization server without requiring a SQL database and all of the maintenance and operational overhead associated with it, since all of our current infrastructure and operations are geared toward Redis. We can try to work around the (de)serialization issues with custom data mapping, but custom data mapping tightly couples our code to the underlying data object implementations and concerns us in terms of maintenance/compatibility with spring-authorization-server upgrades.

[mikesaurus]

One option would be to include a JsonMapper<T> in spring-authorization-server that encapsulates the OAuth2 Object<->JSON mapping so that the actual data objects don't have to change, but anyone wishing to implement a document-based persistence strategy can easily transform the objects to/from JSON without requiring the internal data mapping to be implemented outside of spring-auth-server.

[bjornharvold]

Agree.

I started creating our own MongoOAuth2AuthorizationService and MongoOAuth2AuthorizationConsentService. Serializing / Deserializing to MongoDb is a nightmare. I tried to look at the JDBC version but didn't provide much guidance in terms of an easy way to port.

Attaching what I've done so far. The MongoRegisteredClientRepository seems OK but currently chose to use the in-memory implementations of OAuth2AuthorizationService and OAuth2AuthorizationConsentService.

If someone can provide some guidance, I'd be more than happy to contribute the MongoDb version of these interfaces.

Archive.zip
.

[skhokhar73]

I have implemented MongoDb version of this already for my personal project. But that's not very well structured right now. I am happy to discuss, plan and contribute.

[sjohnr]

@bjornharvold,

Attaching what I've done so far.

The MongoRegisteredClientRepository does indeed seem like a good start! For more examples of how to support another data store (in this case JPA), please see the following examples:

• JpaRegisteredClientRepository
• JpaOAuth2AuthorizationService
• JpaOAuth2AuthorizationConsentService

In particular, notice how the the JpaOAuth2AuthorizationService implementation re-uses the Jackson module from this project to serialize the same fields into JSON as the JdbcOAuth2AuthorizationService does, therefore keeping those details in sync with this project. This is the recommended approach at the moment.

@mikesaurus,

One option would be to include a JsonMapper<T> in spring-authorization-server that encapsulates the OAuth2 Object<->JSON mapping so that the actual data objects don't have to change, but anyone wishing to implement a document-based persistence strategy can easily transform the objects to/from JSON without requiring the internal data mapping to be implemented outside of spring-auth-server.

Thanks for bringing this up! I agree that having something like this would be incredibly convenient. While there are a number of options for how you might support each datastore, we can potentially make mapping to/from JSON easier.

However, I see some challenges with this as well, so we might need to discuss before trying anything in the project. I'm assuming this JsonMapper would be a custom interface, not the Jackson com.fasterxml.jackson.databind.json.JsonMapper one? Would the T type parameter in your case just be OAuth2Authorization, or are you hoping to provide support for many/all domain objects in the project? Can you describe some concrete use case(s) for the JsonMapper?

[bjornharvold]

Just looked at the new Jdbc code. This has changed drastically since last time I wrote my MongoDb implementations. Will use these as a base. Can share code when done.

[mikesaurus]

@sjohnr -
Apologies. I was just throwing an idea out there in my comment and wasn't trying to be specific. Saying JsonMapper<T> was a poor choice of words. My thought was for the Spring project to provide a custom JSON mapping interface along with Spring OAuth class specific implementations for the key OAuth objects that are being persisted by the Spring Auth Server library. I haven't really thought through the implications from a Spring lib perspective. This is just how I was thinking about building it within my code base. I just feel like custom data mapping like this is so tightly coupled to the OAuth class definitions that it belongs in the Spring project with those class definitions. That way changes to the OAuth objects would come with corresponding mapping (and test) updates. This would not only make JSON mapping for Spring Auth Server users easier, but also provide a consistent JSON object structure for persistence.

I think that the OAuth2Authorization class is a good concrete example of where the JSON mapper would be useful. We use Redis on our back-end. I don't have a problem with the Spring project not providing Redis support (you can't support every back end data store) and implementing the Spring Auth Server data server interfaces is reasonable. However, OAuth2Authorization can't be straight mapped to/from JSON using something like a Jackson ObjectMapper. The only option currently available to someone that wants to store a JSON representation of OAuth2Authorization is to implement their own custom data mapping. A Spring OAuth JSON mapper for OAuth2Authorization would provide an "OOTB" mechanism to transform an object to/from JSON and persist.

[sjohnr]

Hey @mikesaurus, thanks for the reply. No problem on the wording, I was also just using your example as an example, so I think we're on the same page. 😄 I also agree that focusing on the OAuth2Authorization could be a good start, and we can sort out the details of what the interface for mapping it looks like closer to implementation. Questions/thoughts:

• If the project can provide the user with the ability to map the entire object to/from JSON, is that enough to implement OAuth2AuthorizationService on top of Redis / MongoDB?
  • For example, can you perform a search for a particular type of token in the findByToken() method with just the JSON stored as a document?
• If yes, great!
• If not, what would be the fallback option for solving that problem?
  • Would you build the data structures for searching separately from storing the JSON data?
• Any other issues with building an implementation that you foresee if the the project only provides the to/from JSON capability?

[mikesaurus]

Hey @sjohnr
TL;DR; -> I think the JSON mapping should be enough to implement a NoSql version of the data service. On a save, we'd be extracting the available token/id values from the OAuth2Authorization object, transforming the object to JSON, and then storing the document for each of the extracted key values. The find process would consist a key lookup using the provided token value and token type and a transform from JSON to an OAuth2Authorization object.

Long-winded thoughts ->
I was working through this a bit last week. It looks like the current implementations of OAuth2AuthorizationService are querying for the authorization record by the generated server state, authorization code, access token, or refresh token. And, there's also the findById() lookup. I was originally thinking that I could make use of the Redis hash set capabilities to store the authorization document using the token values as fields in a hash set, where the hash was a composite of the client id and the user id/subject (since I believe that there should only be a single authorization per client+user). That would enable lookups by the token value or id, but also allow wholesale management (read, write, expire/remove) of the entire set of stored documents. Unfortunately, I don't think that the user information will available in all cases. Since each of those token values and the id should be unique, the most straightforward way to handle this is probably to simply store the authorization document for each of those four token types and the id (for the values that exist when saved/updated). Each of the keys should probably be classified in some way, to prevent any potential crossover between the id, generated state, auth code, and refresh token values.

The remove() process might be a little complicated since we can't use the hash set. I don't know if we can guarantee that all of the token values used to save the authorization document will exist in the provided OAuth2Authorization object? If not, that would mean that we'll have to attempt to extract each of the token values from the object, look up the document, and then check to see if the resulting object has any token values that need to be removed that weren't provided in the call to remove(). A little ugly, but still doable.

Not related specifically to NoSql storage, the one potential pain-point that I can see with creating any implementation of OAuth2AuthorizationService is that there are no bounds defining the token type for searches. Right now, the expected behavior is that an implementation must support state, code, access token, and refresh token. However, state and code token types are essentially plain strings wrapped in an OAuth2TokenType object. The only way to know what "types" need to be supported is to look at other OAuth2AuthorizationService implementations. If the Spring Auth Server framework changes in a way that requires a lookup using some other "token type", there is nothing in the interface contract that indicates the need for a change in custom implementations (no compile errors/warnings upon upgrading).

I have to qualify all of this by saying that I'm in the middle of a few different things right now, one of which is porting an existing auth server over to spring-auth-server. Most of the thoughts above are based on my initial analysis of the customizations that we need to make in order to complete the port. Currently, I'm focused on building out other areas of customization and I'm making use of the available JDBC implementations for testing. I haven't had a chance to implement and test any kind of Redis version yet.

[sjohnr]

Thanks @mikesaurus, very helpful information/feedback to ensure we're working towards something the community can use. You're in a perfect position to give that feedback because you're porting from the old auth server to this one, so that's great!

I will not comment too much on the design principles of a Redis implementation here, but having done one or two similar implementations in the past, I'd say just thinking of the state/tokens as a secondary index for the ID ought to simplify things for you. I wouldn't store the JSON multiple times, since we're not going to be optimizing it too much for space as far as I can see from this vantage point.

I'll take these thoughts away and get back with a final design for the JSON mapping utility/component. Thanks!

[mikesaurus]

@sjohnr, I hadn't thought about indexing since I've been so focused on the JSON transformation. Wrapping the JSON payload in an entity with some indexed token value properties might be a simple/clean solution that I can try out when I get to that point. Thanks for the idea and for all of the hard work on the project!

[bjornharvold]

I've implemented the Mongo variety of the Jdbc implementation as referenced above. Let me know if I should open a PR or if the addition of the spring-data-mongodb dependency warrants a separate module.

[sjohnr]

@bjornharvold, that's great! Feel free to create a separate github project for it. We won't be including a mongo (or redis) implementation in the core, though what we're discussing here is adding some support to make it easier to implement and keep in sync with the project's domain model.

[bjornharvold]

I added the MongoDb implementation here: https://github.com/bjornharvold/spring-authorization-server-mongodb

[jgrandja]

@mikesaurus

Serializing/deserializing spring-auth-server data objects to/from JSON requires custom data mapping that can create a maintenance nightmare in terms spring-auth-server compatibility (especially in the early/pre-v1.x stages of the spring-auth-server project).

Now that Spring Authorization Server 1.0 is released, we will be following Semantic Versioning going forward from the 1.0 release. This will help with upgrades for upcoming minor releases as it will remain compatible. Breaking changes are only allowed in major versions, however, a 2.0 release is a long way out.

As @sjohnr noted in this comment:

For more examples of how to support another data store (in this case JPA), please see the following examples:

• JpaRegisteredClientRepository
• JpaOAuth2AuthorizationService
• JpaOAuth2AuthorizationConsentService

In particular, notice how the the JpaOAuth2AuthorizationService implementation re-uses the Jackson module from this project to serialize the same fields into JSON as the JdbcOAuth2AuthorizationService does, therefore keeping those details in sync with this project. This is the recommended approach at the moment.

I would recommend using that guide as a starting point.

Furthermore, I am very reluctant on providing Json utility classes and/or mappers in the core module as this is not something we want to maintain. Based on your comment, we seem to be on the same page:

I don't have a problem with the Spring project not providing Redis support (you can't support every back end data store) and implementing the Spring Auth Server data server interfaces is reasonable.

However, I would like to find a middle ground here and provide some guidance on how to implement the core services using a JSON-backed store (e.g. Redis). I've opened gh-1019 to address this in a How-to guide that will include sample code that can be used as-is in the consuming application.

I hope this is helpful?

I'm going to close this in favour of gh-1019.

[jgrandja]

@mikesaurus @bjornharvold

See gh-1711 as it adds Redis implementations of the core components (RegisteredClientRepository, OAuth2AuthorizationService and OAuth2AuthorizationConsentService) to the demo-authorizationserver sample.