UriComponentBuilder doesn't work with encoded HTTP URL having '+'. [SPR-14828]

[spring-projects-issues]

Johnny Lim opened SPR-14828 and commented

'+' is valid for a space but with the following code:

String httpUrl = "http://localhost:8080/test/print?value=%EA%B0%80+%EB%82%98";
URI uri = UriComponentsBuilder.fromHttpUrl(httpUrl).build(true).toUri();
System.out.println(uri);

I got the following error:

java.lang.IllegalArgumentException: Invalid character '+' for QUERY_PARAM in "%EA%B0%80+%EB%82%98"

	at org.springframework.web.util.HierarchicalUriComponents.verifyUriComponent(HierarchicalUriComponents.java:313)
	at org.springframework.web.util.HierarchicalUriComponents.verify(HierarchicalUriComponents.java:281)
	at org.springframework.web.util.HierarchicalUriComponents.<init>(HierarchicalUriComponents.java:90)
	at org.springframework.web.util.UriComponentsBuilder.build(UriComponentsBuilder.java:336)
	at learningtest.org.springframework.web.UriComponentBuilderTests.testFromHttpUrlBuildEncoded(UriComponentBuilderTests.java:19)

---

Affects: 4.3.3

Reference URL: https://github.com/izeye/spring-boot-throwaway-branches/blob/rest-and-logback-access/src/test/java/learningtest/org/springframework/web/UriComponentBuilderTests.java

Issue Links:

• UriComponentsBuilder inconsistent encode/decode query params behavior [SPR-16002] #20551 UriComponentsBuilder inconsistent encode/decode query params behavior ("is duplicated by")
• UriComponents.Type.QUERY_PARAM does not match spec [SPR-10172] #14805 UriComponents.Type.QUERY_PARAM does not match spec ("is duplicated by")
• UriComponentsBuilder does not encode "+" properly [SPR-16718] #21259 UriComponentsBuilder does not encode "+" properly
• Not encoding '+' in URLs anymore breaks backwards compatibility with apps running on spring 4 [SPR-17474] #22006 Not encoding '+' in URLs anymore breaks backwards compatibility with apps running on spring 4
• Encoding of URI Variables on RestTemplate [SPR-16202] #20750 Encoding of URI Variables on RestTemplate
• UriUtils query param "safe" encoding mode (%-encode sub-delims) [SPR-16934] #21473 UriUtils query param "safe" encoding mode (%-encode sub-delims)

Referenced from: commits f2e293a

1 votes, 6 watchers

[spring-projects-issues]

Rossen Stoyanchev commented

According to the URI spec RFC 3986, a "+" is allowed in the query (i.e. pchar > sub-delimiters > ";"). This matches our own HierarchicalUriComponents.QUERY except QUERY_PARAM further excludes "=", "&", and "+". The "=", "&" make sense since those are commonly used as query parameter delimiters but I'm not sure where the "+" comes from.

Arjen Poutsma what's your take on this? Is the "+" encoded as an extra precaution against it being decoded and interpreted as a space? I found an old discussion on this here. Or could it be based on the older version of the spec RFC 2396 which does seem to call out "+" as a reserved character in the query?

[spring-projects-issues]

Arjen Poutsma commented

It's been a while since I wrote that code, so I can't remember what was the reason for excluding the '+'. My guess would be the same as yours: it was based on an older version of the RFC.

[spring-projects-issues]

Alexandre Navarro commented

I have exactly the same problem. Interested to fix it in a future version.

[spring-projects-issues]

Kazuki Shimizu commented

Hi,

Probably by this changes, + does't encode to %2B .
Is this behavior working as design?
In this case, I think it should encode to the %2B because there is a case that + is decoded as space by the application server.
What do you think?

@Test
public void test() {
    // http://localhost:8080/?q=+
    System.out.println(UriComponentsBuilder.fromHttpUrl(
            "http://localhost:8080/").queryParam("q", "+").build().encode()
            .toUri());
    // http://localhost:8080/?q=%20
    System.out.println(UriComponentsBuilder.fromHttpUrl(
            "http://localhost:8080/").queryParam("q", " ").build().encode()
            .toUri());
    // http://localhost:8080/?q=%26
    System.out.println(UriComponentsBuilder.fromHttpUrl(
            "http://localhost:8080/").queryParam("q", "&").build().encode()
            .toUri());
    // http://localhost:8080/?q=%3D
    System.out.println(UriComponentsBuilder.fromHttpUrl(
            "http://localhost:8080/").queryParam("q", "=").build().encode()
            .toUri());
}

[spring-projects-issues]

Rossen Stoyanchev commented

This is by design indeed. Only reserved characters are reserved but "+" is allowed. Is this through the RestTemplate? If yes take a look at the uriTemplateHandler and the DefaultUriTemplateHandler implementation (DefaultUriBuilderFactory in 5.0). It has a strict encoding property.

[spring-projects-issues]

Kazuki Shimizu commented

Thanks for quick answer!

RestTemplate ?

No, I don't use the RestTemplate in my application.
I was making a query string for pagination link using the UriComponentsBuilder like above example.
There is a case that includes 0x2b(+) *1 in a pagination link because it is a value that input by an end user.

[spring-projects-issues]

Kazuki Shimizu commented

Oh... Sorry, I click the "Add" button by mistake.

I understood that need to change to use an another APIs.
Thanks.

[spring-projects-issues]

Dmitry Katsubo commented

Rossen Stoyanchev: I've tried to locate "strict" mode in DefaultUriBuilderFactory, but it has only URI_COMPONENT in enum EncodingMode which is the default. So this one does not encode (escape) + in path and query:

assertEquals("http://localhost:8080/test+/print?value=%25EA%25B0%2580+%25EB%2582%2598", new DefaultUriBuilderFactory().uriString("http://localhost:8080/{path}/print?value=%EA%B0%80+%EB%82%98").build("test+").toString());

DefaultUriTemplateHandler encodes only path, but not query:

DefaultUriTemplateHandler defaultUriTemplateHandler = new DefaultUriTemplateHandler();

defaultUriTemplateHandler.setStrictEncoding(true);

assertEquals("http://localhost:8080/test%2B/print?value=%EA%B0%80+%EB%82%98", defaultUriTemplateHandler.expand("http://localhost:8080/{path}/print?value=%EA%B0%80+%EB%82%98", "test+").toString());

Now I have difficulty in how to escape query part. Actually in my case I start from unencoded URL string, hence all characters that have special meaning should be encoded. How to do it?

assertEquals("http://localhost:8080/test+/print?value=v1%20v2%2Bv3", UriComponentsBuilder.fromHttpUrl("http://localhost:8080/{path}/print?value={val}").buildAndExpand("test+", "v1 v2+v3").encode().toUri().toString());

[spring-projects-issues]

Rossen Stoyanchev commented

Dmitry Katsubo, the EncodingMode is an enum with 3 values. You need this:

DefaultUriBuilderFactory factory = new DefaultUriBuilderFactory();
factory.setEncodingMode(EncodingMode.VALUES_ONLY);
URI uri = factory.uriString("http://localhost:8080/{path}/print?value=%EA%B0%80+%EB%82%98").build("test+");

Note that strict encoding is only applied to expanded URI variable values (and the same is true for DefaultUriTemplateHandler). This is why your query is not getting encoded. See the Javadoc for details.

Kazuki Shimizu the above DefaultUriBuilderFactory internally uses UriComponentsBuilder and essentially does this:

List<String> vars = UriUtils.encodeUriVariables("+");
System.out.println(UriComponentsBuilder.fromHttpUrl(
            "http://localhost:8080/?").queryParam("q", "{value}").build().expand(vars)
            .toUri());

[spring-projects-issues]

Dmitry Katsubo commented

OK, here is an example which I expect to succeeded because I set query parameter as unencoded value with intent that + should be passed literally to remote server hence should be encoded:

DefaultUriBuilderFactory factory = new DefaultUriBuilderFactory();
factory.setEncodingMode(EncodingMode.VALUES_ONLY);
UriBuilder uriString = factory.uriString("http://localhost/path");
uriString.queryParam("value", "v1+v2");

assertEquals("http://localhost/path?value=v1%2Bv2", uriString.build().toString());

[spring-projects-issues]

Rossen Stoyanchev commented

Did you miss this part "strict encoding is only applied to expanded URI variable values" ?

[spring-projects-issues]

Dmitry Katsubo commented

Is it possible to further extend API so that one can encode URL query parameters in one go? Namely:

• Add method ```
  UriComponentsBuilder#getQueryParams() { return queryParams; }

* Make methods `UriUtils.encodeUriVariables()` public.

Then it would be possible to encode query params in one go, without extra substitution step.

[spring-projects-issues]

Rossen Stoyanchev commented

I've made UriUtils.encodeUriVariables() public. As for UriComponentsBuilder#getQueryParams(), I don't think we can add just one accessor, there are no others on the builder. Is there a reason you can't call build() first and then access the queryParams?

[spring-projects-issues]

Dmitry Katsubo commented

build() transforms query params into non-mutable state, see HierarchicalUriComponents constructor, line 96. So at the moment it is only possible to build(), then get query params, encode them, put back via UriComponentsBuilder#replaceQueryParams(MultiValueMap<String, String> params) and then build() again...
Another alternative is to encapsulate UriComponentsBuilder the same way as DefaultUriBuilderFactory$DefaultUriBuilder does and do necessary preparations using UriUtils.encodeUriVariables() in corresponding methods (queryParam(), replaceQueryParam(), ...).
Any better idea?

[spring-projects-issues]

Rossen Stoyanchev commented

I see, or you could build a MultiValueMap of query params, use UriUtils to encode, and pass into UriComponentsBuilder#query. The only catch is we need to enhance UriUtils to support MultiValueMaps.

I think adding another encoding mode to DefaultUriBuilderFactory.EncodingMode would be perfectly okay to do. It's exactly what it's meant for -- provide control over encoding strategies. However I am not sure I understand exactly what the new mode is, i.e. aside from fully encoding query params, how does it work for others?

[spring-projects-issues]

Dmitry Katsubo commented

I tried somehow to apply the behaviour I need using the strategies that I've mentioned above, but honestly I've failed. The problem is that once I replace + with %2B, it (correctly) gets double-encoded as %252B when encode() is called. So there are not so many alternatives: either use queryParam("q", "{value}").build().expand(vars) as advised above or do some massaging of URL after it is built.

At the moment I've decided to switch to javax.ws.rs.core.UriBuilder which provides very similar API with minor effect that last one is more strict and %-encodes (among others) , as %2C and / as %2F as well, but that is perfectly OK as receiver should interpret that correctly.

If somebody is interested in code I came up with, here it goes:

public static class UriBuilderFactory extends UriComponentsBuilder {
	
	public static UriComponentsBuilder fromHttpUrl(String httpUrl) {
		return new UriBuilderFactory(UriComponentsBuilder.fromHttpUrl(httpUrl));
	}

	public UriBuilderFactory(UriComponentsBuilder other) {
		super(other);
	}
	
	@Override
	// queryParams(), replaceQueryParam() and replaceQueryParams() should be overridden and values should be processed in similar way 
	public UriComponentsBuilder queryParam(String name, Object... values) {
		return super.queryParam(name, encode(values));
	}
	
	// To be replaced by UriUtils.encodeUriVariables()
	private static Object[] encode(Object[] values) {
		Object[] encodedValues = new Object[values.length];
		
		for (int i = 0; i < values.length; i++) {
			if (values[i] != null) {
				encodedValues[i] = values[i].toString().replace("+", "%2B");
			}
		}
		
		return encodedValues;
	}
}

// Looks OK however not calling encode() has other side effects like passing through '=' which should be encoded:
assertEquals("http://host.com/path?key=%2Ba,b%2Bc&d=e/", UriBuilderFactory.fromHttpUrl("http://host.com/{var}").queryParam("key", "%2Ba,b+c&d=e/").buildAndExpand("path").toUriString());

// This implicitly calls encode() and hence '+' gets double-encoded:
assertEquals("http://host.com/path?key=%252Ba,b%252Bc%26d%3De/", UriBuilderFactory.fromHttpUrl("http://host.com/{var}").queryParam("key", "%2Ba,b+c&d=e/").build("path").toString());

[spring-projects-issues]

Rossen Stoyanchev commented

Dmitry Katsubo, Kazuki Shimizu I've created #21577 which should provide a solution.

[hviranicitco]

I know this is an old issue, but we are facing the exact same issue.

We are using Spring webclient directly. So something like

    WebClient.builder()
        .baseUrl("http://some-base-url")
        .build()
        .get()
        .uri {
          val uriBuilder = it.path("/some-path")
          uriBuilder.queryParam("search", searchText)
          uriBuilder.build()
        }
        .retrieve()
        .awaitBody()

The searchText comes from the user input. Now when the search input contains +, we expect that this would be encoded to %2B and send to client like http://some-base-url/some-path?search=%2B, but instead it gets sent as http://some-base-url/some-path?search=+, which would be interpreted as space rather than plus sign.

So the workaround for us is we do something like:

  val factory = DefaultUriBuilderFactory("http://some-base-url")
  factory.encodingMode = DefaultUriBuilderFactory.EncodingMode.NONE

 WebClient.builder()
       .uriBuilderFactory(factory)
        .build()
        .get()
        .uri {
          val uriBuilder = it.path("/some-path")
          uriBuilder.queryParam("search", urlEncode(searchText))
          uriBuilder.build()
        }
        .retrieve()
        .awaitBody()

So disable encoding and do the encoding our side. Here, urlEncode is just wrapper around java.net.URLEncoder.encode. This works as expected.

Please let me know if I should open a new issue. I posted here, since this has correct context.

[halilbaydar]

Having same issue. UriComponentsBuilder doesn't encode '+'. So I take responsibility of encoding it myself using URLEncoder and it generates %2B but this time UriComponentsBuilder encodes % to 25 I don't know why?

original query param => 2024-10-22T16:42:38+03:00 and it doesn't encode it but it encodes
2024-10-07T16%3A42%3A38%2B03%3A00. %2B => %252B without reason.

Issue should be re-opened

[rstoyanchev]

@hviranicitco please review the section on encoding in the documentation.

You can apply full encoding through URI variables:

val uriBuilder = it.path("/some-path")
uriBuilder.queryParam("search", "{search}")
uriBuilder.build(searchText)