Update maven from 3.5 to 3.6.3 and required dependencies #184

abelsromero · 2021-02-26T22:38:09Z

This PR begun with the intent to update the maven version used 3.5 (2018-06-21) on java8 by latest 3.6.3 (2019-11-25) in java11.
But this broke other things and ended up requiring several changes in all generators.

Now, all but csharp work, which fails with the following error. I suspect it could be the ammount of types in the k8s API is too much.

[INFO] /tmp/.autorest/@[email protected]/node_modules/@autorest/core/dist/src_lib_autorest-core_ts.js - FAILURE {}
[INFO] /tmp/.autorest/@[email protected]/node_modules/@autorest/core/dist/src_lib_autorest-core_ts.js - FAILURE {}
[INFO] Process() cancelled due to exception : Maximum call stack size exceeded / RangeError: Maximum call stack size exceeded
[INFO]   RangeError: Maximum call stack size exceeded
[INFO]     at mergeYamls (/tmp/.autorest/@[email protected]/node_modules/@autorest/core/dist/src_lib_autorest-core_ts.js:10534:18)
[INFO]     at processTicksAndRejections (internal/process/task_queues.js:93:5)
[INFO]     at async LoadLiterateSwagger (/tmp/.autorest/@[email protected]/node_modules/@autorest/core/dist/src_lib_autorest-core_ts.js:3872:18)
[INFO]     at async LoadLiterateSwaggers (/tmp/.autorest/@[email protected]/node_modules/@autorest/core/dist/src_lib_autorest-core_ts.js:3902:29)
[INFO]     at async /tmp/.autorest/@[email protected]/node_modules/@autorest/core/dist/src_lib_autorest-core_ts.js:3926:26
[INFO]     at async ScheduleNode (/tmp/.autorest/@[email protected]/node_modules/@autorest/core/dist/src_lib_autorest-core_ts.js:2543:33)

Note also that CSharp does not currently work neither because it downloads latest version of autorest which requires a newer version of dotnet.

We could try to use the openapi maven plugin, based on this example, there's support for csharp https://openapi-generator.tech/docs/usage/, I cannot validate further than checking than some code is being generated. wdyt, should I replace current autorest usage?

Said, that this PR required the following changes:

Upgraded openapi-generator-maven-plugin for all generators to 4.3.1. This is the latest v4.x. There's a 5.x but didn't want ot push too much. I'd rather do it in separated PR.
Added skipValidateSpec option to several generations, otherwise the k8s was considered not valid.
Updated installed packaged from Debian 9 to Debian 10. This is the version used by the new maven image. NOTE: using maven plugin for csharp would allow to get rid of this extra packages I suspect.
Updated dotnet installation according to official docs https://docs.microsoft.com/en-us/dotnet/core/install/linux-debian#debian-10-
Some minnor fixes: remove unnecessary maven depencies, format,...

k8s-ci-robot · 2021-02-26T22:38:11Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: abelsromero
To complete the pull request process, please assign brendandburns after the PR has been reviewed.
You can assign the PR to them by writing /assign @brendandburns in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

This also includes: * Upgraded openapi-generator-maven-plugin for all generators to 4.3.1. This is the latest v4.x. There's a 5.x but didn't want ot push too much. I'd rather do it in separated PR. * Added skipValidateSpec option to several generations, otherwise the k8s was considered not valid. * Updated installed packaged from Debian 9 to Debian 10. This is the version used by the new maven image. NOTE: using maven plugin for csharp would allow to get rid of this extra packages I suspect. * Updated dotnet installation according to official docs https://docs.microsoft.com/en-us/dotnet/core/install/linux-debian#debian-10- * Some minnor fixes: remove unnecessary maven depencies, format,...

brendandburns · 2021-03-02T20:37:13Z

To fix csharp code-gen we need to pin at version 2 so we should set npm i [email protected]

see: kubernetes-client/csharp#563

We can do that in this PR or a separate PR

brendandburns · 2021-03-02T20:38:11Z

openapi/go.xml

@@ -8,7 +8,7 @@
    <build>
        <plugins>
            <plugin>
-		<groupId>org.openapitools</groupId>
+		        <groupId>org.openapitools</groupId>


This indentation is weird, please format.

The issue is that in that line there are 2 tabs while the rest of the file uses spaces. GitHub may be missleading, but the IDE comparision shows it better:

Right, you need to be consistent with the rest of the file and use spaces.

brendandburns · 2021-03-02T20:38:58Z

openapi/openapi-generator/Dockerfile

@@ -21,7 +21,7 @@ ARG OPENAPI_GENERATOR_USER_ORG=OpenAPITools
 RUN apt-get update && apt-get -y install git python-pip && pip install urllib3==1.24.2

 # Install Autorest
-RUN apt-get update && apt-get -qq -y install libunwind8 libicu57 libssl1.0 liblttng-ust0 libcurl3 libuuid1 libkrb5-3 zlib1g


I think these are required for autorest/dotnet

The only changes are:

Removed libicu57: could not find equivalent for debian10.

Replaced libcurl3 by libcurl4: which is the "offical" replacement. Though there are some mentions of not all software being compatible.

About if the are required, once we have csharp working again we hopefuly be able to put that at rest.

Have a look here:
https://github.com/dotnet/core/blob/main/Documentation/linux-prereqs.md

libicu63 is the debian 10 replacement.

brendandburns · 2021-03-02T20:40:21Z

We need to validate that this doesn't introduce any code differences in the generated clients before this can be merged.

Ideally this would be a pre-merge check, but sadly it's not currently added as a test.

abelsromero · 2021-03-02T22:15:30Z

We can do that in this PR or a separate PR

I can give it a try in this same PR.

abelsromero · 2021-03-03T23:24:02Z

Running with v2.0.4413 resulted in the following error

There is a new version of AutoRest available (3.1.2).
 > You can install the newer version with with npm install -g autorest@latest

Starting @microsoft.azure/autorest-core from /tmp/.autorest/@[email protected]
   Loading AutoRest core      '/tmp/.autorest/@[email protected]/node_modules/@microsoft.azure/autorest-core/dist' (2.0.4417)
No configuration found at 'file:///'.
   Loading local AutoRest extension '@microsoft.azure/autorest.csharp' (/node_modules/@microsoft.azure/autorest.csharp)
   Loading local AutoRest extension '@microsoft.azure/autorest.modeler' (/node_modules/@microsoft.azure/autorest.modeler)
events.js:292
      throw er; // Unhandled 'error' event
      ^

Error: write EPIPE
    at afterWriteDispatched (internal/stream_base_commons.js:156:25)
    at writeGeneric (internal/stream_base_commons.js:147:3)
    at Socket._writeGeneric (net.js:785:11)
    at Socket._write (net.js:797:8)
    at writeOrBuffer (internal/streams/writable.js:358:12)
    at Socket.Writable.write (internal/streams/writable.js:303:10)
    at Writable.write [as _write] (/tmp/.autorest/@[email protected]/node_modules/@microsoft.azure/autorest-core/dist/lib/pipeline/plugin-endpoint.js:38:31)
    at writeOrBuffer (internal/streams/writable.js:358:12)
    at Writable.write (internal/streams/writable.js:303:10)
    at StreamMessageWriter.write (/node_modules/vscode-jsonrpc/lib/messageWriter.js:82:27)
    at /node_modules/vscode-jsonrpc/lib/main.js:761:35
    at new Promise (<anonymous>)
    at Object.sendRequest (/node_modules/vscode-jsonrpc/lib/main.js:751:26)
    at Object.GetPluginNames (/tmp/.autorest/@[email protected]/node_modules/@microsoft.azure/autorest-core/dist/lib/pipeline/plugin-endpoint.js:78:86)
    at AutoRestExtension.GetPluginNames (/tmp/.autorest/@[email protected]/node_modules/@microsoft.azure/autorest-core/dist/lib/pipeline/plugin-endpoint.js:116:31)
    at Function.FromChildProcess (/tmp/.autorest/@[email protected]/node_modules/@microsoft.azure/autorest-core/dist/lib/pipeline/plugin-endpoint.js:110:41)
Emitted 'error' event on Socket instance at:
    at emitErrorNT (internal/streams/destroy.js:106:8)
    at emitErrorCloseNT (internal/streams/destroy.js:74:3)
    at processTicksAndRejections (internal/process/task_queues.js:80:21) {
  errno: -32,
  code: 'EPIPE',
  syscall: 'write'

brendandburns · 2021-03-04T18:36:50Z

I'll take a look at getting the csharp generation working. There was a commit to do this here:

74ee0c6

But it got overwritten for some reason.

brendandburns · 2021-03-04T18:38:58Z

Taking a step back to the beginning, what is the motivation for upgrading to Java8/Maven 3.6? Is there something broken with the existing setup?

abelsromero · 2021-03-04T19:06:41Z

Taking a step back to the beginning, what is the motivation for upgrading to Java8/Maven 3.6? Is there something broken with the existing setup?

Mainly keeping things with up-to-date and supported versions.

brendandburns · 2021-03-05T05:05:09Z

Ok, given that there's no pressing need, I think I would like to delay this PR until we get csharp generating correctly. I'll try to get to that soon-ish. Apologies for the delay.

brendandburns · 2021-03-05T05:05:22Z

Ok, given that there's no pressing need, I think I would like to delay this PR until we get csharp generating correctly. I'll try to get to that soon-ish. Apologies for the delay.

abelsromero · 2021-03-05T07:43:26Z

Ok, given that there's no pressing need, I think I would like to delay this PR until we get csharp generating correctly. I'll try to get to that soon-ish. Apologies for the delay.

No problem at all. JS it not in my area of expertise, but I can do some more research and share what I find.

abelsromero · 2021-03-07T20:20:45Z

Tried to run autorest "natively" in my machine and it seems to go further, but finally fails with

DEBUG: [WRITING] ModelTemplate => Models/V1ValidatingWebhookConfigurationList.cs
DEBUG: [WRITING] ModelTemplate => Models/Corev1EventList.cs
DEBUG: [WRITING] ModelTemplate => Models/V1beta1MutatingWebhookConfigurationList.cs
DEBUG: csharp/generate - END
DEBUG: csharp/simplifier - START
No usable version of the libssl was found
FATAL: csharp/simplifier - FAILED
FATAL: Error: [Exception] AutoRest extension '@microsoft.azure/autorest.csharp' terminated.
Process() cancelled due to exception : [Exception] AutoRest extension '@microsoft.azure/autorest.csharp' terminated.

I have libssl installed, but probably not a compatible version, so I'll try to build a Debian VM. I noted that while running node starts listen on ports 12494 (LISTEN) and 32066 (LISTEN), I wonder if it could be related.

brendandburns · 2021-03-08T19:20:42Z

@abelsromero thanks for pursuing this. Definitely no obligation if you get bored/stuck I will get it done, but the assist is much appreciated!

abelsromero · 2021-03-22T08:14:44Z

Seeeing the interdependencies between tools and how these prevents other converters. I wonder if it would make sense to have different Dockerfiles. I know it would mean some duplication, but the fact that for one language we have to block the mantainance of of al the rest seems a high cost to me.

brendandburns · 2021-03-29T04:23:57Z

@abelsromero probably good observation. We could split them out and have them all depend on a shared base image, which would largely eliminate the duplication.

If this is blocking some work on java generation, let me know and I will increase the priority of fixing this.

k8s-ci-robot · 2021-04-27T04:11:41Z

@abelsromero: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-triage-robot · 2021-07-27T05:21:11Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

k8s-triage-robot · 2021-08-26T06:19:03Z

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot · 2021-09-25T06:47:20Z

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Reopen this issue or PR with /reopen
Mark this issue or PR as fresh with /remove-lifecycle rotten
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

k8s-ci-robot · 2021-09-25T06:47:22Z

@k8s-triage-robot: Closed this PR.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied

After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied

After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Reopen this issue or PR with /reopen

Mark this issue or PR as fresh with /remove-lifecycle rotten

Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Feb 26, 2021

k8s-ci-robot requested a review from brendandburns February 26, 2021 22:38

k8s-ci-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Feb 26, 2021

abelsromero changed the title ~~All working bu csharp~~ Update maven from 3.5 to 3.6.3 and required dependencies Feb 26, 2021

abelsromero force-pushed the upgrade-maven-to-3.6.3 branch from 5b666ff to 9fa5e7e Compare February 26, 2021 22:40

brendandburns reviewed Mar 2, 2021

View reviewed changes

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 27, 2021

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jul 27, 2021

k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Aug 26, 2021

k8s-ci-robot closed this Sep 25, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update maven from 3.5 to 3.6.3 and required dependencies #184

Update maven from 3.5 to 3.6.3 and required dependencies #184

abelsromero commented Feb 26, 2021

k8s-ci-robot commented Feb 26, 2021

brendandburns commented Mar 2, 2021

brendandburns Mar 2, 2021 •

edited

Loading

abelsromero Mar 2, 2021

brendandburns Mar 4, 2021

brendandburns Mar 2, 2021

abelsromero Mar 2, 2021

brendandburns Mar 4, 2021

brendandburns commented Mar 2, 2021

abelsromero commented Mar 2, 2021

abelsromero commented Mar 3, 2021

brendandburns commented Mar 4, 2021

brendandburns commented Mar 4, 2021

abelsromero commented Mar 4, 2021

brendandburns commented Mar 5, 2021

brendandburns commented Mar 5, 2021

abelsromero commented Mar 5, 2021

abelsromero commented Mar 7, 2021

brendandburns commented Mar 8, 2021

abelsromero commented Mar 22, 2021 •

edited

Loading

brendandburns commented Mar 29, 2021

k8s-ci-robot commented Apr 27, 2021

k8s-triage-robot commented Jul 27, 2021

k8s-triage-robot commented Aug 26, 2021

k8s-triage-robot commented Sep 25, 2021

k8s-ci-robot commented Sep 25, 2021

Update maven from 3.5 to 3.6.3 and required dependencies #184

Update maven from 3.5 to 3.6.3 and required dependencies #184

Conversation

abelsromero commented Feb 26, 2021

k8s-ci-robot commented Feb 26, 2021

brendandburns commented Mar 2, 2021

brendandburns Mar 2, 2021 • edited Loading

Choose a reason for hiding this comment

abelsromero Mar 2, 2021

Choose a reason for hiding this comment

brendandburns Mar 4, 2021

Choose a reason for hiding this comment

brendandburns Mar 2, 2021

Choose a reason for hiding this comment

abelsromero Mar 2, 2021

Choose a reason for hiding this comment

brendandburns Mar 4, 2021

Choose a reason for hiding this comment

brendandburns commented Mar 2, 2021

abelsromero commented Mar 2, 2021

abelsromero commented Mar 3, 2021

brendandburns commented Mar 4, 2021

brendandburns commented Mar 4, 2021

abelsromero commented Mar 4, 2021

brendandburns commented Mar 5, 2021

brendandburns commented Mar 5, 2021

abelsromero commented Mar 5, 2021

abelsromero commented Mar 7, 2021

brendandburns commented Mar 8, 2021

abelsromero commented Mar 22, 2021 • edited Loading

brendandburns commented Mar 29, 2021

k8s-ci-robot commented Apr 27, 2021

k8s-triage-robot commented Jul 27, 2021

k8s-triage-robot commented Aug 26, 2021

k8s-triage-robot commented Sep 25, 2021

k8s-ci-robot commented Sep 25, 2021

brendandburns Mar 2, 2021 •

edited

Loading

abelsromero commented Mar 22, 2021 •

edited

Loading