sdl-examples stuck on 2.1.0 and should been updated to 2.5.2 or 2.6.0

[Donneker]

issue

sdl-examples are stuck on sdl 2.1.0-snapshot
they do not work with current versions of sdl

goal

sdl-examples should be up-to-date with current version of sdl (at the moment 2.5.2 or 2.6.0-snapshot)

[Donneker]

I have got this fork here
https://github.com/Donneker/sdl-examples

that already upgraded and adjusted some files. but there is still open problem so that at the moment the examples are not running through.

I think this object still does not work as expected
loadJdbc {
type = CustomDataFrameAction
inputIds = [ab-csv-org]

resulting in exceptions like:
java.io.InvalidObjectException: ReflectiveOperationException during deserialization
Caused by: java.lang.IllegalArgumentException: too many arguments

[pgruetter]

You're right, we didn't update sdl-examples anymore after creating the getting-started repo. Do you think it still gives an added value? I guess we could think about keeping it up to date.

[Donneker]

Ok, I didnt know about that what you said. Yes I know and also looked into the getting-started repo, but still looked mainly at sdl-examples. maybe because of the name of the project.
I think it would be good if it works with actual versions of SDL, but of course there is no obligation.

I did try to update it, also with this branch without changing the code of the one example class -> https://github.com/Donneker/sdl-examples/tree/feature/original-ReduceNycCSVTransformer-as-CustomDfTransformer
but at the moment I do not understand why this doesnt work as expected. I don't see a difference between this and other examples from the getting-started repo, and also not from the test within the SDL itself.

the branch I linked results in the following exception when executing ProgrammaticAccessDemo with SDL version 2.5.2, that I wouldn't expect.

Caused by: java.lang.ClassCastException: class com.sample.ReduceNycCSVTransformer cannot be cast to class io.smartdatalake.workflow.action.spark.customlogic.CustomDfTransformer (com.sample.ReduceNycCSVTransformer and io.smartdatalake.workflow.action.spark.customlogic.CustomDfTransformer are in unnamed module of loader 'app')
		at io.smartdatalake.workflow.action.spark.transformer.ScalaClassSparkDfTransformer.<init>(ScalaClassSparkDfTransformer.scala:46)

also, I worked under Windows with Java17 (that also resulted in some small adjustments)

alltogether, I do not think there is missing much, but ofc. it is not yet finished and there still can be more errors coming up when this part works.

[pgruetter]

Thanks for your work! I also tried it here and I'm at a similar point as you getting ReflectionsException, even with Java 11.
Will have to further investigate.

[zzeekk]

Hi @Donneker, @pgruetter,
Works again on branch https://github.com/smart-data-lake/sdl-examples/tree/feature/upgrade-to-sdl-2.5.x. The main problem was the "Lambda Deserializer raises an IllegalArgumentException on JDK 17", which is a scala bug solved in 2.12.17, see also scala/bug#12419
I updated scala.version to 2.12.18, and did some other structural improvements like using log4j2.yml and moving ProgrammaticAccessMode to test scope to keep normal classpath clean (that triggered the RelectionsException, which actionally was not stopping the process, but spammed the console...)

Let me know if it works for you as well.

[Donneker]

Hi @zzeekk @pgruetter
yes thanks, verify now works fine for me.
for the "install" / build I have to change the in the POM to be in one line
--add-opens=java.base/sun.nio.ch=ALL-UNNAMED
that is at least for Ecplipse Temurin needed, else it doesnt understand that parameter. this should fix the build error in this pipeline as well.

I couldn't push it here though, since this branch is protected.

[Donneker]

sorry, wrong assumption, that build is running with jvm 1.8 and it doesnt recognice this option. anyway for jvm17 my statement can be true.
I used a profile for that that seems to work (found online) - maybe not all options needed here

	<profiles>
	  <profile>
		<id>jvm17</id>
		<activation>
			<jdk>17</jdk>
		</activation>
		<properties>
			<jvm17.opens>--add-exports=java.base/sun.reflect.generics.reflectiveObjects=ALL-UNNAMED
				--add-opens=java.base/java.io=ALL-UNNAMED
				--add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED
				--add-opens=java.base/java.net=ALL-UNNAMED
			</jvm17.opens>
		</properties>
	  </profile>
	</profiles>

[zzeekk]

Thanks, nice solution with profile auto activation. Added it to the PR #25.
@pgruetter: please review

[pgruetter]

Hi both
Thanks for your work on this. It works for me if I run in JDK17 (and the new profile is activated).
Doesn't run in JDK11 anymore though. I'll add a comment on the PR.