cider-doc doesn't work for Java classes (JDK 8 and JDK 9+)

[GreenRecycleBin]

Expected behavior

cider-doc displays the documentation for Java classes, e.g. in JDK 8:

java.lang.String
   Extends: java.lang.Object
Implements: java.io.Serializable
            java.lang.Comparable
            java.lang.CharSequence

The String class represents character strings. All string literals in Java
programs, such as "abc" , are implemented as instances of this class.
8>---<8

Actual behavior

cider-doc fails to display the documentation for Java classes (JDK8 and JDK 9+)

Java 8:

java.lang.String
   Extends: java.lang.Object
Implements: java.io.Serializable
            java.lang.Comparable
            java.lang.CharSequence

Not documented.

For additional documentation, see the Javadoc.

Definition location unavailable.

Java 13:

java.lang.String
   Extends: java.lang.Object
Implements: java.io.Serializable
            java.lang.Comparable
            java.lang.CharSequence
            java.lang.constant.Constable
            java.lang.constant.ConstantDesc

Not documented.

For additional documentation, see the Javadoc.

Definition location unavailable.

[back]

Steps to reproduce the problem

1. Create a new Leiningen project

$ lein new app test

2. Open ./test/src/test/core.clj in Emacs and cider-jack-in to the project
3. Switch to the cider-repl buffer
4. Invoke cider-doc with C-c C-d C-d, type in String, and return

Problem

Java 8: Loading orchard.java in lein repl fails because tools.jar is not visible to the classloader of orchard.java/source-info, which makes loading orchard/java/legacy_parser.clj unsuccessful.

[greenrecyclebin@localhost orchard]$ git rev-parse HEAD
d9f444fc91e07a17ed106afe5fd02dd5bceb87a7
[greenrecyclebin@localhost orchard]$ lein repl
nREPL server started on port 36375 on host 127.0.0.1 - nrepl://127.0.0.1:36375
REPL-y 0.4.3, nREPL 0.6.0
Clojure 1.10.1
OpenJDK 64-Bit Server VM 1.8.0_222-b10
    Docs: (doc function-name-here)
          (find-doc "part-of-name-here")
  Source: (source function-name-here)
 Javadoc: (javadoc java-object-or-class-here)
    Exit: Control+D or (exit) or (quit)
 Results: Stored in vars *1, *2, *3, an exception in *e

user=> (require '[orchard.java])
Syntax error (ClassNotFoundException) compiling at (orchard/java/legacy_parser.clj:1:1).
com.sun.javadoc.ClassDoc

Java 13: Loading orchard.java in lein-repl works.

[greenrecyclebin@localhost orchard]$ lein repl
OpenJDK 64-Bit Server VM warning: Options -Xverify:none and -noverify were deprecated in JDK 13 and will likely be removed in a future release.
nREPL server started on port 46195 on host 127.0.0.1 - nrepl://127.0.0.1:46195
REPL-y 0.4.3, nREPL 0.6.0
Clojure 1.10.1
OpenJDK 64-Bit Server VM 13+33
    Docs: (doc function-name-here)
          (find-doc "part-of-name-here")
  Source: (source function-name-here)
 Javadoc: (javadoc java-object-or-class-here)
    Exit: Control+D or (exit) or (quit)
 Results: Stored in vars *1, *2, *3, an exception in *e

user=> (require '[orchard.java])
nil

Debugging Notes

cider-bisect.txt
cider-nrepl-bisect.txt
orchard-bisect.txt

Java 8: Loading orchard.java in lein repl fails because tools.jar is not visible to the classloader of orchard.java/source-info, which makes loading orchard/java/legacy_parser.clj unsuccessful.

cider-jack-in to orchard in Emacs.

For d9f444fc91e07a17ed106afe5fd02dd5bceb87a7:

user> (require '[orchard.java.classpath :as cp]
               '[orchard.java :as java])
nil
user> (cp/classloaders)
(#object[clojure.lang.DynamicClassLoader 0x4aea7fa8 "clojure.lang.DynamicClassLoader@4aea7fa8"]
 #object[clojure.lang.DynamicClassLoader 0x4bd059cd "clojure.lang.DynamicClassLoader@4bd059cd"]
 #object[sun.misc.Launcher$AppClassLoader 0x5e2de80c "sun.misc.Launcher$AppClassLoader@5e2de80c"]
 #object[sun.misc.Launcher$ExtClassLoader 0xfbe55c6 "sun.misc.Launcher$ExtClassLoader@fbe55c6"])
user> (cp/modifiable-classloader)
#object[clojure.lang.DynamicClassLoader 0x4bd059cd "clojure.lang.DynamicClassLoader@4bd059cd"]
user> (.getClassLoader (class java/source-info))
#object[sun.misc.Launcher$AppClassLoader 0x5e2de80c "sun.misc.Launcher$AppClassLoader@5e2de80c"]
user> (dynapath.dynamic-classpath/can-add? (.getClassLoader (class java/source-info)))
false
user> (cp/classloaders (.getClassLoader (class java/source-info)))
(#object[sun.misc.Launcher$AppClassLoader 0x5e2de80c "sun.misc.Launcher$AppClassLoader@5e2de80c"]
 #object[sun.misc.Launcher$ExtClassLoader 0xfbe55c6 "sun.misc.Launcher$ExtClassLoader@fbe55c6"])
user> (cp/modifiable-classloader (.getClassLoader (class java/source-info)))
nil

1. tools.jar and src.zip are added to clojure.lang.DynamicClassLoader@4bd059cd
2. The classloader of orchard.java/source-info is sun.misc.Launcher$AppClassLoader@33909752 which is not modifiable

Similarly for b83826f42604f5f9bf59e25a535ef9d3ddb3c5db:

user> (require '[orchard.classpath :as cp]
               '[orchard.java :as java])
nil
user> (cp/classloaders)
(#object[clojure.lang.DynamicClassLoader 0x472d353a "clojure.lang.DynamicClassLoader@472d353a"]
 #object[clojure.lang.DynamicClassLoader 0x51dc4dd6 "clojure.lang.DynamicClassLoader@51dc4dd6"]
 #object[sun.misc.Launcher$AppClassLoader 0x33909752 "sun.misc.Launcher$AppClassLoader@33909752"]
 #object[sun.misc.Launcher$ExtClassLoader 0x2e9bf197 "sun.misc.Launcher$ExtClassLoader@2e9bf197"])
user> (cp/modifiable-classloader)
#object[clojure.lang.DynamicClassLoader 0x51dc4dd6 "clojure.lang.DynamicClassLoader@51dc4dd6"]
user> (.getClassLoader (class java/source-info))
#object[sun.misc.Launcher$AppClassLoader 0x33909752 "sun.misc.Launcher$AppClassLoader@33909752"]
user> (dynapath.dynamic-classpath/can-add? (.getClassLoader (class java/source-info)))
false
user> (cp/classloaders (.getClassLoader (class java/source-info)))
(#object[sun.misc.Launcher$AppClassLoader 0x33909752 "sun.misc.Launcher$AppClassLoader@33909752"]
 #object[sun.misc.Launcher$ExtClassLoader 0x2e9bf197 "sun.misc.Launcher$ExtClassLoader@2e9bf197"])
user> (cp/modifiable-classloader (.getClassLoader (class java/source-info)))
Execution error (ArityException) at user/eval6681 (form-init7868615620930048941.clj:70).
Wrong number of args (1) passed to: orchard.classpath/modifiable-classloader
(The above is expected because cp/modifiable-classloader doesn't accept any argument at this commit.)

At both commits, [org.tcrawley/dynapath "1.0.0"] is used which makes URLClassLoader non-modifiable. Note that @jeffvalk attempted to address this issue in clojure-emacs/orchard@314c84a (according to clojure-emacs/orchard#50 (comment)).

Java 8: Loading orchard.java in lein-repl works

cider-jack-in to orchard in Emacs.

For d49e659f9f46381bdf3589b32b5969b263989c4d:

user> (require '[orchard.classpath :as cp]
               '[orchard.java :as java])
nil
user> (->> (orchard.classloader/class-loader)
           (iterate #(.getParent ^ClassLoader %))
           (take-while identity))
(#object[clojure.lang.DynamicClassLoader 0x7f513645 "clojure.lang.DynamicClassLoader@7f513645"]
 #object[clojure.lang.DynamicClassLoader 0x56a9175a "clojure.lang.DynamicClassLoader@56a9175a"]
 #object[sun.misc.Launcher$AppClassLoader 0x70dea4e "sun.misc.Launcher$AppClassLoader@70dea4e"]
 #object[sun.misc.Launcher$ExtClassLoader 0x72a90099 "sun.misc.Launcher$ExtClassLoader@72a90099"])
user> (.getClassLoader (class java/source-info))
#object[sun.misc.Launcher$AppClassLoader 0x70dea4e "sun.misc.Launcher$AppClassLoader@70dea4e"]
user> (dynapath.dynamic-classpath/can-add? (.getClassLoader (class java/source-info)))
true

Solution

1. Easy solution (JDK 8 and JDK 9+) inspired by cider documentation on usage with JBoss AS/JBoss EAP/WildFly

Load orchard.java before loading the middlewares.

$ git --no-pager stash show -p
diff --git a/src/cider/nrepl.clj b/src/cider/nrepl.clj
index b6f39b7..ad7c972 100644
--- a/src/cider/nrepl.clj
+++ b/src/cider/nrepl.clj
@@ -1,5 +1,6 @@
 (ns cider.nrepl
   (:require
+   [orchard.java :as java]
    [cider.nrepl.version :as version]
    [cider.nrepl.middleware.util.cljs :as cljs]
    [cider.nrepl.print-method]

cider-doc works again (JDK 8 and JDK 9+) so tools.jar must be visible to the classloader of orchard.java/source-info. However, the classloader of orchard.java/source-info is not modifiable so I'm not sure how this solution works.

user> (require '[cider.nrepl.inlined-deps.orchard.v0v5v3.orchard.java.classpath :as cp]
               '[cider.nrepl.inlined-deps.orchard.v0v5v3.orchard.java :as java])
nil
user> (cp/classloaders)
(#object[clojure.lang.DynamicClassLoader 0x6941f84 "clojure.lang.DynamicClassLoader@6941f84"]
 #object[clojure.lang.DynamicClassLoader 0x484c12 "clojure.lang.DynamicClassLoader@484c12"]
 #object[clojure.lang.DynamicClassLoader 0x2aeefcc "clojure.lang.DynamicClassLoader@2aeefcc"]
 #object[sun.misc.Launcher$AppClassLoader 0x7852e922 "sun.misc.Launcher$AppClassLoader@7852e922"]
 #object[sun.misc.Launcher$ExtClassLoader 0x4f55957a "sun.misc.Launcher$ExtClassLoader@4f55957a"])
user> (cp/modifiable-classloader)
#object[clojure.lang.DynamicClassLoader 0x2aeefcc "clojure.lang.DynamicClassLoader@2aeefcc"]
user> (.getClassLoader (class java/source-info))
#object[sun.misc.Launcher$AppClassLoader 0x7852e922 "sun.misc.Launcher$AppClassLoader@7852e922"]
user> (cider.nrepl.inlined-deps.dynapath.v1v0v0.dynapath.dynamic-classpath/can-add? (.getClassLoader (class java/source-info)))
false
user> (cp/classloaders (.getClassLoader (class java/source-info)))
(#object[sun.misc.Launcher$AppClassLoader 0x7852e922 "sun.misc.Launcher$AppClassLoader@7852e922"]
 #object[sun.misc.Launcher$ExtClassLoader 0x4f55957a "sun.misc.Launcher$ExtClassLoader@4f55957a"])
user> (cp/modifiable-classloader (.getClassLoader (class java/source-info)))
nil

2. Better solution (?)

As recommended by dynapath, add a modifiable URLClassLoader subclass as the highest loader of the (cider) enviroment.

Test coverage (or why I believe this problem went undetected)

orchard.java-test/source-info-test is conditional on whether tools.jar is visible to the classpath of orchard.java/source-info.

However, even when I ran the tests unconditionally, they still passed (?!). I'm not sure how to improve this.

Comment

I'd appreciate pointers to learn more about this. Thank you for your time and effort in building cider.

Environment & Version information

CIDER version information

;; CIDER 0.24.0snapshot (package: 20191013.828), nREPL 0.6.0
;; Clojure 1.10.1, Java 1.8.0_222

Lein/Boot version

$ lein --version
Leiningen 2.9.1 on Java 1.8.0_222 OpenJDK 64-Bit Server VM

Emacs version

GNU Emacs 26.2 (build 1, x86_64-redhat-linux-gnu, GTK+ Version 3.24.8) of 2019-04-30

Operating system

Fedora 30:

$ cat /etc/os-release 
NAME=Fedora
VERSION="30 (Workstation Edition)"
ID=fedora
VERSION_ID=30
VERSION_CODENAME=""
PLATFORM_ID="platform:f30"
PRETTY_NAME="Fedora 30 (Workstation Edition)"
ANSI_COLOR="0;34"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:30"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f30/system-administrators-guide/"
SUPPORT_URL="https://fedoraproject.org/wiki/Communicating_and_getting_help"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=30
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=30
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
VARIANT="Workstation Edition"
VARIANT_ID=workstation
$ uname -r
5.2.17-200.fc30.x86_64

[bbatsov]

Thanks for the great issue report! I was aware of the problems on Java 9+, but I had definitely missed the regression on Java 8. I'll ping here @jeffvalk as well, as he's our resident expert on the subject. I think there are also a couple of related tickets floating around.

[bbatsov]

Here's the related ticket covering Java 9+ #2687

[GreenRecycleBin]

I'd like to add that the first solution only works when cider-jack-in uses Clojure command line tools (i.e. clj and clojure). The repl output provided in the first solution was within cider-nrepl, which uses lein, hence it showed a different classpath hierarchy from another project that uses Clojure command line tools.

Besides, I was incorrect to hint that the root cause was tools.jar (and src.zip) wasn't visible to the classloader of orchard.java/source-info. (They weren't visible to that classloader, but it didn't matter, because it wasn't the classloader used to load them). They were actually not visible to the context classloader of the current thread where the loading happened.

In summary, loading orchard.java would work when tools.jar is visible to the context classloader of the current thread. This happens when the class hierarchy of the context classloader of the current thread and (clojure.lang.RT/baseLoader) (i.e. Compiler/LOADER) share a DynamicClassLoader that can load tools.jar.

It is indeed the case with Clojure command line tools:

$ clj
Clojure 1.10.1
user=> (require '[clojure.pprint :refer [pprint]])
nil
user=> (defn classloaders
  ([]
   (classloaders (.. Thread currentThread getContextClassLoader)))
  ([classloader]
   (->> classloader
        (iterate #(.getParent %))
        (take-while identity))))
#'user/classloaders
user=> (pprint (classloaders))
(#object[clojure.lang.DynamicClassLoader 0x7bf3a5d8 "clojure.lang.DynamicClassLoader@7bf3a5d8"]
 #object[sun.misc.Launcher$AppClassLoader 0x7852e922 "sun.misc.Launcher$AppClassLoader@7852e922"]
 #object[sun.misc.Launcher$ExtClassLoader 0x1c3146bc "sun.misc.Launcher$ExtClassLoader@1c3146bc"])
nil
user=> (pprint (classloaders @Compiler/LOADER))
(#object[clojure.lang.DynamicClassLoader 0x32c8e539 "clojure.lang.DynamicClassLoader@32c8e539"]
 #object[clojure.lang.DynamicClassLoader 0x7bf3a5d8 "clojure.lang.DynamicClassLoader@7bf3a5d8"]
 #object[sun.misc.Launcher$AppClassLoader 0x7852e922 "sun.misc.Launcher$AppClassLoader@7852e922"]
 #object[sun.misc.Launcher$ExtClassLoader 0x1c3146bc "sun.misc.Launcher$ExtClassLoader@1c3146bc"])
nil
user=> (pprint (classloaders (clojure.lang.RT/baseLoader)))
(#object[clojure.lang.DynamicClassLoader 0x5c8504fd "clojure.lang.DynamicClassLoader@5c8504fd"]
 #object[clojure.lang.DynamicClassLoader 0x7bf3a5d8 "clojure.lang.DynamicClassLoader@7bf3a5d8"]
 #object[sun.misc.Launcher$AppClassLoader 0x7852e922 "sun.misc.Launcher$AppClassLoader@7852e922"]
 #object[sun.misc.Launcher$ExtClassLoader 0x1c3146bc "sun.misc.Launcher$ExtClassLoader@1c3146bc"])
nil

It is also the case with lein outside of a project. It's not the case with lein inside a project. See nrepl/nrepl#113 (comment).

I think nrepl/nrepl#113 (comment) implemented the second solution I suggested above.

[GreenRecycleBin]

Note that both solutions are required for cider-doc to work reliably again with Clojure command line tools or lein.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contribution and understanding!