Sharing experiences with the latest EA build // #ReflectiveAccessToNonExportedTypes #ResourceEncapsulation
rafael.wth at gmail.com
Fri Sep 30 12:35:03 UTC 2016
I have tested the latest EA builds, both on my open-source projects and in
several production applications, one of them quite large with thousands of
development hours invested. I wanted to share my experiences.
After making files with the .class extension visible, my agent/code
generation library Byte Buddy works more or less as it did before.
Previously, I had to apply a work-around where I mapped package names of
all modules to their Module instance what allowed me read class files via
this instance. I discovered all modules via Layer.boot() but was of course
missing support for modules of other layers. This approach also hurt
runtime performance. With the newest EA build, out of 7000 unit tests, only
42 tests still fail and those are all because of using non-exported API of
the JCL. None of those usages are essential to using the library such that
I can live with this minor limitation. (I did not yet look to adding a file
in the META-INF folder which I read somewhere was possible.) I can also
confirm that using the latest EA build, cglib and Javassist work again.
This is great news as this breaking change really concerned me with regards
to Java 9 compatibility. Code generation libraries are used everywhere and
breaking them would have had an impact on any user of the JVM. As I
currently support Java 6, I do not plan to apply modules on my OSS in the
near future. This is of course not great as plain jars cannot be packaged
into jdeps but I hope to include a generated module-info.class at some
point even in a Java 6 jar in order to allow doing this.
I also tried to migrate a small and a large production application. The
larger applications is currently running on a Java 8 VM and code level but
started out in the days of Java 4. As the project is already organized in
Maven modules, approaching the migration was straight-forward. Creating the
first module already helped to uncover a minor breach of the intended
module boundaries that was now discovered by the Java 9 compiler. The
modularization did however not pay off too much due to the previous
modularization where the Maven modularization already offered a good
approximation. Alltogether, my experience was however quite positive. One
minor problem was that many times, several Maven modules declared the same
packages to communicate "cross-module shared classes" via package private
types. This required some refactoring of classes and splitting up of some
Getting the application to run was however not as easy as the adoption of
compile-time modules. As mentioned, the application has grown quite big
over the years and has many dependencies without anybody having a complete
overview of what is used and for what. In this sense, the application is
not polished but runs quite robust in production. (In the end, I think the
application is quite representative of many commercial applications out in
the wild.) Using Java 9 but without applying modules, the application did
start but sometimes fails at runtime due to libraries and code accessing
non-exported classes of the JCL. Many of these breaches happen high up the
stack in library and framework code. For now, I have therefore stripped
down the application to avoid using those application parts which I do not
know intimately enough to fix right now.
After deploying the application with some first modules on the module path,
I encountered similar problems. Adding correct exports to cover the use of
reflection is unfortunately non-trivial. My IDE lists over 420 usages of
Method::invoke in the project and its libraries what makes it practically
impossible to trace all possible reflective access. Generally, I do neither
want to depend too much on such implementation details as it makes updating
dependencies quite tedious as I have to revalidate the access patterns. My
major problem with adding additional modules onto the module path is that
each breaks code that is still on the class path. Step-by-step migration is
unfortunately impossible as I always need to consider all of our code and
the library code. There is (obviously) no way to check at compile time if a
module is accessed refletively from the class path what only leaves
runtime. This is also true for non-reflective access if I do not recompile
all code on the class path when transforming a module. This turned out to
be quite tedious, especially, since some problems only occur when
triggering a particular subroutine of the application. Unit tests do not
offer a good alternative either as I cannot really separate what tests are
supposed to have access to the module internals and which should not. The
application has thousands of tests, some of them not making much sense to
me, going through them would be economically impractical. Integration tests
sometimes uncover those access problems but unfortunately often disguise
them in the error reports.
To be fair, I think that I would get the application going if I invested
more time to trial and error. However, I still find the migration process
too difficult as it requires a lot of manual runs to validate. I think it
would be helpful if code on the class path would behave exactly as before
to allow for a better step-by-step migration, i.e. could access modularized
code reflectively without any constraints. Right now, it feels easiest to
export about everything before I add something to the module path.
Otherwise, it does not feel safe to apply the migration when I think about
putting the application into production: even when the application appears
to be running, errors can always occur later when a specific unit of code
is triggered. I am mostly scared when thinking of rarely triggered routines
like security failovers or error reporting mechanisms which are often
implemented quite generically (reads: use a lot of reflection).
I currently experiment with a Java agent to force Java 9 to adopt this
desired behavior by manipulating the boot layer's module graph or by
stubbing AccessibleObject::checkCanSetAccessible. I hope to trial this a
bit broader soon but it seems promising to allow us a "safe migration" to
Java 9. I would of course prefer to remove this agent at some point in time
but I would not know when or if all of our dependencies are "Java 9 ready".
(Some of them have not been updated for years.) I am also doing this as a
means of delegating migration to developers with less Java experience as
migrating to Java 9 turns out to be quite a demanding task. To be honest,
many junior developers do not even know what reflection is and it would be
difficult to explain to them how they need to modularize the code they work
with. As many libraries (or our own modules) do not document their
implementation but only their API, the only way to figure out the module
requirements is by reading its source code. For framework code, this often
outchallanges less experienced developers.
Finally, I already identified some Maven modules which would be difficult
to Java-9-modularize as they basically only work with reflection and are
used throughout many parts of the application. It feels a bit strange to
export packages from the users of this module as those "reflection modules"
are only used via several indirections. I would prefer it if I could
declare the module to be a form of "transitive module" which is simply
allowed to reflect as it was allowed previously. I understand that security
might be an issue here but why not simply ask a potentially present
security manager if such a transitive module is allowed to be used?
My conclusion is that I wished that migration would be easier to do
step-by-step. This way I can better spread the costs of adoption.
Currently, I think it would be easiest to not migrate to modules as the
benefits are limited in the context of existing Maven modularization. Right
now, any introduced Java module requires a lot of additional runtime
testing and I doubt that my customers would be willing to pay for it. With
a greenfield project, I would always try to adopt modules as they push
people into the right direction but as long as the library landscape is not
modularized, this again bears a high risk of runtime errors if those
libraries are not module aware. As many libraries are careful about
updating to not alienate legacy VM users, I imagine that this might take a
while. As for "hard updates", I still think a "force-exporting" Java agent
is a solution that I consider, simply to run on a Java 9 VM without having
to change or validate any code. There are a lot of great features that I
hope to take advantage of, besides Jigsaw.
Even though I know this idea is not popular, my recommendation is to have
the class path to behave just as it did before, i.e. reflection can cross
module boundaries without preparing those modules. This especially in the
context of the impossibility of putting the JCL onto the class path.
In my trial runs, I did not identify any problems with
#ResourceEncapsulation as all of our custom "cross-module files" were put
into the base folder, i.e. the unnamed module where they remain available.
I do however not know how this would affect other applications that put
resources into packages.
So much for my experience report. Altogether, I really think Java 9 has a
lot to offer and I really appreciate Oracle is not rushing this but takes
the time it takes to getting this right. Great work, also beyond Jigsaw
there is much I am looking forward too.
Best regards, Rafael
PS: I noticed that when running java (without commands), it still lists the
non-GNU-style options that are no longer supported.
More information about the jigsaw-dev