Proposal: #ReflectiveAccessToNonExportedTypes (revised) & #AwkwardStrongEncapsulation: Weak modules & private exports

David M. Lloyd david.lloyd at
Fri Sep 16 17:51:47 UTC 2016


On 09/16/2016 10:30 AM, David M. Lloyd wrote:
> After a fairly detailed review of this proposal, we have determined that
> it is not acceptable to Red Hat in its present form.  I will list the
> primary problems here, and then I'll start up discussion on jigsaw-dev
> of several possible solutions that could work for us.  I'll number these
> in case anyone wants to respond piece-wise.
> #1) The "weak" designation appears to be pejorative
> Under this solution, many existing frameworks and/or the modules which
> consume them must be marked "weak" in order to work correctly, or else a
> much more complex module descriptor must be used.  While this seems to
> be a stepping stone for migration, we find that it rather makes
> middleware appear as a second-class citizen.
> #2) Weak modules are, in fact, weak
> The idea of retaining the Java 8 rules for transition modules appears
> good on the surface, however the problem is that users may be forced to
> use weak modules indefinitely in order to interoperate with certain
> existing middleware and libraries, or else be faced with a potentially
> complex migration.  If the security benefits of "strong" modules are
> real, then users will be rightly inclined to move from weak modules to
> strong modules.  This can be severely problematic if existing
> middleware, libraries, and large applications cannot easily operate as
> or with "strong" modules or cannot do so in an "easy to learn"/"easy to
> use" fashion.  This also means that "weak" modules exist only for the
> purposes of maintaining a transition period - at some point they will
> inevitably transition from "convenience" to "attractive nuisance".  We
> believe that it is feasible to solve compatibility to a satisfactory
> degree without relegating users of existing software to a lower tier of
> security or functionality.
> #3) Controlling reflection access is always bound up with controlling
> exports
> The proposed exports system is simplistic but not exactly simple, yet
> several useful modes remain left unaccounted-for.  Internally we
> examined the usefulness of controlling access to public and so-called
> "deep" reflection in combination with whether or not a member was
> exported, and we came up with this logical table:
>          +------------------+---------------------------+
>          |                  |        Module is...       |
>          |  Reflection is... +------------+--------------+
>          |                  |  Exported  | Not Exported |
>          +----------------------------------------------+
>          |        Forbidden | Not Useful |    Useful    |
>          +----------------------------------------------+
>          |      Public Only |   Useful   |    Useful    |
>          +----------------------------------------------+
>          | Public & Private |   Useful   |    Useful    |
>          +------------------+------------+--------------+
> Some of these useful modes seem not to be available, or are not usable
> in an obvious or simple manner.  These modes are useful, possibly even
> necessary, to introduce in Java 9 in order to provide a good
> compatibility story with the many existing frameworks in the wild, in
> the absence of weak modules.
> It has been observed that in some cases, the necessity of granting
> public or private reflective access to a module is actually bound not
> with what that module exports, but rather with what it requires or uses,
> which might be useful in consideration of a solution; for example,
> "using" a persistence framework which operates only on public types
> would imply that the module would grant back public-only reflective
> access to the framework which provides the service.
> #4) There isn't always a container
> The justification for not providing an easy mechanism to restrict
> private reflective access to a trusted framework is that a container can
> always rewrite a module or inject classes.  However there isn't always a
> container present - something which the existence of weak modules
> acknowledges.  Some frameworks and middleware are meant to be usable
> with or without a container, and it is undesirable to force these use
> cases into a weak module situation or to require a strong coupling in
> the module descriptor in this case.  Some other solution should be found
> that works both for containers and "regular" applications.
> Broadly speaking, a container would be able to do additional things that
> are useful do to specifically in a container.  But a container should
> not be _necessary_ in order to do things that are generally useful, at
> least to the greatest extent possible.
> #6) Not all reflection is 100% "friendly", and that's OK
> Serialization frameworks generally need private reflective access to the
> module which contains the classes being serialized.  For some
> frameworks, this access will already be adequate.  Some existing
> frameworks utilize Unsafe to create uninitialized instances.  However
> other frameworks, including frameworks that seek to comply with the
> Serialization Specification [1], use ReflectionFactory, as that class
> provides (exclusively, as far as I know) the ability to acquire a
> constructor that produces a _partially_-initialized class where only the
> constructors of superclasses to a certain depth are called.  It is our
> view that it would be a regression to force such frameworks to use
> --add-exports or JVMTI, and using Unsafe for this purpose, or continuing
> to use it, (as I understand it) puts at risk the ability to perform
> optimizations which depend on entry point control, which I am given to
> understand are a strong motivator for many of the design constraints
> that exist on the Jigsaw implementation.  It should be possible for
> these frameworks to continue to use ReflectionFactory (unsupported
> though it may be) if suitably privileged (in the security manager
> sense), as long as all of the module(s) upon which the framework
> reflects have granted private reflective access to it (in the module
> declaration sense).  This may be imperfect in the event that class
> hierarchies are spread across several modules, but that is something
> that can be hashed out in detail on jigsaw-dev.
> A better alternative to ReflectionFactory itself could be sought in
> future releases, if necessary, though one hopes that traditional
> serialization will die out in favor of more modern approaches, rendering
> the issue less relevant over time.  Admittedly, maybe a very long period
> of time.
> #7) More loose coupling seems necessary and useful
> In order for typical applications to function with modern middleware as
> modules, without compromising security, it may be necessary to enhance
> the loose coupling mechanism (uses/provides), or to provide an
> additional, similar mechanism, which allows a symbolic coupling which
> would allow modules to declare (in an abstract manner) modules which
> need to have reflective access to it in a "friendly" manner.
> Implementation ideas are forthcoming on jigsaw-dev.
> Providing a way within the system to grant public and/or private
> reflection access in a specific manner, and doing so in a manner which
> is easy to use and not prejudicial against existing or future middleware
> and small or large applications which consume such middleware in the
> Java SE or EE space, is our most basic requirement for satisfactory
> resolution of this issue.  I will follow up on jigsaw-dev with some
> implementation thoughts and more specific requirements.
> This list may not be exhaustive but it should be, at the least, a very
> good starting point for discussion, hopefully with a short path to
> implementation.
> On 09/12/2016 10:08 AM, Mark Reinhold wrote:
>> Issue summary
>> -------------
>>   #ReflectiveAccessToNonExportedTypes --- Some kinds of framework
>>   libraries require reflective access to members of the non-exported
>>   types of other modules; examples include dependency injection (Guice),
>>   persistence (JPA), debugging tools, code-automation tools, and
>>   serialization (XStream).  In some cases the particular library to be
>>   used is not known until run time (e.g., Hibernate and EclipseLink both
>>   implement JPA).  This capability is also sometimes used to work around
>>   bugs in unchangeable code.  Access to non-exported packages can, at
>>   present, only be done via command-line flags, which is extremely
>>   awkward.  Provide an easier way for reflective code to access such
>>   non-exported types. [1]
>>   #AwkwardStrongEncapsulation --- A non-public element of an exported
>>   package can still be accessed via the `AccessibleObject::setAccessible`
>>   method of the core reflection API.  The only way to strongly
>>   encapsulate such an element is to move it to a non-exported package.
>>   This makes it awkward, at best, to encapsulate the internals of a
>>   package that defines a public API. [2]
>> Proposal
>> --------
>> (Warning: This is somewhat long, and in the end it affects both `exports`
>>  and `requires` directives.)
>> Extend the language of module declarations with the concept of _weak
>> modules_.  Weak modules make it easy to modularize components whose
>> internals will be accessed by reflection-based frameworks.  Every
>> package in a weak module has the following properties:
>>   (A) It is exported at both compile time and run time, as if by an
>>       `exports` directive, and
>>   (B) Its non-public elements are available for _deep_ reflection, i.e.,
>>       at run time they can be made accessible to code outside the module
>>       via the `AccessibleObject::setAccessible` method of the core
>>       reflection API.
>> In other words, every type defined in a weak module, whether public or
>> not, is subject to exactly the same access checks as in Java SE 8 and
>> earlier releases.
>> A weak module is declared by placing the modifier `weak` before the
>> `module` keyword.  The declaration of a weak module cannot contain any
>> explicit `exports` directives.  If the `weak` modifier does not appear
>> before the `module` keyword then the declared module is _strong_, and
>> it can contain explicit `exports` directives.
>> Suppose we have a module `` which has an internal package
>> `` that contains entity classes to be manipulated by
>> Hibernate, via core reflection.  Then the module declaration
>>     weak module {
>>         // No exports
>>         requires hibernate.core;
>>         requires hibernate.entitymanager;
>>     }
>> exports the public types in ``, and those of any other
>> packages, in all phases.  It additionally makes all non-public elements
>> of all packages available for deep reflection, enabling Hibernate to
>> access such elements in the `` package via the
>> `setAccessible` method.
>> Weak modules simplify the process of migrating to modules.  The steps
>> to convert an existing component into a module were, previously:
>>   (1) Make any changes necessary to get it working as an automatic
>>       module (e.g., eliminate duplicate packages), and then
>>   (2) Write an explicit module declaration, which entails identifying
>>       both the component's dependences (`requires`) and the packages
>>       whose public types are to be made available to other modules
>>       (`exports`).
>> With weak modules we can now divide the second step into two steps:
>>   (2a) Write an explicit module declaration for a weak module, which
>>        entails identifying just the component's dependences (`requires`).
>>   (2b) Convert the weak module into a strong module, which entails
>>        identifying the packages of the component whose public types
>>        are to be made available to other modules (`exports`).
>> In other words, weak modules make it possible to focus first upon the
>> reliable configuration of a module (`requires`), and then later think
>> about its strong encapsulation (`exports`).
>> Weak modules are "weak" in what they export, but they remain subject
>> to all of the constraints required to achieve reliable configuration.
>> They do not read the unnamed module (i.e., the class path), they do not
>> allow cycles in the module graph, and they do not allow split packages.
>> Weak modules read named modules only as indicated by their `requires`
>> directives, and they consume and provide services only as indicated by
>> their `uses` and `provides` directives.
>>                                   * * *
>> In a strong module, an ordinary `exports` directive exports a package at
>> both compile time and run time (property (A) above) but does not make its
>> non-public types available for deep reflection (B).  In order to enable a
>> package in a strong module to be exported in the same way as in a weak
>> module we introduce the per-export modifier `private` to denote this
>> second property.
>> If the above weak `` module, e.g., contains some other packages
>> besides ``, and we wish to encapsulate those packages,
>> we can convert it into a strong module with the declaration
>>     module {
>>         exports private;
>>         requires hibernate.core;
>>         requires hibernate.entitymanager;
>>     }
>> Now Hibernate can still access any public or non-public entity classes in
>> the `` package, but all the other packages are strongly
>> encapsulated.
>> The `private` modifier should generally not be used to export a package
>> containing an API, since normally an API's internal implementation
>> details should be strongly encapsulated.  It may, however, be useful for
>> legacy APIs whose internals are known to be accessed by existing code.
>> Every package in a weak module, an automatic module, or an unnamed module
>> is exported as if by an `exports private` directive.
>> To ensure the integrity of the platform we expect few, if any, packages
>> in the JDK itself to be exported with the `private` modifier.
>>                                   * * *
>> The new `private` modifier can also be used with qualified exports,
>> though they interact with unqualified exports in a non-obvious way.
>>   - If you write `exports p` then you can also write `exports private
>>     p to m`, so that code in module `m` can access the non-public types
>>     of `p` via deep reflection but code outside of `m` can only access
>>     the public types of `p`.
>>   - If you write `exports p to m1` then you can also write `exports
>>     private p to m2`, so that code in `m2` can access the non-public
>>     types of `p` via deep reflection, code in `m1` can access the
>>     public types of `p`, but no code in any other module can access
>>     any of the types of `p`.
>>   - If you write `exports private p` then you cannot also have a
>>     qualified export of `p`, since code in all other modules already
>>     has access to the non-public types of `p` via deep reflection.
>> Put informally, you can give your friends additional access, but you
>> can't discriminate against them by giving them less access than everyone
>> else.
>> As before, duplicate `exports` directives are not permitted, in order to
>> ensure easy readability.  At most one `exports` directive is relevant to
>> any given package/module pair, and it's easy to determine which one.
>>                                   * * *
>> The introduction of `private` as a modifier of `exports` directives calls
>> the existing syntax of `requires public` even more strongly into question
>> than before.  A module declaration of the form
>>     module {
>>         exports private;
>>         requires public java.sql;
>>     }
>> is likely to be very confusing to an uninformed reader.  The `private`
>> modifier in the `exports` directive means that the private elements of
>> the `` package are exported for deep reflection at run
>> time.  The `public` modifier in the `requires` directive, however, does
>> not mean that the public elements of the `java.sql` module are needed by
>> this module; that is true of any plain `requires` directive.  It means
>> that, additionally, any client of this module is granted implied
>> readability to the `java.sql` module, thereby gaining access to all of
>> its exported types.
>> To reduce this confusion we rename the `public` modifier in `requires`
>> directives to `transitive`.  Thus the above example becomes
>>     module {
>>         exports private;
>>         requires transitive java.sql;
>>     }
>> This is potentially confusing in a different way, since in mathematics
>> the term "transitive" is usually applied to an entire relation rather
>> than to three specific elements of a set.  Its use here does not, in
>> particular, mean that the resolver does not interpret plain `requires`
>> directives when computing the transitive closure of a set of root
>> modules.  "Transitive" as used here is in the more abstract sense,
>> expressing the notion of conveying a property -- in this case, the
>> readability of the required module -- from one thing to another.
>> Notes
>> -----
>>   - This is significantly different from the first proposal [3].  It adds
>>     the notion of weak modules, to ease migration, and also the notion of
>>     exporting a package without enabling deep reflection, to strengthen
>>     encapsulation.
>>   - This proposal removes the notion of dynamic exports, which in the
>>     presence of private exports would introduce considerable complexity
>>     into the interactions between qualified and unqualified exports.
>>     This means that it is no longer possible to export a package only at
>>     run time, so it is no longer possible for the author of a module to
>>     express the intent that the types of a non-API package are meant to
>>     be available to frameworks for deep reflection at run time but
>>     inaccessible at compile time.  The dynamic-export feature could, if
>>     needed, be added in a future release.
>>   - A strong module with no exports makes no types accessible to code in
>>     other modules while a weak module makes all of its types accessible,
>>     both directly and via deep reflection.  The declarations of such
>>     modules are, however, visually similar since most of their text lies
>>     between the curly braces:
>>         module m1 {
>>             requires ...;
>>             uses ...;
>>             provides ...;
>>         }
>>         weak module m2 {
>>             requires ...;
>>             uses ...;
>>             provides ...;
>>         }
>>     We suspect that this visual similarity will not cause much confusion
>>     in practice since strong modules that export no packages will be very
>>     rare.
>>   - If a container is to ensure that a package in an application module
>>     is available for deep reflection only by a trusted framework then it
>>     can arrange for that by rewriting that module's descriptor, as
>>     suggested previously [4], to insert the appropriate qualified private
>>     export.  If there is a possibility that two modules will need to
>>     export a package of the same name to the same framework module, as
>>     suggested by Jason Greene [5], then the container should instead
>>     inject a small class into each module whose static initializer
>>     invokes the `Module::addExports` method in order to export the
>>     package to the framework module.  There is no need any longer for
>>     the resolution algorithm to take this scenario into account [6].
>>   - This proposal primarily addresses "friendly" uses of reflection, such
>>     as dependency injection and persistence, in which the author of a
>>     module knows in advance that one or more packages must be exported at
>>     run time for deep reflective access by frameworks.  Intrusive access
>>     to arbitrary packages of arbitrary modules by, e.g., serialization
>>     frameworks or debugging tools, will still require the use of sharp
>>     knives such as the `--add-exports` command-line option, the legacy
>>     unsupported `sun.misc.Unsafe` API and related APIs, or JVM TI.
>>   - Using the `--add-exports` option or its equivalent remains awkward,
>>     and sometimes it's the only way out.  To ease migration it's worth
>>     considering some way for an application packaged as a JAR file to
>>     include such options in its `MANIFEST.MF` file, as suggested by Simon
>>     Nash [7].  This is tracked as #AddExportsInManifest [8].
>> [1]
>> [2]
>> [3]
>> [4]
>> [5]
>> [6]
>> [7]
>> [8]


More information about the jpms-spec-experts mailing list