From glyn_normington at UK.IBM.COM  Fri Jun  1 05:47:53 2007
From: glyn_normington at UK.IBM.COM (Glyn Normington)
Date: Fri, 01 Jun 2007 13:47:53 +0100
Subject: Module isolation
In-Reply-To: <465F5198.2090805@oracle.com>
Message-ID: <OFD20EE2C8.4BD4A1F7-ON802572ED.00360624-802572ED.00464DBD@uk.ibm.com>

Bryan Atsatt <bryan.atsatt at ORACLE.COM> wrote on 31/05/2007 23:52:08:

> I'd like to continue this discussion separately from the
> 291/import-by-package thread :^).
>
> To recap, the spec (sections 1.2, 6.1, 6.4) talks about using separate
> Repository instances as a means to provide isolation for environments
> like EE and Applets.
>
> I believe this is an oversight--actually, a holdover from the ideas I
> put forth in the prototype, where Repository cached Module instances.
>
> But in the current design, that is no longer the case. Repositories hold
> ModuleDefinitions, not Modules, so they are not a suitable isolation
> mechanism. It is *Module* instances with their associated loaders that
> may require isolation.

Yes, I think you're right.

>
> The current design also assumes that ModuleSystems cache Module
> instances. This certainly makes more sense than Repository. However, it
> means that systems that require isolation must arrange for separate
> ModuleSystem instances. This seems quite awkward, and a poor
abstraction.
>
> So I think we need to introduce an explicit notion of an isolation
> context that:
>
> 1. Acts as a cache for Module instances.
> 2. Enables multiple instances, each forming a separate Module namespace.
> 3. Provides out-of-the box instances for bootstrap and "system" modules.
> 4. Provides a delegation model to link instances.
> 5. Provides a lookup mechanism identical to that of Repository.
>
> Any given resolution would take place within a single context. That is,
> an initial context is selected, and while existing instances can be
> retrieved from delegated contexts, *new* instances must be created in
> the initial context.

This seems like a module analogue of a class loader except that new module
instances are created in the initial context, which seems a bit peculiar.
Wouldn't this mean that Java platform module instances could get created
in an application context?

>
> To enforce this model, and to simplify the api, I believe the context
> should directly expose a "resolve" convenience method (which simply
> delegates the actual implementation).
>
> Here is a strawman API, with a simple parent-first delegation model:
>
> public abstract class ModuleContext {
>
>      // Get the context used to resolve JRE modules.
>      public static ModuleContext getBootstrapContext(){...};
>
>      // Get the context used to resolve the main module.
>      public static ModuleContext getSystemContext(){...};
>
>      // Get all contexts.
>      public static List<ModuleContext> getContexts() {...};
>
>      // Add a new context.
>      public static void addContext(ModuleContext ctx) {...}
>
>      // Remove a context (error if == default).
>      public static boolean removeContext(ModuleContext ctx) {...}
>
>      // Get the parent context (null if bootstrap).
>      public ModuleContext getParentContext(){...}
>
>      // Get the name of this context.
>      public String getContextName() {...}
>
>      // Add a new Module instance to the cache. Throw an error
>      // if an existing module contains the same definition.
>      public abstract void addModule(Module module);
>
>      // Find cached Module instances. Must check parent first.
>      public abstract List<Module> findModules(Query query);
>
>      // Remove a Module.
>      public abstract boolean remove(Module module);
>
>      // Remove all Modules created from the specified definition.
>      public static void removeAll(ModuleDefinition def) {...}
>
>      // Resolve a definition.
>      public Module resolve(ModuleDefinition def) {
>          ModuleSystem ms = def.getModuleSystem();
>          return ms.resolve(def, this);
>      }

Using module context as a resolution context seems to imply that
concurrent resolutions need to be serialised to avoid interference and so
some synchronisation needs adding. Or maybe the idea is to use a module
context instance per resolution, but that wouldn't fit with a module
context being a cache spanning multiple resolutions.

>
>      // Set the context used for JRE modules.
>      static void setBootstrapContext(ModuleContext ctx){...}
>
>      // Set the context used to define the main module.
>      static void setSystemContext(ModuleContext ctx){...}
> }
>
> The JVM would create an appropriate subtype and call
> setBootstrapContext(). JRE module instances would be cached here.
>
> The launcher would create a child context and call setSystemContext().
> The main module and any of its dependencies would be resolved here.
>
> An EE/Applet (or similar) environment can create/remove/use new contexts
> as needed for application isolation. Note that the these new contexts
> need not have the system context in their parent chain. But they do need
> the bootstrap context so the code should enforce that.
>
> The resolve() method here assumes that ModuleSystem.getModule() is
> replaced by:
>
> ModuleSystem {
>     ...
>     public abstract Module resolve(ModuleDefinition def,
>                                    ModuleContext ctx);
> }
>
> The ModuleSystem is no longer required to directly cache Modules, but to
> implement the resolution algorithm using the supplied context.
>
> ModuleSystem releaseModule() and disableModuleDefinition() methods just
> use remove() and removeAll(), respectively.
>
> This class could easily be made concrete, as the implementations are
> obviously simple. Regardless, for extensibility the actual types used at
> runtime should be under the control of the JVM/JRE and those systems
> that need to create new contexts.
>
> And, we might want to tie other existing ideas in here. For example,
> ModuleContext could be the home for "default" policies:
>
>     public abstract ImportPolicy getImportPolicy();
>     public abstract ImportOverridePolicy getImportOverridePolicy();
>
> Thoughts?

As I've pointed out on this list before, custom import policies have many
disadvantages including making it very hard to write management systems
that can infer dependencies before modules are installed in the module
system. Moving custom import policies out of individual modules into the
repository (which came up in discussion with Stanley and Michal at
JavaOne) or module context, leaving purely declarative metadata behind in
individual modules, would solve this problem. Then the default policy
would simply be the one returned by default. (I guess the same is true for
custom import override policies, which I'm also not fond of, particularly
at the module level.)

>
> // Bryan

Glyn





Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU





-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/jsr277-eg-observer/attachments/20070601/3268311a/attachment-0001.html 

From bryan.atsatt at oracle.com  Fri Jun  1 14:00:09 2007
From: bryan.atsatt at oracle.com (Bryan Atsatt)
Date: Fri, 01 Jun 2007 14:00:09 -0700
Subject: Module isolation
In-Reply-To: <OFD20EE2C8.4BD4A1F7-ON802572ED.00360624-802572ED.00464DBD@uk.ibm.com>
References: <OFD20EE2C8.4BD4A1F7-ON802572ED.00360624-802572ED.00464DBD@uk.ibm.com>
Message-ID: <466088D9.9020903@oracle.com>

Glyn Normington wrote:
>
> *Bryan Atsatt <bryan.atsatt at ORACLE.COM>* wrote on 31/05/2007 23:52:08:
>
>  > I'd like to continue this discussion separately from the
>  > 291/import-by-package thread :^).
>  >
>  > To recap, the spec (sections 1.2, 6.1, 6.4) talks about using separate
>  > Repository instances as a means to provide isolation for environments
>  > like EE and Applets.
>  >
>  > I believe this is an oversight--actually, a holdover from the ideas I
>  > put forth in the prototype, where Repository cached Module instances.
>  >
>  > But in the current design, that is no longer the case. Repositories hold
>  > ModuleDefinitions, not Modules, so they are not a suitable isolation
>  > mechanism. It is *Module* instances with their associated loaders that
>  > may require isolation.
>
> Yes, I think you're right.
>
>  >
>  > The current design also assumes that ModuleSystems cache Module
>  > instances. This certainly makes more sense than Repository. However, it
>  > means that systems that require isolation must arrange for separate
>  > ModuleSystem instances. This seems quite awkward, and a poor abstraction.
>  >
>  > So I think we need to introduce an explicit notion of an isolation
>  > context that:
>  >
>  > 1. Acts as a cache for Module instances.
>  > 2. Enables multiple instances, each forming a separate Module namespace.
>  > 3. Provides out-of-the box instances for bootstrap and "system" modules.
>  > 4. Provides a delegation model to link instances.
>  > 5. Provides a lookup mechanism identical to that of Repository.
>  >
>  > Any given resolution would take place within a single context. That is,
>  > an initial context is selected, and while existing instances can be
>  > retrieved from delegated contexts, *new* instances must be created in
>  > the initial context.
>
> This seems like a module analogue of a class loader except that new
> module instances are created in the initial context, which seems a bit
> peculiar. Wouldn't this mean that Java platform module instances could
> get created in an application context?


Argh. You're right, and this isn't appropriate for normal sharing. I was
thinking about this from the other perspective: isolation, where new
Module instances should end up in the new context.

But even there, we don't want to allow bootstrap Modules to be
re-instantiated (although this would be useful in some cases, if it
could be made to work).

Hmm. In my original model, binding a Module to a "context" (Repository,
in that case) was natural because, like Class.getClassLoader(),
ModuleDefinition.getRepository() allowed you to get the "defining"
context easily, so recursive resolution could always use the correct
context.

Given the current design, my premise was that it doesn't seem to make
any sense to bind a Module cache directly to a Repository when
Repository.find() returns ModuleDefinition. And ModuleSystem.getModule()
requires caching behavior, so I think this might have thrown me off track.

I'm guessing now that I've just not thinking about this the right way
(and the spec is misleading?). I had forgotten a conversation Stanley
and I had on this topic, which may be lurking somewhere between the
lines of the spec.

Stanley's idea was that a Repository implementation could return a
*copy* of a ModuleDefinition created by a different Repository. Such a
Repository would in fact form an isolation context. And caching Module
instances becomes trivial: each definition can just have one as a field.

While there are definite lifecycle issues here between such repository
instances, they are probably solvable.

And then we really *do* have an exact analogue of the loader delegation
model, in which:

    Repository       ~= ClassLoader
    ModuleDefinition ~= Class
    Module           ~= Object

Isolation of classes requires different Class instances. So isolation of
  modules requires different ModuleDefinition instances.

But this puts us back awfully close to my original design.

>
>  >
>  > To enforce this model, and to simplify the api, I believe the context
>  > should directly expose a "resolve" convenience method (which simply
>  > delegates the actual implementation).
>  >
>  > Here is a strawman API, with a simple parent-first delegation model:
>  >
>  > public abstract class ModuleContext {
>  >
>  >      // Get the context used to resolve JRE modules.
>  >      public static ModuleContext getBootstrapContext(){...};
>  >
>  >      // Get the context used to resolve the main module.
>  >      public static ModuleContext getSystemContext(){...};
>  >
>  >      // Get all contexts.
>  >      public static List<ModuleContext> getContexts() {...};
>  >
>  >      // Add a new context.
>  >      public static void addContext(ModuleContext ctx) {...}
>  >
>  >      // Remove a context (error if == default).
>  >      public static boolean removeContext(ModuleContext ctx) {...}
>  >
>  >      // Get the parent context (null if bootstrap).
>  >      public ModuleContext getParentContext(){...}
>  >
>  >      // Get the name of this context.
>  >      public String getContextName() {...}
>  >
>  >      // Add a new Module instance to the cache. Throw an error
>  >      // if an existing module contains the same definition.
>  >      public abstract void addModule(Module module);
>  >
>  >      // Find cached Module instances. Must check parent first.
>  >      public abstract List<Module> findModules(Query query);
>  >
>  >      // Remove a Module.
>  >      public abstract boolean remove(Module module);
>  >
>  >      // Remove all Modules created from the specified definition.
>  >      public static void removeAll(ModuleDefinition def) {...}
>  >
>  >      // Resolve a definition.
>  >      public Module resolve(ModuleDefinition def) {
>  >          ModuleSystem ms = def.getModuleSystem();
>  >          return ms.resolve(def, this);
>  >      }
>
> Using module context as a resolution context seems to imply that
> concurrent resolutions need to be serialised to avoid interference and
> so some synchronisation needs adding. Or maybe the idea is to use a
> module context instance per resolution, but that wouldn't fit with a
> module context being a cache spanning multiple resolutions.

Well, regardless of the resolution mechanism, there must be some
serialization in the process, else how will sharing occur? If two
modules both depend on X, and X hasn't been instantiated, we must ensure
that only 1 thread does the instantiation, and any others see the result.

>
>  >
>  >      // Set the context used for JRE modules.
>  >      static void setBootstrapContext(ModuleContext ctx){...}
>  >
>  >      // Set the context used to define the main module.
>  >      static void setSystemContext(ModuleContext ctx){...}
>  > }
>  >
>  > The JVM would create an appropriate subtype and call
>  > setBootstrapContext(). JRE module instances would be cached here.
>  >
>  > The launcher would create a child context and call setSystemContext().
>  > The main module and any of its dependencies would be resolved here.
>  >
>  > An EE/Applet (or similar) environment can create/remove/use new contexts
>  > as needed for application isolation. Note that the these new contexts
>  > need not have the system context in their parent chain. But they do need
>  > the bootstrap context so the code should enforce that.
>  >
>  > The resolve() method here assumes that ModuleSystem.getModule() is
>  > replaced by:
>  >
>  > ModuleSystem {
>  >     ...
>  >     public abstract Module resolve(ModuleDefinition def,
>  >                                    ModuleContext ctx);
>  > }
>  >
>  > The ModuleSystem is no longer required to directly cache Modules, but to
>  > implement the resolution algorithm using the supplied context.
>  >
>  > ModuleSystem releaseModule() and disableModuleDefinition() methods just
>  > use remove() and removeAll(), respectively.
>  >
>  > This class could easily be made concrete, as the implementations are
>  > obviously simple. Regardless, for extensibility the actual types used at
>  > runtime should be under the control of the JVM/JRE and those systems
>  > that need to create new contexts.
>  >
>  > And, we might want to tie other existing ideas in here. For example,
>  > ModuleContext could be the home for "default" policies:
>  >
>  >     public abstract ImportPolicy getImportPolicy();
>  >     public abstract ImportOverridePolicy getImportOverridePolicy();
>  >
>  > Thoughts?
>
> As I've pointed out on this list before, custom import policies have
> many disadvantages including making it very hard to write management
> systems that can infer dependencies before modules are installed in the
> module system. Moving custom import policies out of individual modules
> into the repository (which came up in discussion with Stanley and Michal
> at JavaOne) or module context, leaving purely declarative metadata
> behind in individual modules, would solve this problem. Then the default
> policy would simply be the one returned by default. (I guess the same is
> true for custom import override policies, which I'm also not fond of,
> particularly at the module level.)

Yep, I have also suggested moving this kind of logic to Repository. I
don't much like bundled ImportPolicies, even though Stanley has defended
most of the issues well, simply because of the extra complexity.
>
>  >
>  > // Bryan
>
> Glyn
>
>
> ------------------------------------------------------------------------
>
> /
> /
>
> /Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU/
>
>
>
>
>
>

From bryan.atsatt at oracle.com  Fri Jun  1 14:48:24 2007
From: bryan.atsatt at oracle.com (Bryan Atsatt)
Date: Fri, 01 Jun 2007 14:48:24 -0700
Subject: Exported resources
In-Reply-To: <465F68C8.4000103@sun.com>
References: <4654109D.40205@sun.com> <46548736.7020507@oracle.com>
	<465629EA.2050008@sun.com> <465658CE.5020409@oracle.com>
	<465DCE51.4010304@sun.com> <465DDCD0.8090703@oracle.com>
	<465DF6D6.5050608@sun.com> <465E0201.2030308@oracle.com>
	<465F68C8.4000103@sun.com>
Message-ID: <46609428.4000909@oracle.com>

Hi Stanley,

Sorry if I'm not being clear. Let me try to summarize:

- I *do* want 277 to support private (non-exported) resources, but

- I do not think we should use permissions as the enforcement mechanism.

- I believe the 'caller-class-is-member' enforcement mechanism is
sufficient, and

- I do realize that this means ResourceBundle.getBundle() (or similar)
will fail to find private resources; these must be made public.


Let me give more detailed reasoning for this position...

You are of course correct that a class loader is free to add permissions
to any class by assigning a ProtectionDomain with those permissions
during defineClass(). (Let's call these "hard-wired" permissions.)

So permission to access private resources could easily be hard-wired for
other classes *in the same module*.

To this point, there really isn't any meaningful difference between
using permissions or using my approach: both allow module classes to
directly access their private resources.

But if we want to grant that same permission to *any* class outside of
the module, it gets far more interesting. And it is this problem that
I'm concerned with.

For ResourceBundle.getBundle(), for example, to access a private
resource, we must grant permission to the *ResourceBundle* class. And
how do we do that?

We could hard-wire permission for ResourceBundle, but... what about
ServiceLoader? Ok, so let's just hard-wire permission for any JRE module.

Problem solved? Not at all: what about all the non JRE frameworks out in
the world that need the same kind of access?

Clearly we cannot hard-wire permissions for these. So now we need to use
policy file entries to do so. We could of course take this approach, but
it either leaves some poor admin running around to update policy files
as things break, or some fancy mechanism to grant access during
framework module install. Ick.

And, hard-wired or not, permissions *are* a friend model.

But 294 is not likely to provide such a model for classes.

So, even if we hard-wire permission for ResourceBundle to access a
private resource, how is it going to access a private ListResourceBundle
class? Or how will the ServiceLoader gain access to the service provider
class? How will third-party frameworks solve this problem?

It is hard for me to see how all this complexity is justified, simply to
hide resources that have never been hidden before.

This doesn't mean there is no place for private resources! It just means
that they can only be used by the module classes themselves. And this is
still a very valuable feature: developers can restrict access to
resources that are strictly implementation details.

// Bryan

Stanley M. Ho wrote:
> Hi Bryan,
>
> Bryan Atsatt wrote:
>> ...
>> No, I don't agree. That's what I said before :^). I *really* don't like
>> the idea that I'm going to have to explicitly grant permission to some
>> module to access my resource. This *significantly* complicates
>
> No, this was not what I suggested. If some other modules want to access
> your resource, your resources need to be exported. However, if you want
> to access your own private resource in your module (call
> ClassLoader.getResource*() directly or call ResourceBundle.getBundle(),
> etc.), then your code would require additional permissions.
>
>> deployment. How will the grant occur, especially given that each
>> environment can use an entirely different permission storage mechanism?
>>
>> Or are you thinking that the loaders in the module system will construct
>> ProtectionDomains pre-wired with permissions? If so, exactly which
>> entities will be blessed? Just the JRE code? What about all the existing
>> non-JRE frameworks in use?
>
> When your module instance is initialized, the module system can
> construct the ProtectionDomain with the appropriate permissions and
> associate that with the classes in your module, so your classes in your
> module will always have the necessary permissions to access your own
> resources.
>
>> Sure, resources have never had JVM access control applied to them. But I
>> don't think we should go to the other extreme (permissions) just so that
>> I can give "friend" access to private resources. I'd much prefer that
>> such resources simply be exported.
>>
>> And I don't see this as a particular problem, either. We've gotten along
>> just fine without private resources so far (or maybe there is some big
>> demand for this that I'm missing?).
>>
>> And what about the mismatch issue I brought up? Lots of existing
>> frameworks use the "services" pattern, in which:
>>
>> 1. A well known resource name is used in a getResource() call.
>> 2. The resource contains a string naming a provider class.
>> 3. The provider class is loaded using Class.forName().
>>
>> This is a simple but very powerful pattern, but it requires that *both*
>> the resource and the named class be accessible to the framework. If we
>> require a permission grant for the resource, but not for the class, it
>> will be very easy to get access to one and not the other. Is it really
>> worth opening up this potential headache to enable a friend model for
>> resources?
>>
>> // Bryan
>
> Note that the security permission approach can be abused if a module has
> all permissions, it can access even the private resources in other
> modules. This is the friend access as you mentioned, but this is not my
> intention of using the security permission approach.
>
> Sorry, I have to go and I can't response in details here. Are you saying
> that we should probably give up the idea of exported/private resources
> and make all resources in a module to be always public and accessible?
> Or are you saying that you still like the idea of exported/private
> resources but you don't like to address it using the security permission
> approach?
>
> - Stanley
>

From bryan.atsatt at oracle.com  Mon Jun  4 11:38:18 2007
From: bryan.atsatt at oracle.com (Bryan Atsatt)
Date: Mon, 04 Jun 2007 11:38:18 -0700
Subject: Import constraints
Message-ID: <46645C1A.2090809@oracle.com>

Currently the spec provides a somewhat odd mix of functionality here...

1. Module name. We have declaration syntax (superpackage name) and
runtime support for using a module name as a constraint
(ImportDependency class and Query.name()).

2. Versions. We have declaration syntax (VersionConstraint annotation)
and runtime support for using a collection of version/version-ranges as
a constraint (VersionConstraint/ImportDependency classes, and
Query.version()).

3. Attributes. We have declaration syntax for *assigning* attributes to
a definition (ModuleAttributes annotation), but no declaration syntax
for using them as constraints (a custom ImportPolicy is required). We
have runtime support for using them as constraints (Query.attribute()).

4. Extensible constraints. We have no declaration syntax, but we *do*
have runtime support for expressing any type of constraint and combining
them with boolean operators (Query).


I'd like to emphasize that Query should be seen *as the runtime
representation of a constraint*, and that:

     Any import declaration must be expressible as a Query.

This includes the module name (and potentially a package name, TBD).

This last is an important point: other than optimizing searches, a
module name is not special! It's just another Query element (ditto for a
package name Query), evaluated along with any others in a compound Query
(i.e. Query elements combined with boolean operators). A definition can
even be looked up *without* a module name, using nothing but attribute
matching.

When I initially designed Query (and Repository.find()), I wanted
something very flexible and extensible, with the expectation that it
could be used to represent the entire constraint model. But I don't
think we're using it as effectively as we can...

Ok, so?

Well, first, I think we ought to generalize the runtime dependency type
to fully leverage Query. So change ImportDependency to:

public class ImportDependency extends Query {
     private Query constraint;
     private boolean optional;
     private boolean reExport;

     public ImportDependency (Query constraint,
                              boolean optional,
                              boolean reExport) {
         this.constraint = constraint;
         this.optional = optional;
         this.reExport = reExport;
     }

     public boolean match(ModuleDefinition target) {
         return constraint.match(target);
     }

     public boolean isOptional() {
         return optional;
     }

     public boolean isReExported() {
         return reExport;
     }

     public String toString() {
         ...
     }
}

This way, we rely on the flexibility of Query to express *all*
constraints, and have a single concrete type to represent dependencies.

By having Dependency extend Query, instances can be directly passed to
Repository.find(); no conversion required. We could obviously just make
a getter for the Query if we want, but why not make this convenient to use?

And, more importantly, ModuleDefinition.getImportDependency()
implementations are free to return any kind of constraints they want,
simply by varying the Query.

Module name, if present, is just a Query element. Again, I think the
only place this (and/or package name) is explicitly needed is to enable
search *optimization*: if we want to avoid linear searches in
Repository.findModuleDefinition(Query) implementations, there needs to
be a way to narrow the search efficiently. But it's just an
optimization, and a topic for a different thread.


Second, I think we should eliminate VersionConstraint as a separate
class. It's functionality is easy to replace with Query elements that
match Version or VersionRange, strung together with appropriate boolean
operators.


Third, we should add a getExportedPackages() method to ModuleDefinition.
This way a Query can easily be created to search by package name.
Whether or not we choose to surface import-by-package declaratively, it
should at least be possible for a ModuleSystem to use such a Query.


Fourth, we should add support for simple attribute constraint
declarations. Enabling simple equality constraints ought to be easy
enough: an AttributeConstraint annotation that takes key/value lists
(key1==value1;key2==value2;...). We already have support for expressing
these as Query instances.

This simple type of attribute constraint is supported by OSGi. I don't
have a good sense of how common its usage is (Glyn/Richard?), but some
will likely view the absence of this support in 277 as an
interoperability issue.

More complex attribute comparisons could still be left up to custom
import policies. Or...


<dreaming-out-loud>

We could instead create a generic *query* declaration syntax, with
support for expressing module-name, package-name, version,
version-range, attribute matching, as well as boolean operators and
precedence.

This would eliminate the need for VersionConstraint/AttributeConstraint
annotations. The current VersionConstraint syntax using ";" separators
would no longer be required, as the boolean operators could be used instead.

And the expressiveness might reduce or eliminate the need for many
custom import policies.

</dreaming-out-loud>

Finally, Query ought to have a toString() implementation so that we can
log them, examine in a debugger, put them in error messages, etc.


Regardless of the approach, it would certainly be nice to provide some
symmetry between our declarative syntax and our runtime capabilities.

Thoughts?

// Bryan

From bryan.atsatt at oracle.com  Mon Jun  4 12:42:40 2007
From: bryan.atsatt at oracle.com (Bryan Atsatt)
Date: Mon, 04 Jun 2007 12:42:40 -0700
Subject: JSR 277 EG observer mailing list
In-Reply-To: <4648BA20.5070206@sun.com>
References: <46400CFB.1020004@sun.com> <46412161.9020902@sun.com>
	<4648BA20.5070206@sun.com>
Message-ID: <46646B30.2030500@oracle.com>

How are people supposed to find this? Shouldn't there be an easily
spotted link somewhere on the public 277 page?

// Bryan

Stanley M. Ho wrote:
> Hi JSR 277 experts,
>
> Just an update. This EG mailing list has been hooked up with the
> observer list over the weekend, and you will find the public mailing
> list archive here:
>
> http://mail.openjdk.java.net/pipermail/jsr277-eg-observer/
>
> - Stanley
>

From bryan.atsatt at ORACLE.COM  Mon Jun  4 12:21:29 2007
From: bryan.atsatt at ORACLE.COM (Bryan Atsatt)
Date: Mon, 04 Jun 2007 12:21:29 -0700
Subject: Relationship to JSR 291 [was: Re: Bryan's comments]
In-Reply-To: <465F6157.7070706@sun.com>
References: <OF62C11B58.0C61920A-ON802572EB.0055969A-802572EB.005897E5@uk.ibm.com>
	<465DF2E4.4090603@sun.com> <465E2051.7020008@oracle.com>
	<465EBE00.2060306@ungoverned.org> <465F0D8C.5010606@oracle.com>
	<465F6157.7070706@sun.com>
Message-ID: <46646639.7080606@oracle.com>

Stanley M. Ho wrote:
> Hi Bryan,
>
> I don't think I will have time to response to all your comments this
> week, so my response would focus on the import-by-package discussion.
>
> First, the only import dependency we have consensus in this EG so far is
> import-by-module-name (or import-by-superpackage-name), and this was
> also what's in the EDR. That said, it is by no mean that we can't
> consider other import dependency granularities (e.g.
> import-by-package-name, or import-by-contract) in addition to
> import-by-module-name. I would like to focus this discussion more on
> whether it makes sense for JSR 277 to support additional import
> dependency granularity, mainly import-by-package-name, and if so, in
> what context.
>
> There are three cases we should consider:
>
> 1. 277 module imports another 277 module by package name
>
> I think our consensus so far is that this is nice-to-have or
> non-critical, given we already have support for import-by-module-name.
> To keep this discussion in focus, let's not dive into
> import-by-package-name v.s. import-by-module-name, and let's assume
> there is no support for #1 for now.

Agreed.

>
> 2. OSGi module imports 277 module by package name
>
> As I discussed this with Richard in previous emails, majority of the
> works for this fall under the OSGi framework if it wants to enable this
> kind of wiring, but we could probably make the implementor's life easier
> by provide the appropriate 277 reflective API if necessary.

Yes, the API would be useful.

>
> 3. 277 module imports OSGi module by package name
>
> If I understand you correctly, I think what you really want is the
> support for this case, so the existing OSGi bundles can be leveraged by
> 277 modules in a nature way.
>
> Let me say it up front that I don't oppose the idea of #3 in the context
> of interoperability. If #3 is what you really want, I think where we
> diverge is simply how to make it possible.
>
> Do you agree so far?

Yes.

>
> Bryan Atsatt wrote:
>> ...
>> I want 277 to explicitly expose the concept of import-by-package through
>> appropriate APIs.
>>
>> It would be very nice if the 277ModuleSystem implementation itself
>> directly supports import-by-package (case #3). But this is less critical
>> to me than case #4...
>>
>> I want a 277ModuleDefinition to be able to import-by-package an
>> OSGiModuleDefinition (i.e. a bundle exposed via the 277 APIs). I believe
>> this case will be a particularly important one. By the time SE 7 ships,
> ...
>
> As I described several emails back:
>
>  > This doesn't mean supporting the notion of import by package is
>  > impossible. What OSGi will expose from its repository is simply module
>  > definitions, and module definition is basically an abstraction.
>  > Therefore, it should be possible to treat an OSGi bundle as a module
>  > definition, as well as to treat each exported package in an OSGi
>  > bundle a module definition as well. The dependency supported in JSR
>  > 277 is import-by-module. In the JSR 277 module system, this is mapped
>  > to import-by-superpackage. In the OSGi module system, this can be
>  > mapped to import-by-OSGi-bundle or import-by-package. If other module
>  > systems want to support different import granularity or if OSGi
>  > evolves to add new import granularity in the future, this model should
>  > still work.
>
> Rather than surfacing the import-by-package at the API level, I think we
> can solve #3 by generalizing the import dependency concept as
> import-by-name, and this name can be mapped to superpackage-name in JSR
> 277 and OSGi-bundle-name or OSGi-exported-package-name in OSGi (i.e.
> exposing OSGi bundles and exported packages through implementing
> OSGiBundleModuleDefinition and OSGiPackageModuleDefinition). I think
> this is also what you intended to say at one point (correct me if I
> misunderstand you) by suggesting to change
> ImportDependency.getModuleName() into something like
> ImportDependency.getName().

I believe we are now thinking along the same lines.

I just sent an email on the topic of generalizing the import dependency
concept, with the subject "Import constraints". With the approach I
suggest there, the distinctions between module-name and package-name are
mostly transparent to the runtime (declarative support for package-name
is not addressed).

That isn't to say that there aren't resolution issues to be worked out
for package-name, but its a start.

>
> We expect the module-aware compiler would somehow recognize the import
> dependency defined in 277. In other words, if the OSGi framework exposes
> the module definitions appropriately, you'll get the compiler support.
> Since the import dependency is generic, the compiler and other tools ed
> to do extra work to support other forms of import dependency.

Yep. As I've said from the very beginning, I think the compiler is going
to have to rely on the 277 runtime to resolve *external* (i.e.
non-source) dependencies. So as long as the runtime can deal with
package-name, the compiler will too.

(Dependencies resolvable via included source will, I assume, just
*directly* resolve, without applying any constraint logic. That is, if a
class refers to a class in another superpackage, and that class &
superpackage are available in the same compilation, the compiler will
simply select them without applying *any* constraints. This may be too
simple of a model, but maybe it is sufficient.)

>
> In case if the 277 module would like to import these OSGi exported
> packages using an OSGi-like resolution algorithm, it should be possible
> for it to use a custom import policy (or import override policy or maybe
> some new override mechanisms) to get the behavior it wants.
>
> Do you think this is a workable approach to address #3?

I don't think custom code should be required; we need to make it easier
than that or it doesn't qualify as "first-class" citizenship, at least
not to me :^).


I guess I am still missing something fundamental--I keep thinking that
import-by-package is actually not so hard. There are three main components:

1. Declaration.
2. Runtime infrastructure.
3. Resolution algorithm.

I'm pretty sure that #2 is easy (and covered in my "Import constraints"
thread).

And we don't yet have *any* import declaration mechanism (yes, we're
assuming that this will be handled in 294, but it hasn't been brought up
yet). I don't see this as a difficult issue either way. It would be
natural to have both import-by-name and import-by-package in
superpackages, if we have import at all. If we need to use annotations
instead, it is easy to have both types.

So I can only assume that the *real* concern here is #3. And I can't
tell if this is a valid concern or not. I hope Glyn and Richard will
point out what I'm missing here...

What is the difference between selecting a ModuleDefinition based on its
name, or selecting it based on its contents? Either way, "shallow"
validation must still occur.

// Bryan

>
> - Stanley
>

From Stanley.Ho at sun.com  Mon Jun  4 18:56:01 2007
From: Stanley.Ho at sun.com (Stanley M. Ho)
Date: Mon, 04 Jun 2007 18:56:01 -0700
Subject: Exported resources
In-Reply-To: <46609428.4000909@oracle.com>
References: <4654109D.40205@sun.com> <46548736.7020507@oracle.com>
	<465629EA.2050008@sun.com> <465658CE.5020409@oracle.com>
	<465DCE51.4010304@sun.com> <465DDCD0.8090703@oracle.com>
	<465DF6D6.5050608@sun.com> <465E0201.2030308@oracle.com>
	<465F68C8.4000103@sun.com> <46609428.4000909@oracle.com>
Message-ID: <4664C2B1.1050109@sun.com>

Hi Bryan,

I would like to get closures on a few open issues first, so my responses
will be limited to those threads rather than all the new threads that
have been started recently.

Bryan Atsatt wrote:
> Hi Stanley,
>
> Sorry if I'm not being clear. Let me try to summarize:
>
> - I *do* want 277 to support private (non-exported) resources, but

Good.

> - I do not think we should use permissions as the enforcement mechanism.
>
> - I believe the 'caller-class-is-member' enforcement mechanism is
> sufficient, and
>
> - I do realize that this means ResourceBundle.getBundle() (or similar)
> will fail to find private resources; these must be made public.
>
> Let me give more detailed reasoning for this position...
>
> You are of course correct that a class loader is free to add permissions
> to any class by assigning a ProtectionDomain with those permissions
> during defineClass(). (Let's call these "hard-wired" permissions.)
>
> So permission to access private resources could easily be hard-wired for
> other classes *in the same module*.
>
> To this point, there really isn't any meaningful difference between
> using permissions or using my approach: both allow module classes to
> directly access their private resources.
>
> But if we want to grant that same permission to *any* class outside of
> the module, it gets far more interesting. And it is this problem that
> I'm concerned with.
>
> For ResourceBundle.getBundle(), for example, to access a private
> resource, we must grant permission to the *ResourceBundle* class. And
> how do we do that?
>
> We could hard-wire permission for ResourceBundle, but... what about
> ServiceLoader? Ok, so let's just hard-wire permission for any JRE module.

The system classes are granted with all permissions, so they will have
sufficient permissions to access the private resources in a module. I
don't think this is a real issue.

> Problem solved? Not at all: what about all the non JRE frameworks out in
> the world that need the same kind of access?

I agreed that using other frameworks might be a concern.

> Clearly we cannot hard-wire permissions for these. So now we need to use
> policy file entries to do so. We could of course take this approach, but
> it either leaves some poor admin running around to update policy files
> as things break, or some fancy mechanism to grant access during
> framework module install. Ick.

If the framework is bundled as part of your module, it will have the
same set of permissions as the other code in your module, so it will
have sufficient permissions to allow your code to access the private
resources. If the framework is deployed through the typical mechanisms
(e.g. extension classpath, etc.), it will have all permissions, hence it
will also have sufficient permissions to access the private resources.
Further, if the framework is deployed as another module in one of the
system's repositories, it will likely have all permissions to allow your
code to access the private resources in your module as well. Moreover,
if the framework JAR/module is signed and is trusted, the system will
typically grant all permissions to the code in the framework when the
code is loaded. In other words, there is no need to explicitly grant
permissions in most scenarios.

The scenario where this may be an issue is when the framework is
deployed through some other means (e.g. other modules in a
URLRepository, JARs in custom classloaders) *and* the framework is
unsigned (or if the framework is signed but not trusted for some
reasons), it may not have sufficient permissions to allow your code to
access the private resources through the framework API. However, it is
unclear to me if there are frameworks deployed like this for this issue
to become a real problem. Do you have any example that shows this is a
real problem?

> And, hard-wired or not, permissions *are* a friend model.
>
> But 294 is not likely to provide such a model for classes.
>
> So, even if we hard-wire permission for ResourceBundle to access a
> private resource, how is it going to access a private ListResourceBundle
> class? Or how will the ServiceLoader gain access to the service provider
> class? How will third-party frameworks solve this problem?

In a typical classloader, all public and private classes are visible
(return from loadClass()) externally but only the public ones are
accessible (invoke without access control failure). When a module uses
the ResourceBundle API to load the ListResourceBundle from the module
itself, the ResourceBundle API only loads the class from the module
classloader and the class is returned to the code in the module for
actual access. ServiceLoader works in similar way that the ServiceLoader
only loads the service-provider class but it's the actual caller who
accesses the class. Therefore, I don't think there is a real problem here.

> It is hard for me to see how all this complexity is justified, simply to
> hide resources that have never been hidden before.
>
> This doesn't mean there is no place for private resources! It just means
> that they can only be used by the module classes themselves. And this is
> still a very valuable feature: developers can restrict access to
> resources that are strictly implementation details.

I don't think the security permission approach is actually complex, and
I do have two concerns with the approach you suggested:

1. A module has to export its own supposed-to-be-private resources in
the same module in order to use the ResourceBundle API to load them.

2. It does not allow private resources in a module to be accessed
externally by any mean.

My main concern is around #1 'cause it doesn't sound right to me - I
think a private resource should stay non-exported. #2 is an issue if we
want to support a way for external code to access the private resources.
There is already a setAccessible() method that allows you to override
the default access control and to access private methods/fields in
classes, so I think it would also make sense to have a way for external
code to access the private resources when necessary (e.g. debugging,
diagnosis, etc.).

- Stanley

From Stanley.Ho at sun.com  Mon Jun  4 19:17:05 2007
From: Stanley.Ho at sun.com (Stanley M. Ho)
Date: Mon, 04 Jun 2007 19:17:05 -0700
Subject: Query...
In-Reply-To: <46548491.7040408@oracle.com>
References: <46548491.7040408@oracle.com>
Message-ID: <4664C7A1.1030303@sun.com>

Bryan Atsatt wrote:
> And we haven't done so in the spec, either. I think we should.

Query was not declared as Serializable in the spec because it was
unclear if we wanted to allow custom Query implementations and how it
might impact these custom Query implementations.

Now that we have decided to allow custom Query implementations, if we
make the Query serializable, the custom implementations will have to
implement serialization properly, but I think this is still okay.
However, if the query is serialized for it to be sent over the wire to a
remote repository, then its usefulness is pretty much limited to the
default query, because it is very unlikely that the remote repository
will happen to have the custom implementation classes in place for the
remote repository to deserialize the custom query properly.

If the EG still thinks this is a useful thing to add, I can incorporate
it into the next revision of the specification.

- Stanley

From Stanley.Ho at sun.com  Mon Jun  4 20:16:33 2007
From: Stanley.Ho at sun.com (Stanley M. Ho)
Date: Mon, 04 Jun 2007 20:16:33 -0700
Subject: Relationship to JSR 291 [was: Re: Bryan's comments]
In-Reply-To: <2071f55c84384fb5d141568f491677c9@ungoverned.org>
References: <OF62C11B58.0C61920A-ON802572EB.0055969A-802572EB.005897E5@uk.ibm.com>
	<465DF2E4.4090603@sun.com>
	<9E52FB97-F312-4A4F-A096-ECA0430F9D7C@ungoverned.org>
	<465F4D94.9060809@sun.com>
	<0e79b722e610c3de839326fb68539389@ungoverned.org>
	<465F62FC.7070201@sun.com>
	<2071f55c84384fb5d141568f491677c9@ungoverned.org>
Message-ID: <4664D591.1060608@sun.com>

Hi Richard,

Richard S.Hall wrote:
> ...
> Perhaps so.
>
> I am operating off my [possibly dated] recollection that 277 modules
> explicitly list the classes that they export, which meant that they may
> expose arbitrary public classes from a package. If this is no longer
> the case, then this is fine.
>
> The point remains, all legacy OSGi dependencies on a package assume
> that all public classes are exported from that package. An OSGi
> dependency on a given package cannot be resolved to a 277 module whose
> "exported public" classes are not the same as all "public" classes in
> the specific package.

I was under the impression that the class filtering is also in effect to
determine which public classes are exported from that package in OSGi
and that the OSGi dependency on a package already has a similar
"exported public" concept.

Anyway, these are all related to how the OSGi framework might import 277
modules by package name. I think it would be good to know if this is
actually a sensible thing that the OSGi framework wants to support
before we dive too deep on this. Glyn/Richard, is this something that
you foresee the OSGi framework will support?

- Stanley

From bryan.atsatt at oracle.com  Mon Jun  4 21:42:46 2007
From: bryan.atsatt at oracle.com (Bryan Atsatt)
Date: Mon, 04 Jun 2007 21:42:46 -0700
Subject: Exported resources
In-Reply-To: <4664C2B1.1050109@sun.com>
References: <4654109D.40205@sun.com> <46548736.7020507@oracle.com>
	<465629EA.2050008@sun.com> <465658CE.5020409@oracle.com>
	<465DCE51.4010304@sun.com> <465DDCD0.8090703@oracle.com>
	<465DF6D6.5050608@sun.com> <465E0201.2030308@oracle.com>
	<465F68C8.4000103@sun.com> <46609428.4000909@oracle.com>
	<4664C2B1.1050109@sun.com>
Message-ID: <4664E9C6.30608@oracle.com>

Hey Stanley,

Yeah, sorry about the flood of mail; I've had a little time to catch up.

You seem to have a JRE-centric perspective here ;^). Third-party
frameworks/libraries are precisely what a module system is for: to allow
their clients to *avoid* bundling the jars. And they aren't likely to be
deployed as extensions, either; not when there's this nice repository
model. Consider Spring. Or any of the hundreds of other third party
frameworks/libraries out there. We *want* these to be deployed as
normal, separate modules.

And signing a jar by itself doesn't grant any additional permissions, it
simply enables verification of the contained classes. Some environments
may support UI to grant additional permissions, of course, but that is
orthogonal.

But, lets leave aside the whole grant issue for now, and assume that we
can grant permissions as needed to access module private resources.

There is a much bigger issue, which is that this whole model falls apart
simply because we can't grant this type of "friend" access to module
private classes...

Your description of how ResourceBundle or ServiceLoader accesses the
class contains a big assumption, one that *could* be correct for such
JRE code, but is almost certainly wrong for third-party code.

These types of frameworks call Class.forName() or loader.loadClass().
And then they call either Class.newInstance() or Constructor.newInstance().

Regardless of whether or not they invoke any other methods, JSR 294 will
require an access control check at the point of instantiation, extending
those it performs today. Look at how Class.newInstance() and
Constructor.newInstance() both use:

     Reflection.ensureMemberAccess()

This is the logical point at which to inject the new module private
access control check.

Could this be made to succeed for JRE callers? Of course (but it
wouldn't be right, IMO).

But, more importantly: it won't, and cannot, succeed for third-party
callers without some sort of friend model for classes. Which isn't on
the table.


So, again, if we're not going to do this for classes, why should we do
it for resources?

// Bryan

Stanley M. Ho wrote:
> Hi Bryan,
>
> I would like to get closures on a few open issues first, so my responses
> will be limited to those threads rather than all the new threads that
> have been started recently.
>
> Bryan Atsatt wrote:
>> Hi Stanley,
>>
>> Sorry if I'm not being clear. Let me try to summarize:
>>
>> - I *do* want 277 to support private (non-exported) resources, but
>
> Good.
>
>> - I do not think we should use permissions as the enforcement mechanism.
>>
>> - I believe the 'caller-class-is-member' enforcement mechanism is
>> sufficient, and
>>
>> - I do realize that this means ResourceBundle.getBundle() (or similar)
>> will fail to find private resources; these must be made public.
>>
>> Let me give more detailed reasoning for this position...
>>
>> You are of course correct that a class loader is free to add permissions
>> to any class by assigning a ProtectionDomain with those permissions
>> during defineClass(). (Let's call these "hard-wired" permissions.)
>>
>> So permission to access private resources could easily be hard-wired for
>> other classes *in the same module*.
>>
>> To this point, there really isn't any meaningful difference between
>> using permissions or using my approach: both allow module classes to
>> directly access their private resources.
>>
>> But if we want to grant that same permission to *any* class outside of
>> the module, it gets far more interesting. And it is this problem that
>> I'm concerned with.
>>
>> For ResourceBundle.getBundle(), for example, to access a private
>> resource, we must grant permission to the *ResourceBundle* class. And
>> how do we do that?
>>
>> We could hard-wire permission for ResourceBundle, but... what about
>> ServiceLoader? Ok, so let's just hard-wire permission for any JRE module.
>
> The system classes are granted with all permissions, so they will have
> sufficient permissions to access the private resources in a module. I
> don't think this is a real issue.
>
>> Problem solved? Not at all: what about all the non JRE frameworks out in
>> the world that need the same kind of access?
>
> I agreed that using other frameworks might be a concern.
>
>> Clearly we cannot hard-wire permissions for these. So now we need to use
>> policy file entries to do so. We could of course take this approach, but
>> it either leaves some poor admin running around to update policy files
>> as things break, or some fancy mechanism to grant access during
>> framework module install. Ick.
>
> If the framework is bundled as part of your module, it will have the
> same set of permissions as the other code in your module, so it will
> have sufficient permissions to allow your code to access the private
> resources. If the framework is deployed through the typical mechanisms
> (e.g. extension classpath, etc.), it will have all permissions, hence it
> will also have sufficient permissions to access the private resources.
> Further, if the framework is deployed as another module in one of the
> system's repositories, it will likely have all permissions to allow your
> code to access the private resources in your module as well. Moreover,
> if the framework JAR/module is signed and is trusted, the system will
> typically grant all permissions to the code in the framework when the
> code is loaded. In other words, there is no need to explicitly grant
> permissions in most scenarios.
>
> The scenario where this may be an issue is when the framework is
> deployed through some other means (e.g. other modules in a
> URLRepository, JARs in custom classloaders) *and* the framework is
> unsigned (or if the framework is signed but not trusted for some
> reasons), it may not have sufficient permissions to allow your code to
> access the private resources through the framework API. However, it is
> unclear to me if there are frameworks deployed like this for this issue
> to become a real problem. Do you have any example that shows this is a
> real problem?
>
>> And, hard-wired or not, permissions *are* a friend model.
>>
>> But 294 is not likely to provide such a model for classes.
>>
>> So, even if we hard-wire permission for ResourceBundle to access a
>> private resource, how is it going to access a private ListResourceBundle
>> class? Or how will the ServiceLoader gain access to the service provider
>> class? How will third-party frameworks solve this problem?
>
> In a typical classloader, all public and private classes are visible
> (return from loadClass()) externally but only the public ones are
> accessible (invoke without access control failure). When a module uses
> the ResourceBundle API to load the ListResourceBundle from the module
> itself, the ResourceBundle API only loads the class from the module
> classloader and the class is returned to the code in the module for
> actual access. ServiceLoader works in similar way that the ServiceLoader
> only loads the service-provider class but it's the actual caller who
> accesses the class. Therefore, I don't think there is a real problem here.
>
>> It is hard for me to see how all this complexity is justified, simply to
>> hide resources that have never been hidden before.
>>
>> This doesn't mean there is no place for private resources! It just means
>> that they can only be used by the module classes themselves. And this is
>> still a very valuable feature: developers can restrict access to
>> resources that are strictly implementation details.
>
> I don't think the security permission approach is actually complex, and
> I do have two concerns with the approach you suggested:
>
> 1. A module has to export its own supposed-to-be-private resources in
> the same module in order to use the ResourceBundle API to load them.
>
> 2. It does not allow private resources in a module to be accessed
> externally by any mean.
>
> My main concern is around #1 'cause it doesn't sound right to me - I
> think a private resource should stay non-exported. #2 is an issue if we
> want to support a way for external code to access the private resources.
> There is already a setAccessible() method that allows you to override
> the default access control and to access private methods/fields in
> classes, so I think it would also make sense to have a way for external
> code to access the private resources when necessary (e.g. debugging,
> diagnosis, etc.).
>
> - Stanley
>

From bryan.atsatt at oracle.com  Mon Jun  4 22:37:55 2007
From: bryan.atsatt at oracle.com (Bryan Atsatt)
Date: Mon, 04 Jun 2007 22:37:55 -0700
Subject: Query...
In-Reply-To: <4664C7A1.1030303@sun.com>
References: <46548491.7040408@oracle.com> <4664C7A1.1030303@sun.com>
Message-ID: <4664F6B3.7010109@oracle.com>

Sure, but it isn't like this problem has no solution. RMI (JRMP or IIOP)
will marshall classes, and surely some enterprising developer will
implement a solution using our very own Repository model (marshall an
"import"+class-name+repository-name back to the originator, or better
yet, to a large centralized repository, etc.).

// Bryan

Stanley M. Ho wrote:
> Bryan Atsatt wrote:
>> And we haven't done so in the spec, either. I think we should.
>
> Query was not declared as Serializable in the spec because it was
> unclear if we wanted to allow custom Query implementations and how it
> might impact these custom Query implementations.
>
> Now that we have decided to allow custom Query implementations, if we
> make the Query serializable, the custom implementations will have to
> implement serialization properly, but I think this is still okay.
> However, if the query is serialized for it to be sent over the wire to a
> remote repository, then its usefulness is pretty much limited to the
> default query, because it is very unlikely that the remote repository
> will happen to have the custom implementation classes in place for the
> remote repository to deserialize the custom query properly.
>
> If the EG still thinks this is a useful thing to add, I can incorporate
> it into the next revision of the specification.
>
> - Stanley
>

From Stanley.Ho at sun.com  Mon Jun  4 22:49:46 2007
From: Stanley.Ho at sun.com (Stanley M. Ho)
Date: Mon, 04 Jun 2007 22:49:46 -0700
Subject: Relationship to JSR 291 [was: Re: Bryan's comments]
In-Reply-To: <46646639.7080606@oracle.com>
References: <OF62C11B58.0C61920A-ON802572EB.0055969A-802572EB.005897E5@uk.ibm.com>
	<465DF2E4.4090603@sun.com> <465E2051.7020008@oracle.com>
	<465EBE00.2060306@ungoverned.org> <465F0D8C.5010606@oracle.com>
	<465F6157.7070706@sun.com> <46646639.7080606@oracle.com>
Message-ID: <4664F97A.20007@sun.com>

Hi Bryan,

Bryan Atsatt wrote:
>> ...
>> Rather than surfacing the import-by-package at the API level, I think we
>> can solve #3 by generalizing the import dependency concept as
>> import-by-name, and this name can be mapped to superpackage-name in JSR
>> 277 and OSGi-bundle-name or OSGi-exported-package-name in OSGi (i.e.
>> exposing OSGi bundles and exported packages through implementing
>> OSGiBundleModuleDefinition and OSGiPackageModuleDefinition). I think
>> this is also what you intended to say at one point (correct me if I
>> misunderstand you) by suggesting to change
>> ImportDependency.getModuleName() into something like
>> ImportDependency.getName().
>
> I believe we are now thinking along the same lines.

Good to know.

> I just sent an email on the topic of generalizing the import dependency
> concept, with the subject "Import constraints". With the approach I
> suggest there, the distinctions between module-name and package-name are
> mostly transparent to the runtime (declarative support for package-name
> is not addressed).

I will reply to that thread later.

>> In case if the 277 module would like to import these OSGi exported
>> packages using an OSGi-like resolution algorithm, it should be possible
>> for it to use a custom import policy (or import override policy or maybe
>> some new override mechanisms) to get the behavior it wants.
>>
>> Do you think this is a workable approach to address #3?
>
> I don't think custom code should be required; we need to make it easier
> than that or it doesn't qualify as "first-class" citizenship, at least
> not to me :^).

The OSGi resolution algorithm has evolved significantly between R3 and
R4, and it will likely continue to evolve in certain degree in future
releases (Glyn/Richard, please correct me if I'm wrong). From this
perspective, I don't think it makes sense to have build-in OSGi-like
resolution algorithm in 277, because it will stick forever once 277 is
part of the SE platform and cannot be evolve easily.

I think it would makes more sense to allow OSGi-like resolution logic to
be plugged into the 277 module which imports OSGi bundles/packages. We
could argue whether it should be plugged in per-module in the form of
custom import policy, or per-system(or per-repository?) in the form of
import override policy, or maybe even have a new mechanism to allow
certain class of modules (e.g. 277 modules importing OSGi modules, or
277 modules importing NetBeans modules, etc.) to always use a specific
import policy automatically.  But my point is that OSGi-like resolution
logic is something that will need to evolve in the future and should not
be built-in. I think if there is a way to plug in OSGi-like resolution
logic into 277 module which imports OSGi bundles, it would still qualify
as "first-class" citizenship to me. ;)

> And we don't yet have *any* import declaration mechanism (yes, we're
> assuming that this will be handled in 294, but it hasn't been brought up
> yet). I don't see this as a difficult issue either way. It would be
> natural to have both import-by-name and import-by-package in
> superpackages, if we have import at all. If we need to use annotations
> instead, it is easy to have both types.

As we discussed (and I think you also agreed), we can make the import
dependency to be generic with a name, and the name could be a module
name (or a package name in the context of OSGi interoperability). We'll
only need to have a single import-by-name either in the superpackage or
in the form of annotation.

- Stanley

From glyn_normington at UK.IBM.COM  Fri Jun  1 02:43:27 2007
From: glyn_normington at UK.IBM.COM (Glyn Normington)
Date: Fri, 01 Jun 2007 10:43:27 +0100
Subject: Relationship to JSR 291 [was: Re: Bryan's comments]
In-Reply-To: <OF084663C8.EE3A0E10-ON802572EC.0058BFEB-802572EC.00594B64@uk.ibm.com>
Message-ID: <OF89FAD0A0.1337D999-ON802572ED.0034FF2E-802572ED.00356AED@uk.ibm.com>

After writing:

>I think ResolutionContext doesn't need any detail

I remembered the class in the interoperation kernel prototype did have a
little detail (but not much):

public interface ResolutionContext extends KernelRelated {

    /**
     *  Notify completion.
     */
    void complete();

}

I blogged ([1]) the README and class diagrams from the prototype for ease
of reference.

Glyn
[1]
http://underlap.blogspot.com/2007/06/module-system-interoperation-kernel.html

glyn_normington at uk.ibm.com wrote on 31/05/2007 17:15:19:

>
> Bryan Atsatt <bryan.atsatt at ORACLE.COM> wrote on 30/05/2007 19:11:02:
>
> > Responses inline, and a few clarifications here (I was a bit tired
when
> > I finished this last night :^)...
> >
> > The main point I was trying to make is that resolution must occur
within
> > a specific context, but I don't think my example APIs showed that
well.
> > I was assuming that ImportResolver had a ModuleContext as a field, but
> > we can make this much cleaner and easier to understand by passing it
as
> > an argument:
> >
> > public abstract class ImportResolver {
> >      public abstract Module resolve(ModuleDefinition def,
> >                                     ModuleContext ctx);
> > }
> >
> > And we can really tie this together by adding a convenience method to
> > ModuleContext:
> >
> > public abstract class ModuleContext {
> >      ...
> >      public Module resolve(ModuleDefinition def) {
> >          return getImportResolver().resolve(def, this);
> >      }
> > }
> >
> > Now resolution becomes simply:
> >
> >      context.resolve(definition);
> >
> >
> > (I also left out any use of the ImportPolicy, as it isn't yet clear to
> > me that it should remain a separate type in this model.)
> >
> > // Bryan
> >
> > Glyn Normington wrote:
> > >
> > > *Bryan Atsatt <bryan.atsatt at ORACLE.COM>* wrote on 30/05/2007
07:57:59:
> > >
> > >  > Andy Piper wrote:
> > >  >  > At 23:19 25/05/2007, Stanley M. Ho wrote:
> > >  >  >> Anyway, it seems the EG consensus so far is to not add
> import package
> > >  >  >> support. If any EG member disagrees, please speak up.
> > >  >  >
> > >  >  > Well, it depends on what the solution for enabling
interoperation
> > >  >  > with JSR 291 is.
> > >  >  > Our requirement is that there must be a solution, if that
requires
> > >  >  > import package, so be it. If not then not.
> > >  >
> > >  > Exactly.
> > >  >
> > >  > I think we can all agree that, at minimum, interoperation means
that
> > >  > classes and resources are sharable *across* ModuleSystems at
runtime.
> > >  >
> > >  > Which implies that *import dependencies must be resolvable across
> > >  > multiple ModuleSystem instances*. (BTW, I think we should change
the
> > >  > state name "PREPARING" to "RESOLVING" in 7.2.1.)
> > >
> > > Agreed. We must avoid the trap of thinking that module system
interop.
> > > can be achieved by exposing class loaders (as loadClass will happily
> > > load unexported classes).
> > >
> > >  >
> > >  > So the open issue is the richness of the import "language": must
we
> > >  > support only lowest-common-denominator, or can we do better
without
> > >  > over-complicating the design?
> > >  >
> > >  > I for one would like to be able to have a single module express
> > >  > dependencies on modules from both the same and different
ModuleSystems,
> > >  > *using the standard semantics of each*. This may be reaching too
far,
> > >  > but we should at least explore it seriously while we figure out
what
> > >  > interop means here...
> > >
> > > At this point, I feel that is likely to be reaching too far, but I'm
> > > happy to play along and see what we can learn along the way.
> > >
> > >  >
> > >  >
> > >  > BASICS
> > >  >
> > >  > So far, we only know of two different import semantics:
> module-name, and
> > >  > package-name. For discussion, let's call these:
> > >  >
> > >  > a. import-module
> > >  > b. import-package
> > >  >
> > >  > So, to start, we could:
> > >  >
> > >  > 1. Support declaration of both import types. If 294 supports
imports at
> > >  > all, it should be relatively easy to support both, since a
superpackage
> > >  >   name is a module name, and it contains member package
> names. (Compiler
> > >  > support is clearly the critical issue here, but it will obviously
> > >  > require use of the 277 runtime, so the import *type* should be
> > >  > transparent to it.) At worst, we'd need two new annotation types.
> > >
> > > A superpackage name is a deployment module name in the JSR 277 model
of
> > > one superpackage per deployment module, but I don't see any reason
why a
> > > JSR 291 deployment module should not contain more than one
superpackage.
> > > So if 294 were to support import, then its import-module would
really be
> > > a superpackage import rather than a development module import.
> >
> > If we end up with nested superpackages, might it make sense to model
> > multiple superpackages by enclosing them in a single top-level one?
>
> That is an option, but of course each nested superpackage has to
> name its parent, so it wouldn't be possible to combine superpackages
> from independent groups or sources without either modifying their
> superpackage declarations or getting them to agree on the name of
> the parent superpackage and code it themselves.
>
> >
> > >
> > >  >
> > >  > 2. Provide API for both import types (e.g. ImportDependency has
> > >  > getModuleName() and getPackageName() methods, one of which will
return
> > >  > null on a given instance).
> > >  >
> > >  > However, we know these are necessary but not sufficient. Leaving
aside
> > >  > the resolution issue for a moment, support for import-package
also
> > >  > suggests that we:
> > >  >
> > >  > 3. Enable a single module to declare different versions for each
of its
> > >  > member packages.
> > >  >
> > >  > 4. Enable efficient Repository lookup by package name.
> > >  >
> > >  > I think these are relatively easy (but *could* be considered
optional).
> > >  >
> > >  > We also need:
> > >  >
> > >  > 5. Standard Query types for lookup by module and package name.
> > >  >
> > >  >
> > >  > EXISTING DEPENDENCY RESOLUTION MODEL
> > >  >
> > >  > The more interesting issue is dependency resolution. But this
hasn't
> > >  > been discussed in any real detail, so lets do so before talking
further
> > >  > about import-package. To simplify this discussion, I'm ignoring
> > >  > bundled/custom import policies for now...
> > >  >
> > >  > Resolution in the current spec is delegated to the associated
> > >  > ModuleSystem instance (7.2.2 #8). While the details are not
> spelled out,
> > >  > the expectation appears to be that
> > >  > ModuleSystem.getModule(ModuleDefinition) must:
> > >  >
> > >  > - Select an initial repository. Call getRepository() on the input
> > > parameter.
> > >  >
> > >  > Then, for each ImportDependency of the definition:
> > >  >
> > >  > - Select a matching definition. Construct a Query from the
> > >  > ImportDependency and use Repository.find() to lookup a matching
> > >  > ModuleDefinition.
> > >  >
> > >  > - Get an instance. Use def.getModuleSystem().getModule(def). The
> > >  > ModuleSystem is expected to return a cached instance if
available, or
> > >  > create/cache/return one if not.
> > >
> > > I think there also needs to be some 'resolution context' object
which
> > > explicitly denotes a particular resolution so that each module
system
> > > can keep track of the state of a resolution. This is required when
two
> > > or more imports of a given module from another module system need to
> > > resolve to the same module instance. A resolution context may also
be
> > > needed for back-tracking when a set of module instances created
earlier
> > > in resolution turn out not to satisfy all the necessary constraints.
> >
> > Yes. I had (briefly :^) thought that we need only a List<Module> as
> > field in the ImportResolver to hold the resolved modules as we go. But
> > probably we need to keep some additional state along with each Module,
> > so a new type may be required. Let's call it ResolutionContext, and
> > create one at the start of each resolution:
> >
> > public abstract class ImportResolver {
> >
> >      public Module resolve(ModuleDefinition def, ModuleContext modCtx)
{
> >          ResolutionContext resCtx = createResolutionContext();
> >          return resolve(def, modCtx, resCtx);
> >      }
> >
> >      protected abstract ResolutionContext createResolutionContext();
> >
> >      protected abstract Module resolve(ModuleDefinition def,
> >                                        ModuleContext modCtx,
> >                                        ResolutionContext resCtx);
> > }
> >
> > Does this approach make sense? Any thoughts on details for
> > ResolutionContext?
>
> Seems reasonable. I think ResolutionContext doesn't need any detail
> - it can simply be used as the key of a map recording the
> resolutions that a module system is currently aware of.
>
> >
> > >
> > >  >
> > >  >
> > >  > (TBD: The PlatformBinding must be taken into account somehow
during
> > >  > selection. ModuleDefinition must include an accessor for it, and
either
> > >  > Repository.find() should implicitly filter them, or the caller
must
> > >  > construct a Query which will do so. I think we should add a
> > >  > CURRENT_PLATFORM constant to Query, which will evaluate to true
if no
> > >  > binding is present in a definition.)
> > >  >
> > >  > (The spec also talks about Repository as the mechanism of
isolation
> > >  > (6.4). This was the case in the prototype, where the repository
itself
> > >  > provided caching. It doesn't appear to work with the current
design.
> > >  > There is no need that I can see to isolate ModuleDefinition
> > >  > instances--it is Module instances with their associated
> loaders that may
> > >  > require isolation.)
> > >  >
> > >  > (Also note that if ImportDependency was itself a Query subclass,
there
> > >  > would be no need to do any mapping. And since the
ModuleDefinition
> > >  > subclass must produce ImportDependency instances, it can even
produce
> > >  > more specialized Query instances if desired.)
> > >  >
> > >  >
> > >  > REFINEMENT
> > >  >
> > >  > I think we can improve on the existing model in several ways:
> > >  >
> > >  > A. Provide a model for Module isolation (e.g. for EE, Applets,
etc).
> > >  >
> > >  > B. Encapsulate all selection logic in a single mechanism.
> > >  >
> > >  > C. Eliminate the overhead of the repository lookup when a cached
> > >  > instance exists.
> > >  >
> > >  > Let me propose a new class that encapsulates the caching logic,
enables
> > >  > lookup using Query, and supports multiple instances for
isolation:
> > >  >
> > >  > public abstract class ModuleContext {
> > >  >
> > >  >      // Get the context used to define JRE modules.
> > >  >      public static ModuleContext getBootstrapContext(){...};
> > >  >
> > >  >      // Get the context used to define the main module.
> > >  >      public static ModuleContext getSystemContext(){...};
> > >  >
> > >  >      // Get all contexts.
> > >  >      public static List<ModuleContext> getContexts() {...};
> > >  >
> > >  >      // Add a new context.
> > >  >      public static void addContext(ModuleContext ctx) {...}
> > >  >
> > >  >      // Remove a context (error if == default).
> > >  >      public static boolean removeContext(ModuleContext ctx) {...}
> > >  >
> > >  >      // Get the parent context (null if bootstrap).
> > >  >      public ModuleContext getParentContext(){...}
> > >  >
> > >  >      // Get the name of this context.
> > >  >      public String getContextName() {...}
> > >  >
> > >  >      // Create a new Module instance and store it in the cache.
> > >  >      public abstract Module createModule(ModuleDefinition def);
> > >  >
> > >  >      // Find cached Module instances. Must check parent first.
> > >  >      public abstract List<Module> findModules(Query query);
> > >  >
> > >  >      // Set the context used for JRE modules.
> > >  >      static void setBootstrapContext(ModuleContext ctx){...}
> > >  >
> > >  >      // Set the context used to define the main module.
> > >  >      static void setSystemContext(ModuleContext ctx){...}
> > >  > }
> > >  >
> > >  > The JVM will create an appropriate subtype and call
> > >  > setBootstrapContext(). The launcher will create a child
> context and call
> > >  > setSystemContext(). An EE (or similar) environment can
> create/remove new
> > >  > contexts as needed for application isolation.
> > >  >
> > >  > And the resolution algorithm can now check the cache *first*,
before
> > >  > doing a repository lookup, using the same mechanism in both.
Query
> > >  > should be used to express *all* selection criteria (including
> > >  > attributes, which we have not yet directly supported).
> > >  >
> > >  > Caches are no longer tied to ModuleSystem instances.
> > >  > ModuleSystem.getModule() can become simply createModule(). The
normal
> > >  > implementation of ModuleContext.createModule() just calls
> > >  > ModuleSystem.createModule() and caches/returns the result.
> > >  >
> > >  >
> > >  > This class could easily be made concrete, but it may be useful to
> > >  > support subtypes for specialization (e.g. event generation,
lifecycle
> > >  > management, specialized diagnostics, etc).
> > >  >
> > >  >
> > >  > RESOLUTION MODELS
> > >  >
> > >  > The current design requires that each ModuleSystem provide its
own
> > >  > resolution logic, and that each definition will be resolved by
its
> > >  > owning ModuleSystem. This model appears to provide flexibility
for
> > >  > significant differences in implementation, but we really don't
know
> > >  > enough at this point. Perhaps only an actual second
implementation will
> > >  > tell us if this provides useful flexibility.
> > >  >
> > >  > It wouldn't surprise me if we have to keep tightening the
> spec as we go,
> > >  > in order to remove inconsistencies that arise from the separate
> > >  > algorithms. And this may eliminate flexibility to the point where
it is
> > >  > no longer useful. Much worse, we may not even discover this until
after
> > >  > the spec and RI are released, if that second implementation (e.g.
OSGi)
> > >  > is not completed beforehand.
> > >  >
> > >  > We should at least consider the obvious alternative: one
algorithm (to
> > >  > rule them all :^). And I don't mean one hard-coded algorithm,I
mean one
> > >  >   replaceable, extensible class, such as:
> > >  >
> > >  > public abstract class ImportResolver {
> > >  >      public abstract Module resolve(ModuleDefinition def);
> > >  > }
> > >  >
> > >  > And we add a method to ModuleContext to retrieve an instance:
> > >  >
> > >  > public abstract class ModuleContext {
> > >  >      ...
> > >  >      public abstract ImportResolver getImportResolver();
> > >  > }
> > >  >
> > >  > (Note that ImportResolver is now in a position to subsume the
> > >  > functionality of both VisibilityPolicy and ImportOverridePolicy.)
> > >  >
> > >  > Repository usage is encapsulated within the implementation of the
> > >  > resolve() method. The full resolution algorithm becomes:
> > >  >
> > >  >     context.getImportResolver().resolve(definition);
> > >  >
> > >  > The launcher uses the "system" context for this. EE, Applets,etc.
make
> > >  > and use their own distinct, isolated instances.
> > >  >
> > >  >
> > >  > RESOLUTION WITH IMPORT-NAME AND IMPORT-PACKAGE
> > >  >
> > >  > With this scaffolding in place we can easily take a phased
approach to
> > >  > supporting import-package: the initial implementation simply does
not
> > >  > support it at runtime.
> > >
> > > A phased approach would be particularly beneficial if the initial
phase
> > > could be delivered as part of Java 7 and subsequent phases
implemented
> > > strictly on top of Java 7. But getting the API right up front might
be
> > > tricky unless we can spot some really good abstractions or prototype
the
> > > later phases sufficiently well. Is that the kind of phasing you had
in
> > > mind?
> >
> > Yes. It would be *extremely* useful to have an OSGi
> > implementation/prototype well under way before Java 7 is completed, so
> > that we can fine tune the model as we learn.
> >
> > >
> > >  >
> > >  > A subsequent implementation may support import-package, but only
within
> > >  > the boundaries of the same ModuleSystem.
> > >  >
> > >  > And a full blown implementation may support import-package across
> > >  > ModuleSystems.
> > >  >
> > >  > We can build in support for selecting/configuring the
ImportResolver as
> > >  > a service, just as we plan to do with ModuleSystem (and
Repository, I
> > >  > presume).
> > >  >
> > >  >
> > >  > And maybe, just maybe, we can find a way to abstract and re-use
the
> > >  > mature resolution logic from the OSGi reference implementation
*as* the
> > >  > one implementation to rule them all.
> > >  >
> > >  > // Bryan
> > >
> > > Glyn
> > >
> > >
> > >
------------------------------------------------------------------------
> > >
> > > /
> > > /
> > >
> > > /Unless stated otherwise above:
> > > IBM United Kingdom Limited - Registered in England and Wales with
number
> > > 741598.
> > > Registered office: PO Box 41, North Harbour, Portsmouth,
> Hampshire PO6 3AU/
> > >
> > >
> > >
> > >
> > >
> > >
>
> Glyn
>
>
>

> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with
> number 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6
3AU
>
>
>
>






Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU





-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/jsr277-eg-observer/attachments/20070601/2ffb304f/attachment-0001.html 

From glyn_normington at UK.IBM.COM  Tue Jun  5 02:27:07 2007
From: glyn_normington at UK.IBM.COM (Glyn Normington)
Date: Tue, 05 Jun 2007 10:27:07 +0100
Subject: Relationship to JSR 291 [was: Re: Bryan's comments]
In-Reply-To: <46646639.7080606@oracle.com>
Message-ID: <OFC12639E7.6C1E83B9-ON802572F1.00310AA2-802572F1.0033EC6C@uk.ibm.com>

Bryan Atsatt <bryan.atsatt at ORACLE.COM> wrote on 04/06/2007 20:21:29:

> Stanley M. Ho wrote:
> > Hi Bryan,
> >
> > I don't think I will have time to response to all your comments this
> > week, so my response would focus on the import-by-package discussion.
> >
> > First, the only import dependency we have consensus in this EG so far
is
> > import-by-module-name (or import-by-superpackage-name), and this was
> > also what's in the EDR. That said, it is by no mean that we can't
> > consider other import dependency granularities (e.g.
> > import-by-package-name, or import-by-contract) in addition to
> > import-by-module-name. I would like to focus this discussion more on
> > whether it makes sense for JSR 277 to support additional import
> > dependency granularity, mainly import-by-package-name, and if so, in
> > what context.
> >
> > There are three cases we should consider:
> >
> > 1. 277 module imports another 277 module by package name
> >
> > I think our consensus so far is that this is nice-to-have or
> > non-critical, given we already have support for import-by-module-name.
> > To keep this discussion in focus, let's not dive into
> > import-by-package-name v.s. import-by-module-name, and let's assume
> > there is no support for #1 for now.
>
> Agreed.
>
> >
> > 2. OSGi module imports 277 module by package name
> >
> > As I discussed this with Richard in previous emails, majority of the
> > works for this fall under the OSGi framework if it wants to enable
this
> > kind of wiring, but we could probably make the implementor's life
easier
> > by provide the appropriate 277 reflective API if necessary.
>
> Yes, the API would be useful.
>
> >
> > 3. 277 module imports OSGi module by package name
> >
> > If I understand you correctly, I think what you really want is the
> > support for this case, so the existing OSGi bundles can be leveraged
by
> > 277 modules in a nature way.
> >
> > Let me say it up front that I don't oppose the idea of #3 in the
context
> > of interoperability. If #3 is what you really want, I think where we
> > diverge is simply how to make it possible.
> >
> > Do you agree so far?
>
> Yes.
>
> >
> > Bryan Atsatt wrote:
> >> ...
> >> I want 277 to explicitly expose the concept of import-by-package
through
> >> appropriate APIs.
> >>
> >> It would be very nice if the 277ModuleSystem implementation itself
> >> directly supports import-by-package (case #3). But this is less
critical
> >> to me than case #4...
> >>
> >> I want a 277ModuleDefinition to be able to import-by-package an
> >> OSGiModuleDefinition (i.e. a bundle exposed via the 277 APIs). I
believe
> >> this case will be a particularly important one. By the time SE 7
ships,
> > ...
> >
> > As I described several emails back:
> >
> >  > This doesn't mean supporting the notion of import by package is
> >  > impossible. What OSGi will expose from its repository is simply
module
> >  > definitions, and module definition is basically an abstraction.
> >  > Therefore, it should be possible to treat an OSGi bundle as a
module
> >  > definition, as well as to treat each exported package in an OSGi
> >  > bundle a module definition as well. The dependency supported in JSR
> >  > 277 is import-by-module. In the JSR 277 module system, this is
mapped
> >  > to import-by-superpackage. In the OSGi module system, this can be
> >  > mapped to import-by-OSGi-bundle or import-by-package. If other
module
> >  > systems want to support different import granularity or if OSGi
> >  > evolves to add new import granularity in the future, this model
should
> >  > still work.
> >
> > Rather than surfacing the import-by-package at the API level, I think
we
> > can solve #3 by generalizing the import dependency concept as
> > import-by-name, and this name can be mapped to superpackage-name in
JSR
> > 277 and OSGi-bundle-name or OSGi-exported-package-name in OSGi (i.e.
> > exposing OSGi bundles and exported packages through implementing
> > OSGiBundleModuleDefinition and OSGiPackageModuleDefinition). I think
> > this is also what you intended to say at one point (correct me if I
> > misunderstand you) by suggesting to change
> > ImportDependency.getModuleName() into something like
> > ImportDependency.getName().
>
> I believe we are now thinking along the same lines.
>
> I just sent an email on the topic of generalizing the import dependency
> concept, with the subject "Import constraints". With the approach I
> suggest there, the distinctions between module-name and package-name are
> mostly transparent to the runtime (declarative support for package-name
> is not addressed).
>
> That isn't to say that there aren't resolution issues to be worked out
> for package-name, but its a start.
>
> >
> > We expect the module-aware compiler would somehow recognize the import
> > dependency defined in 277. In other words, if the OSGi framework
exposes
> > the module definitions appropriately, you'll get the compiler support.
> > Since the import dependency is generic, the compiler and other tools
ed
> > to do extra work to support other forms of import dependency.
>
> Yep. As I've said from the very beginning, I think the compiler is going
> to have to rely on the 277 runtime to resolve *external* (i.e.
> non-source) dependencies. So as long as the runtime can deal with
> package-name, the compiler will too.
>
> (Dependencies resolvable via included source will, I assume, just
> *directly* resolve, without applying any constraint logic. That is, if a
> class refers to a class in another superpackage, and that class &
> superpackage are available in the same compilation, the compiler will
> simply select them without applying *any* constraints. This may be too
> simple of a model, but maybe it is sufficient.)
>
> >
> > In case if the 277 module would like to import these OSGi exported
> > packages using an OSGi-like resolution algorithm, it should be
possible
> > for it to use a custom import policy (or import override policy or
maybe
> > some new override mechanisms) to get the behavior it wants.
> >
> > Do you think this is a workable approach to address #3?
>
> I don't think custom code should be required; we need to make it easier
> than that or it doesn't qualify as "first-class" citizenship, at least
> not to me :^).
>
>
> I guess I am still missing something fundamental--I keep thinking that
> import-by-package is actually not so hard. There are three main
components:
>
> 1. Declaration.
> 2. Runtime infrastructure.
> 3. Resolution algorithm.
>
> I'm pretty sure that #2 is easy (and covered in my "Import constraints"
> thread).
>
> And we don't yet have *any* import declaration mechanism (yes, we're
> assuming that this will be handled in 294, but it hasn't been brought up
> yet). I don't see this as a difficult issue either way. It would be
> natural to have both import-by-name and import-by-package in
> superpackages, if we have import at all. If we need to use annotations
> instead, it is easy to have both types.
>
> So I can only assume that the *real* concern here is #3. And I can't
> tell if this is a valid concern or not. I hope Glyn and Richard will
> point out what I'm missing here...

Sounds right to me. However, the JSR 291 support for import-package is
more subtle, and I would say was generally harder to get right (over
several versions and several years), than require-bundle which was added
fairly simply (despite the semantic rough edges, pointed out in the
spec.). Although JSR 277 has paid lip service to learning from OSGi, the
design is radically different in several ways:

* dependencies expressed in metadata and code rather than purely in
metadata
* split packages not allowed
* shallow/deep validation to avoid class space inconsistency rather than
metadata expressing package dependencies and a system provided resolver
that aims to find a "best fit"

so my feeling is that we've got a quite a lot of work to do to support
something equivalent to import-by-package. Of course, it may turn out to
be simpler than I expect...

>
> What is the difference between selecting a ModuleDefinition based on its
> name, or selecting it based on its contents? Either way, "shallow"
> validation must still occur.
>
> // Bryan
>
> >
> > - Stanley
> >

Glyn





Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU





-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/jsr277-eg-observer/attachments/20070605/c58a9202/attachment.html 

From heavy at UNGOVERNED.ORG  Tue Jun  5 05:23:15 2007
From: heavy at UNGOVERNED.ORG (Richard S. Hall)
Date: Tue, 05 Jun 2007 08:23:15 -0400
Subject: Relationship to JSR 291 [was: Re: Bryan's comments]
In-Reply-To: <4664D591.1060608@sun.com>
References: <OF62C11B58.0C61920A-ON802572EB.0055969A-802572EB.005897E5@uk.ibm.com>
	<465DF2E4.4090603@sun.com>
	<9E52FB97-F312-4A4F-A096-ECA0430F9D7C@ungoverned.org>
	<465F4D94.9060809@sun.com>
	<0e79b722e610c3de839326fb68539389@ungoverned.org>
	<465F62FC.7070201@sun.com>
	<2071f55c84384fb5d141568f491677c9@ungoverned.org>
	<4664D591.1060608@sun.com>
Message-ID: <466555B3.9000903@ungoverned.org>

Stanley M. Ho wrote:
> Hi Richard,
>
> Richard S.Hall wrote:
>> ...
>> Perhaps so.
>>
>> I am operating off my [possibly dated] recollection that 277 modules
>> explicitly list the classes that they export, which meant that they may
>> expose arbitrary public classes from a package. If this is no longer
>> the case, then this is fine.
>>
>> The point remains, all legacy OSGi dependencies on a package assume
>> that all public classes are exported from that package. An OSGi
>> dependency on a given package cannot be resolved to a 277 module whose
>> "exported public" classes are not the same as all "public" classes in
>> the specific package.
>
> I was under the impression that the class filtering is also in effect to
> determine which public classes are exported from that package in OSGi
> and that the OSGi dependency on a package already has a similar
> "exported public" concept.

This is definitely true, but this was only introduced in R4 and is still
intended to be the exception, not the norm...from my understanding, 277
defined the norm in the opposite direction.

Ultimately, if we are making the same assumption, which is that people
will generally be exporting all public classes from a package and only
occasionally filtering some classes that should not be public, and we
are promoting this as the best practice, then we are at least in the
same ball park.

> Anyway, these are all related to how the OSGi framework might import 277
> modules by package name. I think it would be good to know if this is
> actually a sensible thing that the OSGi framework wants to support
> before we dive too deep on this. Glyn/Richard, is this something that
> you foresee the OSGi framework will support?

Well, effectively all OSGi dependency resolution comes down to packages
at some point, so if we are going to use 277 modules at all in the OSGi
framework, then we have to have some way to determine their set of
exported packages, which is necessary for determining consistency.

-> richard

From bryan.atsatt at oracle.com  Tue Jun  5 14:58:39 2007
From: bryan.atsatt at oracle.com (Bryan Atsatt)
Date: Tue, 05 Jun 2007 14:58:39 -0700
Subject: Relationship to JSR 291 [was: Re: Bryan's comments]
In-Reply-To: <OFC12639E7.6C1E83B9-ON802572F1.00310AA2-802572F1.0033EC6C@uk.ibm.com>
References: <OFC12639E7.6C1E83B9-ON802572F1.00310AA2-802572F1.0033EC6C@uk.ibm.com>
Message-ID: <4665DC8F.8000402@oracle.com>

Glyn Normington wrote:
>
> *Bryan Atsatt <bryan.atsatt at ORACLE.COM>* wrote on 04/06/2007 20:21:29:
>
>  > Stanley M. Ho wrote:
>  > > Hi Bryan,
>  > >
>  > > I don't think I will have time to response to all your comments this
>  > > week, so my response would focus on the import-by-package discussion.
>  > >
>  > > First, the only import dependency we have consensus in this EG so
> far is
>  > > import-by-module-name (or import-by-superpackage-name), and this was
>  > > also what's in the EDR. That said, it is by no mean that we can't
>  > > consider other import dependency granularities (e.g.
>  > > import-by-package-name, or import-by-contract) in addition to
>  > > import-by-module-name. I would like to focus this discussion more on
>  > > whether it makes sense for JSR 277 to support additional import
>  > > dependency granularity, mainly import-by-package-name, and if so, in
>  > > what context.
>  > >
>  > > There are three cases we should consider:
>  > >
>  > > 1. 277 module imports another 277 module by package name
>  > >
>  > > I think our consensus so far is that this is nice-to-have or
>  > > non-critical, given we already have support for import-by-module-name.
>  > > To keep this discussion in focus, let's not dive into
>  > > import-by-package-name v.s. import-by-module-name, and let's assume
>  > > there is no support for #1 for now.
>  >
>  > Agreed.
>  >
>  > >
>  > > 2. OSGi module imports 277 module by package name
>  > >
>  > > As I discussed this with Richard in previous emails, majority of the
>  > > works for this fall under the OSGi framework if it wants to enable this
>  > > kind of wiring, but we could probably make the implementor's life
> easier
>  > > by provide the appropriate 277 reflective API if necessary.
>  >
>  > Yes, the API would be useful.
>  >
>  > >
>  > > 3. 277 module imports OSGi module by package name
>  > >
>  > > If I understand you correctly, I think what you really want is the
>  > > support for this case, so the existing OSGi bundles can be leveraged by
>  > > 277 modules in a nature way.
>  > >
>  > > Let me say it up front that I don't oppose the idea of #3 in the
> context
>  > > of interoperability. If #3 is what you really want, I think where we
>  > > diverge is simply how to make it possible.
>  > >
>  > > Do you agree so far?
>  >
>  > Yes.
>  >
>  > >
>  > > Bryan Atsatt wrote:
>  > >> ...
>  > >> I want 277 to explicitly expose the concept of import-by-package
> through
>  > >> appropriate APIs.
>  > >>
>  > >> It would be very nice if the 277ModuleSystem implementation itself
>  > >> directly supports import-by-package (case #3). But this is less
> critical
>  > >> to me than case #4...
>  > >>
>  > >> I want a 277ModuleDefinition to be able to import-by-package an
>  > >> OSGiModuleDefinition (i.e. a bundle exposed via the 277 APIs). I
> believe
>  > >> this case will be a particularly important one. By the time SE 7
> ships,
>  > > ...
>  > >
>  > > As I described several emails back:
>  > >
>  > >  > This doesn't mean supporting the notion of import by package is
>  > >  > impossible. What OSGi will expose from its repository is simply
> module
>  > >  > definitions, and module definition is basically an abstraction.
>  > >  > Therefore, it should be possible to treat an OSGi bundle as a module
>  > >  > definition, as well as to treat each exported package in an OSGi
>  > >  > bundle a module definition as well. The dependency supported in JSR
>  > >  > 277 is import-by-module. In the JSR 277 module system, this is
> mapped
>  > >  > to import-by-superpackage. In the OSGi module system, this can be
>  > >  > mapped to import-by-OSGi-bundle or import-by-package. If other
> module
>  > >  > systems want to support different import granularity or if OSGi
>  > >  > evolves to add new import granularity in the future, this model
> should
>  > >  > still work.
>  > >
>  > > Rather than surfacing the import-by-package at the API level, I
> think we
>  > > can solve #3 by generalizing the import dependency concept as
>  > > import-by-name, and this name can be mapped to superpackage-name in JSR
>  > > 277 and OSGi-bundle-name or OSGi-exported-package-name in OSGi (i.e.
>  > > exposing OSGi bundles and exported packages through implementing
>  > > OSGiBundleModuleDefinition and OSGiPackageModuleDefinition). I think
>  > > this is also what you intended to say at one point (correct me if I
>  > > misunderstand you) by suggesting to change
>  > > ImportDependency.getModuleName() into something like
>  > > ImportDependency.getName().
>  >
>  > I believe we are now thinking along the same lines.
>  >
>  > I just sent an email on the topic of generalizing the import dependency
>  > concept, with the subject "Import constraints". With the approach I
>  > suggest there, the distinctions between module-name and package-name are
>  > mostly transparent to the runtime (declarative support for package-name
>  > is not addressed).
>  >
>  > That isn't to say that there aren't resolution issues to be worked out
>  > for package-name, but its a start.
>  >
>  > >
>  > > We expect the module-aware compiler would somehow recognize the import
>  > > dependency defined in 277. In other words, if the OSGi framework
> exposes
>  > > the module definitions appropriately, you'll get the compiler support.
>  > > Since the import dependency is generic, the compiler and other tools ed
>  > > to do extra work to support other forms of import dependency.
>  >
>  > Yep. As I've said from the very beginning, I think the compiler is going
>  > to have to rely on the 277 runtime to resolve *external* (i.e.
>  > non-source) dependencies. So as long as the runtime can deal with
>  > package-name, the compiler will too.
>  >
>  > (Dependencies resolvable via included source will, I assume, just
>  > *directly* resolve, without applying any constraint logic. That is, if a
>  > class refers to a class in another superpackage, and that class &
>  > superpackage are available in the same compilation, the compiler will
>  > simply select them without applying *any* constraints. This may be too
>  > simple of a model, but maybe it is sufficient.)
>  >
>  > >
>  > > In case if the 277 module would like to import these OSGi exported
>  > > packages using an OSGi-like resolution algorithm, it should be possible
>  > > for it to use a custom import policy (or import override policy or
> maybe
>  > > some new override mechanisms) to get the behavior it wants.
>  > >
>  > > Do you think this is a workable approach to address #3?
>  >
>  > I don't think custom code should be required; we need to make it easier
>  > than that or it doesn't qualify as "first-class" citizenship, at least
>  > not to me :^).
>  >
>  >
>  > I guess I am still missing something fundamental--I keep thinking that
>  > import-by-package is actually not so hard. There are three main
> components:
>  >
>  > 1. Declaration.
>  > 2. Runtime infrastructure.
>  > 3. Resolution algorithm.
>  >
>  > I'm pretty sure that #2 is easy (and covered in my "Import constraints"
>  > thread).
>  >
>  > And we don't yet have *any* import declaration mechanism (yes, we're
>  > assuming that this will be handled in 294, but it hasn't been brought up
>  > yet). I don't see this as a difficult issue either way. It would be
>  > natural to have both import-by-name and import-by-package in
>  > superpackages, if we have import at all. If we need to use annotations
>  > instead, it is easy to have both types.
>  >
>  > So I can only assume that the *real* concern here is #3. And I can't
>  > tell if this is a valid concern or not. I hope Glyn and Richard will
>  > point out what I'm missing here...
>
> Sounds right to me. However, the JSR 291 support for import-package is
> more subtle, and I would say was generally harder to get right (over
> several versions and several years), than require-bundle which was added
> fairly simply (despite the semantic rough edges, pointed out in the
> spec.). Although JSR 277 has paid lip service to learning from OSGi, the
> design is radically different in several ways:
>
> * dependencies expressed in metadata and code rather than purely in
> metadata
> * split packages not allowed
> * shallow/deep validation to avoid class space inconsistency rather than
> metadata expressing package dependencies and a system provided resolver
> that aims to find a "best fit"
>
> so my feeling is that we've got a quite a lot of work to do to support
> something equivalent to import-by-package. Of course, it may turn out to
> be simpler than I expect...

And that's what I'm struggling to determine. I do understand that 277
has taken a different approach, and so there is some cognitive
dissonance here.

But, the list you cite above seems a bit hand-wavy to me (sorry :^),
perhaps just distinctions without a difference.

My question below still stands. In general, it seems to me that
resolution consists of:

1. Selection of candidate module/bundles based on import declarations.

2. Class space consistency validation (which may narrow the list if
multiple choices, or result in a failure).

I don't understand what possible difference it could make whether we use
a module name or a package name to select candidates in step 1. Or even
if some custom code makes that selection (though I'm not a fan of this
either.)

Can anyone explain what difference it might make?

The selection criteria are entirely up to the developer/deployer; our
job is to make the best choices *within* that scope at runtime, or fail
if required if they violate the rules.

// Bryan

>  > What is the difference between selecting a ModuleDefinition based on its
>  > name, or selecting it based on its contents? Either way, "shallow"
>  > validation must still occur.
>  >
>  > // Bryan
>  >
>  > >
>  > > - Stanley
>  > >
>
> Glyn
>
>
> ------------------------------------------------------------------------
>
> /
> /
>
> /Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU/
>
>
>
>
>
>

From heavy at UNGOVERNED.ORG  Tue Jun  5 15:33:41 2007
From: heavy at UNGOVERNED.ORG (Richard S.Hall)
Date: Tue, 05 Jun 2007 18:33:41 -0400
Subject: Relationship to JSR 291 [was: Re: Bryan's comments]
In-Reply-To: <4665DC8F.8000402@oracle.com>
References: <OFC12639E7.6C1E83B9-ON802572F1.00310AA2-802572F1.0033EC6C@uk.ibm.com>
	<4665DC8F.8000402@oracle.com>
Message-ID: <81d16f0ad9ea912c29d9ba241ef0ce33@ungoverned.org>

On Jun 5, 2007, at 5:58 PM, Bryan Atsatt wrote:

> Glyn Normington wrote:
>>
>> *Bryan Atsatt <bryan.atsatt at ORACLE.COM>* wrote on 04/06/2007 20:21:29:
>>
>>  > Stanley M. Ho wrote:
>>  > > Hi Bryan,
>>  > >
>>  > > I don't think I will have time to response to all your comments
>> this
>>  > > week, so my response would focus on the import-by-package
>> discussion.
>>  > >
>>  > > First, the only import dependency we have consensus in this EG so
>> far is
>>  > > import-by-module-name (or import-by-superpackage-name), and this
>> was
>>  > > also what's in the EDR. That said, it is by no mean that we can't
>>  > > consider other import dependency granularities (e.g.
>>  > > import-by-package-name, or import-by-contract) in addition to
>>  > > import-by-module-name. I would like to focus this discussion
>> more on
>>  > > whether it makes sense for JSR 277 to support additional import
>>  > > dependency granularity, mainly import-by-package-name, and if
>> so, in
>>  > > what context.
>>  > >
>>  > > There are three cases we should consider:
>>  > >
>>  > > 1. 277 module imports another 277 module by package name
>>  > >
>>  > > I think our consensus so far is that this is nice-to-have or
>>  > > non-critical, given we already have support for
>> import-by-module-name.
>>  > > To keep this discussion in focus, let's not dive into
>>  > > import-by-package-name v.s. import-by-module-name, and let's
>> assume
>>  > > there is no support for #1 for now.
>>  >
>>  > Agreed.
>>  >
>>  > >
>>  > > 2. OSGi module imports 277 module by package name
>>  > >
>>  > > As I discussed this with Richard in previous emails, majority of
>> the
>>  > > works for this fall under the OSGi framework if it wants to
>> enable this
>>  > > kind of wiring, but we could probably make the implementor's life
>> easier
>>  > > by provide the appropriate 277 reflective API if necessary.
>>  >
>>  > Yes, the API would be useful.
>>  >
>>  > >
>>  > > 3. 277 module imports OSGi module by package name
>>  > >
>>  > > If I understand you correctly, I think what you really want is
>> the
>>  > > support for this case, so the existing OSGi bundles can be
>> leveraged by
>>  > > 277 modules in a nature way.
>>  > >
>>  > > Let me say it up front that I don't oppose the idea of #3 in the
>> context
>>  > > of interoperability. If #3 is what you really want, I think
>> where we
>>  > > diverge is simply how to make it possible.
>>  > >
>>  > > Do you agree so far?
>>  >
>>  > Yes.
>>  >
>>  > >
>>  > > Bryan Atsatt wrote:
>>  > >> ...
>>  > >> I want 277 to explicitly expose the concept of import-by-package
>> through
>>  > >> appropriate APIs.
>>  > >>
>>  > >> It would be very nice if the 277ModuleSystem implementation
>> itself
>>  > >> directly supports import-by-package (case #3). But this is less
>> critical
>>  > >> to me than case #4...
>>  > >>
>>  > >> I want a 277ModuleDefinition to be able to import-by-package an
>>  > >> OSGiModuleDefinition (i.e. a bundle exposed via the 277 APIs). I
>> believe
>>  > >> this case will be a particularly important one. By the time SE 7
>> ships,
>>  > > ...
>>  > >
>>  > > As I described several emails back:
>>  > >
>>  > >  > This doesn't mean supporting the notion of import by package
>> is
>>  > >  > impossible. What OSGi will expose from its repository is
>> simply
>> module
>>  > >  > definitions, and module definition is basically an
>> abstraction.
>>  > >  > Therefore, it should be possible to treat an OSGi bundle as a
>> module
>>  > >  > definition, as well as to treat each exported package in an
>> OSGi
>>  > >  > bundle a module definition as well. The dependency supported
>> in JSR
>>  > >  > 277 is import-by-module. In the JSR 277 module system, this is
>> mapped
>>  > >  > to import-by-superpackage. In the OSGi module system, this
>> can be
>>  > >  > mapped to import-by-OSGi-bundle or import-by-package. If other
>> module
>>  > >  > systems want to support different import granularity or if
>> OSGi
>>  > >  > evolves to add new import granularity in the future, this
>> model
>> should
>>  > >  > still work.
>>  > >
>>  > > Rather than surfacing the import-by-package at the API level, I
>> think we
>>  > > can solve #3 by generalizing the import dependency concept as
>>  > > import-by-name, and this name can be mapped to superpackage-name
>> in JSR
>>  > > 277 and OSGi-bundle-name or OSGi-exported-package-name in OSGi
>> (i.e.
>>  > > exposing OSGi bundles and exported packages through implementing
>>  > > OSGiBundleModuleDefinition and OSGiPackageModuleDefinition). I
>> think
>>  > > this is also what you intended to say at one point (correct me
>> if I
>>  > > misunderstand you) by suggesting to change
>>  > > ImportDependency.getModuleName() into something like
>>  > > ImportDependency.getName().
>>  >
>>  > I believe we are now thinking along the same lines.
>>  >
>>  > I just sent an email on the topic of generalizing the import
>> dependency
>>  > concept, with the subject "Import constraints". With the approach I
>>  > suggest there, the distinctions between module-name and
>> package-name are
>>  > mostly transparent to the runtime (declarative support for
>> package-name
>>  > is not addressed).
>>  >
>>  > That isn't to say that there aren't resolution issues to be worked
>> out
>>  > for package-name, but its a start.
>>  >
>>  > >
>>  > > We expect the module-aware compiler would somehow recognize the
>> import
>>  > > dependency defined in 277. In other words, if the OSGi framework
>> exposes
>>  > > the module definitions appropriately, you'll get the compiler
>> support.
>>  > > Since the import dependency is generic, the compiler and other
>> tools ed
>>  > > to do extra work to support other forms of import dependency.
>>  >
>>  > Yep. As I've said from the very beginning, I think the compiler is
>> going
>>  > to have to rely on the 277 runtime to resolve *external* (i.e.
>>  > non-source) dependencies. So as long as the runtime can deal with
>>  > package-name, the compiler will too.
>>  >
>>  > (Dependencies resolvable via included source will, I assume, just
>>  > *directly* resolve, without applying any constraint logic. That
>> is, if a
>>  > class refers to a class in another superpackage, and that class &
>>  > superpackage are available in the same compilation, the compiler
>> will
>>  > simply select them without applying *any* constraints. This may be
>> too
>>  > simple of a model, but maybe it is sufficient.)
>>  >
>>  > >
>>  > > In case if the 277 module would like to import these OSGi
>> exported
>>  > > packages using an OSGi-like resolution algorithm, it should be
>> possible
>>  > > for it to use a custom import policy (or import override policy
>> or
>> maybe
>>  > > some new override mechanisms) to get the behavior it wants.
>>  > >
>>  > > Do you think this is a workable approach to address #3?
>>  >
>>  > I don't think custom code should be required; we need to make it
>> easier
>>  > than that or it doesn't qualify as "first-class" citizenship, at
>> least
>>  > not to me :^).
>>  >
>>  >
>>  > I guess I am still missing something fundamental--I keep thinking
>> that
>>  > import-by-package is actually not so hard. There are three main
>> components:
>>  >
>>  > 1. Declaration.
>>  > 2. Runtime infrastructure.
>>  > 3. Resolution algorithm.
>>  >
>>  > I'm pretty sure that #2 is easy (and covered in my "Import
>> constraints"
>>  > thread).
>>  >
>>  > And we don't yet have *any* import declaration mechanism (yes,
>> we're
>>  > assuming that this will be handled in 294, but it hasn't been
>> brought up
>>  > yet). I don't see this as a difficult issue either way. It would be
>>  > natural to have both import-by-name and import-by-package in
>>  > superpackages, if we have import at all. If we need to use
>> annotations
>>  > instead, it is easy to have both types.
>>  >
>>  > So I can only assume that the *real* concern here is #3. And I
>> can't
>>  > tell if this is a valid concern or not. I hope Glyn and Richard
>> will
>>  > point out what I'm missing here...
>>
>> Sounds right to me. However, the JSR 291 support for import-package is
>> more subtle, and I would say was generally harder to get right (over
>> several versions and several years), than require-bundle which was
>> added
>> fairly simply (despite the semantic rough edges, pointed out in the
>> spec.). Although JSR 277 has paid lip service to learning from OSGi,
>> the
>> design is radically different in several ways:
>>
>> * dependencies expressed in metadata and code rather than purely in
>> metadata
>> * split packages not allowed
>> * shallow/deep validation to avoid class space inconsistency rather
>> than
>> metadata expressing package dependencies and a system provided
>> resolver
>> that aims to find a "best fit"
>>
>> so my feeling is that we've got a quite a lot of work to do to support
>> something equivalent to import-by-package. Of course, it may turn out
>> to
>> be simpler than I expect...
>
> And that's what I'm struggling to determine. I do understand that 277
> has taken a different approach, and so there is some cognitive
> dissonance here.
>
> But, the list you cite above seems a bit hand-wavy to me (sorry :^),
> perhaps just distinctions without a difference.
>
> My question below still stands. In general, it seems to me that
> resolution consists of:
>
> 1. Selection of candidate module/bundles based on import declarations.
>
> 2. Class space consistency validation (which may narrow the list if
> multiple choices, or result in a failure).
>
> I don't understand what possible difference it could make whether we
> use
> a module name or a package name to select candidates in step 1. Or even
> if some custom code makes that selection (though I'm not a fan of this
> either.)

In general, I can say your two steps above are correct...Felix actually
has what amounts to a two-pass resolve that does (1) from above in the
first step and (2) from above in the second step. However, it turns out
that (2) is actually quite difficult and difficult to do in an
efficient way.

Perhaps it is more difficult for the OSGi framework due to some of the
sophistication of its model, e.g., package substitutability, fragments,
split packages, package-level uses constraints, etc.

All I know is that we are still finding cases that need to be clarified
in the OSGi framework for this process...

I won't say that I am disagreeing with you...if we can strip away some
of the complexity of the OSGi model, then I am sure you could implement
a reasonably straightforward resolver. That won't necessarily help us
in achieving interoperability, though.

-> richard


>
> Can anyone explain what difference it might make?
>
> The selection criteria are entirely up to the developer/deployer; our
> job is to make the best choices *within* that scope at runtime, or fail
> if required if they violate the rules.
>
> // Bryan
>
>>  > What is the difference between selecting a ModuleDefinition based
>> on its
>>  > name, or selecting it based on its contents? Either way, "shallow"
>>  > validation must still occur.
>>  >
>>  > // Bryan
>>  >
>>  > >
>>  > > - Stanley
>>  > >
>>
>> Glyn
>>
>>
>> ----------------------------------------------------------------------
>> --
>>
>> /
>> /
>>
>> /Unless stated otherwise above:
>> IBM United Kingdom Limited - Registered in England and Wales with
>> number
>> 741598.
>> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire
>> PO6 3AU/
>>
>>
>>
>>
>>
>>

From Stanley.Ho at sun.com  Wed Jun  6 14:51:26 2007
From: Stanley.Ho at sun.com (Stanley M. Ho)
Date: Wed, 06 Jun 2007 14:51:26 -0700
Subject: JSR 277 EG observer mailing list
In-Reply-To: <46646B30.2030500@oracle.com>
References: <46400CFB.1020004@sun.com> <46412161.9020902@sun.com>
	<4648BA20.5070206@sun.com> <46646B30.2030500@oracle.com>
Message-ID: <46672C5E.9070607@sun.com>

Hi Bryan,

Bryan Atsatt wrote:
> How are people supposed to find this? Shouldn't there be an easily
> spotted link somewhere on the public 277 page?

Yes, it is mentioned in the JSR 277 community update page:

http://jcp.org/en/egc/view?id=277

but unfortunately it requires JCP member login.

It is also mentioned in the OpenJDK project which will cover the
reference implementation (RI) of JSR 277 and JSR 294, and hopefully
people will be able to find it:

http://openjdk.java.net/projects/modules

- Stanley

From Stanley.Ho at sun.com  Wed Jun  6 15:02:28 2007
From: Stanley.Ho at sun.com (Stanley M. Ho)
Date: Wed, 06 Jun 2007 15:02:28 -0700
Subject: Relationship to JSR 291 [was: Re: Bryan's comments]
In-Reply-To: <466555B3.9000903@ungoverned.org>
References: <OF62C11B58.0C61920A-ON802572EB.0055969A-802572EB.005897E5@uk.ibm.com>
	<465DF2E4.4090603@sun.com>
	<9E52FB97-F312-4A4F-A096-ECA0430F9D7C@ungoverned.org>
	<465F4D94.9060809@sun.com>
	<0e79b722e610c3de839326fb68539389@ungoverned.org>
	<465F62FC.7070201@sun.com>
	<2071f55c84384fb5d141568f491677c9@ungoverned.org>
	<4664D591.1060608@sun.com> <466555B3.9000903@ungoverned.org>
Message-ID: <46672EF4.7060201@sun.com>

Hi Richard,

Richard S. Hall wrote:
> This is definitely true, but this was only introduced in R4 and is still
> intended to be the exception, not the norm...from my understanding, 277
> defined the norm in the opposite direction.
>
> Ultimately, if we are making the same assumption, which is that people
> will generally be exporting all public classes from a package and only
> occasionally filtering some classes that should not be public, and we
> are promoting this as the best practice, then we are at least in the
> same ball park.

Whether people will be exporting individual exported public classes or
exporting all public classes from a package but excluding a few specific
classes is simply a syntax issue; the result is still a set of "exported
public classes." I think as long as we have the same concept of
"exported public classes," we are in the same ball park.

> Well, effectively all OSGi dependency resolution comes down to packages
> at some point, so if we are going to use 277 modules at all in the OSGi
> framework, then we have to have some way to determine their set of
> exported packages, which is necessary for determining consistency.

Sounds reasonable.

- Stanley

From Stanley.Ho at sun.com  Wed Jun  6 16:17:45 2007
From: Stanley.Ho at sun.com (Stanley M. Ho)
Date: Wed, 06 Jun 2007 16:17:45 -0700
Subject: Exported resources
In-Reply-To: <4664E9C6.30608@oracle.com>
References: <4654109D.40205@sun.com> <46548736.7020507@oracle.com>
	<465629EA.2050008@sun.com> <465658CE.5020409@oracle.com>
	<465DCE51.4010304@sun.com> <465DDCD0.8090703@oracle.com>
	<465DF6D6.5050608@sun.com> <465E0201.2030308@oracle.com>
	<465F68C8.4000103@sun.com> <46609428.4000909@oracle.com>
	<4664C2B1.1050109@sun.com> <4664E9C6.30608@oracle.com>
Message-ID: <46674099.5040908@sun.com>

Hi Bryan,

Bryan Atsatt wrote:
> ...
> frameworks/libraries are precisely what a module system is for: to allow
> their clients to *avoid* bundling the jars. And they aren't likely to be
> deployed as extensions, either; not when there's this nice repository
> model. Consider Spring. Or any of the hundreds of other third party
> frameworks/libraries out there. We *want* these to be deployed as
> normal, separate modules.

I agreed. My point was that in many cases, these frameworks/libraries
would be deployed into the repository and loaded with all permissions at
runtime, so this should not be an issue even they are deployed as
normal, separate modules.

> There is a much bigger issue, which is that this whole model falls apart
> simply because we can't grant this type of "friend" access to module
> private classes...
>
> Your description of how ResourceBundle or ServiceLoader accesses the
> class contains a big assumption, one that *could* be correct for such
> JRE code, but is almost certainly wrong for third-party code.
>
> These types of frameworks call Class.forName() or loader.loadClass().
> And then they call either Class.newInstance() or Constructor.newInstance().
>
> Regardless of whether or not they invoke any other methods, JSR 294 will
> require an access control check at the point of instantiation, extending
> those it performs today. Look at how Class.newInstance() and
> Constructor.newInstance() both use:
>
>     Reflection.ensureMemberAccess()
>
> This is the logical point at which to inject the new module private
> access control check.
>
> Could this be made to succeed for JRE callers? Of course (but it
> wouldn't be right, IMO).
>
> But, more importantly: it won't, and cannot, succeed for third-party
> callers without some sort of friend model for classes. Which isn't on
> the table.
>
> So, again, if we're not going to do this for classes, why should we do
> it for resources?

Sorry, I still don't think I fully understand your point. From my
perspective, the access model for exported classes and exported
resources are orthogonal. I agreed that it would be nice to have some
symmetries between them, but I think we already concluded why this is
not feasible because of their fundamental differences. The fact that
this is not how we support classes today doesn't mean we could not do it
for resources.

Perhaps it would help to pull out the following question you had couple
emails back for our discussion:

Bryan Atsatt wrote:
 > And what about the mismatch issue I brought up? Lots of existing
 > frameworks use the "services" pattern, in which:
 >
 > 1. A well known resource name is used in a getResource() call.
 > 2. The resource contains a string naming a provider class.
 > 3. The provider class is loaded using Class.forName().
 >
 > This is a simple but very powerful pattern, but it requires that *both*
 > the resource and the named class be accessible to the framework. If we
 > require a permission grant for the resource, but not for the class, it
 > will be very easy to get access to one and not the other. Is it really
 > worth opening up this potential headache to enable a friend model for
 > resources?

In today's model, if the resource is available, #1 will always succeed.
If the provider class is available, then #3 will also succeed, and
whether the class is accessible will be determined by the JVM. The
entire "exported resources" issue is basically around the behavioral
change in #1; this is orthogonal to #2 and #3, and I think we should
ignore the entire "exported classes" issue for the moment to keep this
discussion focus.

That said, I realize that we might have been thinking about two very
different use cases (correct me if I misunderstand you) here:

a. Some service-related frameworks (e.g. ServiceLoader) need to retrieve
resources like META-INF/services/<service-provider> from the module
classloader.
b. A module uses the ResourceBundle APIs to retrieve property-based
resources or list-based resources from its own module.

For (a) alone, I would agree with you that this kind of resource might
simply be exported. In fact, I don't think this should even be
considered a private resource because its content is intended for
external consumption.

For (b), I don't think it makes sense to require a module exporting the
private resources for its own consumption, and this is where I hope the
security permission approach would help to make it work.

Suppose we say that resources like (a) should be exported while
resources like (b) should remain private (and we'll use the security
permission approach to allow module to access its own private
resources), do you think you can live with this?

- Stanley

From bryan.atsatt at oracle.com  Wed Jun  6 16:48:07 2007
From: bryan.atsatt at oracle.com (Bryan Atsatt)
Date: Wed, 06 Jun 2007 16:48:07 -0700
Subject: Exported resources
In-Reply-To: <46674099.5040908@sun.com>
References: <4654109D.40205@sun.com> <46548736.7020507@oracle.com>
	<465629EA.2050008@sun.com> <465658CE.5020409@oracle.com>
	<465DCE51.4010304@sun.com> <465DDCD0.8090703@oracle.com>
	<465DF6D6.5050608@sun.com> <465E0201.2030308@oracle.com>
	<465F68C8.4000103@sun.com> <46609428.4000909@oracle.com>
	<4664C2B1.1050109@sun.com> <4664E9C6.30608@oracle.com>
	<46674099.5040908@sun.com>
Message-ID: <466747B7.2030709@oracle.com>

Stanley M. Ho wrote:
> Hi Bryan,
>
> Bryan Atsatt wrote:
>> ...
>> frameworks/libraries are precisely what a module system is for: to allow
>> their clients to *avoid* bundling the jars. And they aren't likely to be
>> deployed as extensions, either; not when there's this nice repository
>> model. Consider Spring. Or any of the hundreds of other third party
>> frameworks/libraries out there. We *want* these to be deployed as
>> normal, separate modules.
>
> I agreed. My point was that in many cases, these frameworks/libraries
> would be deployed into the repository and loaded with all permissions at
> runtime,

That last phrase seems like a huge leap to me. How exactly would this
grant all permissions? And, even if it did, it seems like an *extremely*
large security hole!

(But, again, I don't feel the need to dwell on the grant issue at the
moment--we have bigger fish to fry.)

> so this should not be an issue even they are deployed as
> normal, separate modules.
>
>> There is a much bigger issue, which is that this whole model falls apart
>> simply because we can't grant this type of "friend" access to module
>> private classes...
>>
>> Your description of how ResourceBundle or ServiceLoader accesses the
>> class contains a big assumption, one that *could* be correct for such
>> JRE code, but is almost certainly wrong for third-party code.
>>
>> These types of frameworks call Class.forName() or loader.loadClass().
>> And then they call either Class.newInstance() or
>> Constructor.newInstance().
>>
>> Regardless of whether or not they invoke any other methods, JSR 294 will
>> require an access control check at the point of instantiation, extending
>> those it performs today. Look at how Class.newInstance() and
>> Constructor.newInstance() both use:
>>
>>     Reflection.ensureMemberAccess()
>>
>> This is the logical point at which to inject the new module private
>> access control check.
>>
>> Could this be made to succeed for JRE callers? Of course (but it
>> wouldn't be right, IMO).
>>
>> But, more importantly: it won't, and cannot, succeed for third-party
>> callers without some sort of friend model for classes. Which isn't on
>> the table.
>>
>> So, again, if we're not going to do this for classes, why should we do
>> it for resources?
>
> Sorry, I still don't think I fully understand your point. From my
> perspective, the access model for exported classes and exported
> resources are orthogonal. I agreed that it would be nice to have some
> symmetries between them, but I think we already concluded why this is
> not feasible because of their fundamental differences. The fact that
> this is not how we support classes today doesn't mean we could not do it
> for resources.
>
> Perhaps it would help to pull out the following question you had couple
> emails back for our discussion:
>
> Bryan Atsatt wrote:
>  > And what about the mismatch issue I brought up? Lots of existing
>  > frameworks use the "services" pattern, in which:
>  >
>  > 1. A well known resource name is used in a getResource() call.
>  > 2. The resource contains a string naming a provider class.
>  > 3. The provider class is loaded using Class.forName().
>  >
>  > This is a simple but very powerful pattern, but it requires that *both*
>  > the resource and the named class be accessible to the framework. If we
>  > require a permission grant for the resource, but not for the class, it
>  > will be very easy to get access to one and not the other. Is it really
>  > worth opening up this potential headache to enable a friend model for
>  > resources?
>
> In today's model, if the resource is available, #1 will always succeed.
> If the provider class is available, then #3 will also succeed, and
> whether the class is accessible will be determined by the JVM. The
> entire "exported resources" issue is basically around the behavioral
> change in #1; this is orthogonal to #2 and #3, and I think we should
> ignore the entire "exported classes" issue for the moment to keep this
> discussion focus.
>
> That said, I realize that we might have been thinking about two very
> different use cases (correct me if I misunderstand you) here:
>
> a. Some service-related frameworks (e.g. ServiceLoader) need to retrieve
> resources like META-INF/services/<service-provider> from the module
> classloader.
> b. A module uses the ResourceBundle APIs to retrieve property-based
> resources or list-based resources from its own module.
>
> For (a) alone, I would agree with you that this kind of resource might
> simply be exported. In fact, I don't think this should even be
> considered a private resource because its content is intended for
> external consumption.
>
> For (b), I don't think it makes sense to require a module exporting the
> private resources for its own consumption, and this is where I hope the
> security permission approach would help to make it work.
>
> Suppose we say that resources like (a) should be exported while
> resources like (b) should remain private (and we'll use the security
> permission approach to allow module to access its own private
> resources), do you think you can live with this?

It isn't just me that can't live with it :^). You are correct that
private *property-based* resources could be retrieved this way, but
incorrect about *list-based* resources.

"List-based" resources are... *Class* instances. Specifically, they are
subclasses of ListResourceBundle. They must be loaded and instantiated
by ResourceBundle.

If ListResourceBundle subclasses are module private,
ResourceBundle.getBundle() will fail. And we have no mechanism to grant
access.

// Bryan

>
> - Stanley
>

From Stanley.Ho at sun.com  Wed Jun  6 18:36:52 2007
From: Stanley.Ho at sun.com (Stanley M. Ho)
Date: Wed, 06 Jun 2007 18:36:52 -0700
Subject: Exported resources
In-Reply-To: <466747B7.2030709@oracle.com>
References: <4654109D.40205@sun.com> <46548736.7020507@oracle.com>
	<465629EA.2050008@sun.com> <465658CE.5020409@oracle.com>
	<465DCE51.4010304@sun.com> <465DDCD0.8090703@oracle.com>
	<465DF6D6.5050608@sun.com> <465E0201.2030308@oracle.com>
	<465F68C8.4000103@sun.com> <46609428.4000909@oracle.com>
	<4664C2B1.1050109@sun.com> <4664E9C6.30608@oracle.com>
	<46674099.5040908@sun.com> <466747B7.2030709@oracle.com>
Message-ID: <46676134.8020206@sun.com>

Hi Bryan,

Bryan Atsatt wrote:
>> Suppose we say that resources like (a) should be exported while
>> resources like (b) should remain private (and we'll use the security
>> permission approach to allow module to access its own private
>> resources), do you think you can live with this?
>
> It isn't just me that can't live with it :^). You are correct that
> private *property-based* resources could be retrieved this way, but
> incorrect about *list-based* resources.
>
> "List-based" resources are... *Class* instances. Specifically, they are
> subclasses of ListResourceBundle. They must be loaded and instantiated
> by ResourceBundle.

Oops... I missed that. ;-)

> If ListResourceBundle subclasses are module private,
> ResourceBundle.getBundle() will fail. And we have no mechanism to grant
> access.

If some class needs access to non-public/exported details of another
class, you can always use privileged reflection. That is the solution
today and I expect it will be available with superpackages too. In other
words, it is possible for Res