From dbhole at redhat.com Mon Jul 13 10:37:14 2009 From: dbhole at redhat.com (Deepak Bhole) Date: Mon, 13 Jul 2009 13:37:14 -0400 Subject: Jigsaw/jpkg meeting synopsis Message-ID: <20090713173714.GA3453@redhat.com> Hi, My name is Deepak Bhole and I work at Red Hat on the Java team. Last week, we had a meeting with Mark and Dalibor from Sun, and few others here at Red Hat. The purpose was to try and come up with a plan for Jigsaw/jpkg integration for various Linux distros. Here is a summary of what came out of it: The meeting focused mostly on the jpkg tool, and how it and similar tools/scripts can help distributions integrate the Jigsaw module system. For posterity, I am going to include all approaches that were discussed, and their pros and cons. ------------------------------------------------------------------------------- == Iteration 1 == Goal ---- Make it easy to produce native packages that take advantage of the dependency information in module-info file. Approach -------- The original idea was to have a tool -- jpkg -- that would accept built class files, resources, and additional metadata and use it to create consumable packages (rpm, deb, etc). Pros ---- Easy for independent developers to provide natively consumable distribution units. Cons ---- The problem with this approach is integration with various major distribution build systems. Fedora for example, uses a system called Koji. The workflow for Koji is as follows: 1. Download upstream source code 2a. Write up spec file that will call ant/maven/etc. 2b. Set up locations for where the jars will end up on the target system 2c. Write post install/uninstall scripts as needed 3. Upload the spec file and sources to cvs, and call Koji build 4. Koji then checks out from cvs, unpacks the sources, builds, writes the final rpms, sends them for signing, and makes them available in the repos jpkg tries to solve #2. However, 2b varies based on distributions. That is the first part of the problem -- different distributions have different conventions, and coming up with a generic one that appeals to everyone would be is quite difficult, if not impossible. The second part of the problem is that jpkg produces rpms, which have no entry point into #4 (once step 3 is done, everything is automated). With no entry point to #4, it can have no influence over final packages. == Iteration 2 == Goal ---- Unchanged. Approach -------- As seen in the cons for iteration 1, jpkg would be unusable in distributions that have their own build infrastructure. So the next iteration was to have jpkg create rpm spec file. The developer could then modify these as needed, and send them off to build. Pros ---- Allows flexibility across distributions, as packagers can adapt the spec file to their distribution conventions. Allows integration with the build system. Cons ---- Once a spec file is written by the tool, it will begin to diverge. Essentially creating a branched version. This could be mitigated by having jpkg take an existing spec file and update it. However, what was a 1 stage process before, is now 2 stages and more work. == Iteration 3 == Goal ---- Unchanged. Approach -------- The approach that we ended up with, is one where packagers have to do minimal work and the benefits of the module system are reaped behind the scenes. The jpkg tool can have 2 phases. The first phase will compute the dependencies, module database location, etc., and there will be an option to have it dump this data into some agreed upon format, and exit. Otherwise, it will continue on to the second phase and produce final rpms. Distributions will integrate scripts that will be able to call jpkg phase 1, and take its output and dynamically change the output RPM to inject dependencies, add post scripts, etc. Thus under this approach, the distribution packgers would have to simply update their packages to remove manual dependencies, and rebuild -- the new rpm scripts would take care of the rest. Pros ---- Packagers will have to do little to provide Jigsaw integration. Independent developers can use jpkg to build full rpms. Since jpkg output will be used by rpm finalization scripts, it can provide metadata that conforms to a globally unique java namespace, so that independently developed packages can work with system installed ones. Cons ---- Integration with rpm/deb will be challenging work. Especially ensuring that independent + distribution packages play nice. Most of the onus will be on Java however, as rpm/deb can only unpack to target locations and update databases. It is upto Java to figure out how to use it all in an integrated manner. ------------------------------------------------------------------------------- For the first little while, we would need a hybrid system that remains compatible with what distributions currently ship, and Jigsaw. This wouldn't be too difficult thanks to symlinks. RPM can continue installing wherever they usually install. The automated scripts can take care of linking to these from a centralized location. The implementation details of it all still need to be hammered out, but it should be doable. Deepak From Mandy.Chung at Sun.COM Mon Jul 13 16:35:50 2009 From: Mandy.Chung at Sun.COM (Mandy Chung) Date: Mon, 13 Jul 2009 16:35:50 -0700 Subject: Startup Improvement with -Xverify:none Message-ID: <4A5BC4D6.8050109@sun.com> Mark, I did some experiments in running the startup benchmarks with -Xverify:none to get a closer approximation of the potential gain if the module system supports pre-verification. http://cr.openjdk.java.net/~mchung/startup_measurement/startup.result.b63.noverify Summary (based on the measurement with the refworkload benchmarks) ======= 1) Pre-verification could improve startup time by 8-13%. 2) Class data sharing improves the startup time by 4.5-18.5%, half of the time is class parsing time. Class parsing could be one candidate to look for optimization opportunity. Need further brainstorming. 3) Looking up application classes takes about 2.27-7.0%. With module system, class lookup should speed up since the module system knows which module a class belongs to. Need to measure this when it's ready. 4) The data archive (JRE/bin/client/classes.jsa) is generated at JRE install time with a predefined classlist. With modularized JDK, we need to make sure either the class data sharing technique to work with JDK modules (e.g. regenerate the classes.jsa when new JDK module is installed), or other optimization in place that the startup time improvement due to class data sharing would still be observed. FYI. The changeset for my HotSpot instrumentation is already in hotspot-rt [1]. Mandy [1] http://hg.openjdk.java.net/jdk7/hotspot-rt/hotspot/rev/6a93908f268f From mr at sun.com Mon Jul 13 22:13:08 2009 From: mr at sun.com (Mark Reinhold) Date: Mon, 13 Jul 2009 22:13:08 -0700 Subject: Startup Improvement with -Xverify:none In-Reply-To: mandy.chung@sun.com; Mon, 13 Jul 2009 16:35:50 PDT; <4A5BC4D6.8050109@sun.com> Message-ID: <20090714051308.CE539931A@callebaut.niobe.net> > Date: Mon, 13 Jul 2009 16:35:50 -0700 > From: mandy.chung at sun.com > I did some experiments in running the startup benchmarks with > -Xverify:none to get a closer approximation of the potential gain if the > module system supports pre-verification. > > http://cr.openjdk.java.net/~mchung/startup_measurement/startup.result.b63.noverify Thanks -- these are promising results! - Mark From mcconnell at dpml.net Fri Jul 31 09:24:13 2009 From: mcconnell at dpml.net (Stephen J. McConnell) Date: Sat, 01 Aug 2009 01:54:13 +0930 Subject: Runtime impact on URL handlers Message-ID: <1249057453.14328.20.camel@gloria> Hello everyone. I've been reading through all of the JSR 294 and Jigsaw material with interest. One of the questions I have had in the back of my mind (which remains unresolved) is the impact that 294 and Jigsaw would have of URL handler management. As things stand in SE6, a URL handler must be installed as a part of a static system classloader (unless an alternative system classloader is defined in which case the alternative must be included in the system classpath). If we take a leap forward to Java 7 with modules, I would like to get some clarification on how the following would work: Sample Java source: URL url = URI.create( "link:part:org/acme/widget" ).toURL(); Widget widget = (Widget) url.getContent( new Class[]{ Widget.class } ); In the above example we have some code which has a dependency on the availability of a 'link' protocol handler. Under my SE6 implementation this works providing the class executes in a JVM with the link handler installed as a system extension. In a Java 7/294/Jigsaw world, (a) will it be possible to define a module that has a declared dependency on a system-level module (i.e. the link protocol handler module described above), and (b) will it be possible to load the consumer module (the module containing the sample code) into a runing JVM with assurance that the system level protocol handler will be available (in effect dynamically) to the Java URLHandler machinery? Cheers, Steve. -- Stephen J. McConnell mailto:mcconnell at dpml.net http://www.dpml.net mobile: 04 5800 3980