Modularization [was: Jigsaw prototype, take 2]
niftiness at gmail.com
Fri Sep 6 07:33:33 PDT 2013
I think there are two very different things which a "module system"
addresses, and it seems like folks on this list have differing views of
which is important or even if both of them are:
1. Carving up the JDK into smaller chunks for performance
2. Defining how Java software can be componentized in general and providing
discovery and enforcement mechanisms, to increase the efficiency of Java
developers and decrease the maintenance cost of what they write.
The most straightforward way to do 1. for the JDK team will result in doing
at least some of 2. - add at least minimal dependency management. Sure,
that could be done as a one-off, ad-hoc kind of thing - but it's going to
be about the same amount of work as to create a more generalized mechanism
which other things can use. And because Java licensees will need to use
the result, it has to be documented and not terribly crufty. So the desire
to do a very scoped back "module system" to solve just that problem is
There are a lot of degrees of "modularity", and probably a lot of different
definitions of what that is on this list. My background is a lot of work
with the NetBeans module system when I was at Sun, which does quite a few
- Fail fast on startup if a dependency is missing, or disable functionality
- Discover and validate dependencies at runtime, including checking
- Forbid circular dependencies
- Require all dependencies to be declared explicitly (no transient closure)
- Enforce that a module may only reference classes from its dependencies
with hierarchical classloaders
- Extend the Java visibility rules to make possible "jar-private" public
classes (non-public packages)
- Provide a mechanism for a module to specify that it needs *some*
implementation of a spec but doesn't care which one (similar to a program
needing a mailer but not caring if it's sendmail, postfix or exim)
- Allow modules to be installed and uninstalled dynamically at runtime
All of these things are useful; a few of them are difficult to do without
others, but Many are fairly orthogonal and could stand alone. You could
probably meet most people's test of modularity with only some of them. Not
modular with a capital-M, but the bare minimum.
I often find when there is a lot of iteration without a lot of progress (or
with a lot of argument), usually people are using the same words for
different things. When somebody says "modular", in fact, they could be
talking about any of, or any subset of, the list above.
Re the JDK: You could probably just do "fails fast if a dependency is
missing", claim "we modularized the JDK" and declare victory (which seemed
to be the proposal in the last few messages). All you'd have to implement
is a classloader which fails if any JAR it expects is missing, and a way to
say what JARs should be there. That's hardly worth a JSR - it's an evening
or two's coding (and a lot more testing). But as an incremental starting
step, if it leaves the door open for the other things in the above list, it
might be a pretty good start.
The problems which need to be solved are larger than breaking up the JDK
into more JARs + a spec to specify them. The last ten years have been about
our entire industry discovering that directed graphs of dependencies are
really, really important. You can't write software or use computers the
way a developer does without tripping over somebody's dependency management
system every few minutes - maven dependencies, netbeans module
dependencies, Gentoo Linux ebuild dependencies, SmartOS pkgsrc
dependencies, RPMs, Debian packages, NodeJS modules, Bower client-side
they have the same problem and invented things that all converge on the
same patterns of solution (yet are incompatible :-)).
Another way of saying this is that it turns out, everything is system
software - or benefits by pretending to be - and there is no class of
software developer that benefits long-term from the freedom to be sloppy.
There is an annoying hubris among framework authors that goes something
like "I'm writing a big important framework. People who use my framework
are writing sloppy little tinkertoy apps and couldn't possibly need the
tools to solve the problems big important frameworks have!" It leads to
solutions which are just naive enough not to be usable. The JDK has been
guilty of this (ServiceLoader, anyone?), but *everybody* does it unless
they're on guard for it.
The problems the list above solves are fundamental to distributed software
development and large teams. I compile my OS from source - and it's safe
to say that most of the people who wrote the software on my computer have
never met. Nobody's going to get everybody who works on Linux in a room or
on a call, and if you could it would be wildly unproductive. Modularity
and it's enforcement - whether as OS-level packages or Java modules - makes
it possible for people not co-located in space or time to work
independently and be productive.
With our entire industry discovering and dealing with those problems,
because solving it is critical to getting stuff done, it seems unlikely
that Java gets a pass to ignore it without bad consequences. Which is why
just writing a classloader and a spec for dependencies and saying "Done!
It's modular!" is a good start, but isn't actually going to help enough to
I get that many people are clamoring for a leaner JDK. Is it the "most
pressing concern"? It depends whom you talk to. Certainly, looking at
V8's startup performance, it's clear there's room for improvement.
In my post-Sun consulting life, aside from fixing people's threading, the
thing I often get hired to do is to detangle someone's dependencies, inject
some discipline into how their codebase is architected - by small-m
modularizing it; teach them how to think about software design so that
they won't end up back in a similar mess later; and add some enforcement
mechanisms to their build process so that the wrong thing stops being the
easiest thing. Nobody sets out to end up with a codebase their own people
don't understand - yet routinely that's where software shops end up -
really, really often.
The value of medium-m to capital-M modularity isn't that it makes things
pretty. I'm a pragmatist. It's that it improves quality and productivity (I
can back up that assertion, but this email is already long).
The value is that you get better parallelism from developers if the
software is less coupled and has natural areas - modules - in which
developers can work independently. Things like API package enforcement
further improve that because it is easier to predict the effect of changes
in components with well-defined, tightly scoped contracts. The value of
modularity, to me, is all about people, not architecture for architecture's
All of that is to say, there is another pressing requirement from Java
developers (one fewer know they have, but which keeps me employed): Make
it harder to write broken things and easier to write maintainable things.
It's the difference between trying to invent the world's best ice scraper
for car windshields versus inventing the remote control that lets you start
your car from in the house so it can defrost itself.
I don't think it's an all or nothing proposition. A lot of the things on
that list above are orthagonal. I don't think they're deserving of
individual JSRs, but it would make sense to carve them up as separate
More information about the jigsaw-dev