<html>
<head>
<meta http-equiv="content-type" content="text/html;
charset=ISO-8859-1">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<b>Background</b><br>
<br>
The fundamental challenge with serialization is that the code that
defined a class at serialization time may have changed by the time
deserialization happens. Serialization is defined to be tolerant of
change to a certain extent, and admits a degree of customization to
allow additional flexibility. <br>
<br>
For ordinary classes, there are three lines of defense:<br>
<ul>
<li>serialVersionUID</li>
<li>serialization hooks</li>
<li>default schema evolution</li>
</ul>
<p>Serial version UID for the target class must match exactly. By
default, serialization uses a serial version UID which is a hash
of the classes signatures. So this default approach means "any
significant change to the structure of the class (adding new
methods, changing method or field signatures, etc) renders
serialized forms invalid". It is a common practice to explicitly
assign a serial version UID to a class, thereby disabling this
mechanism. <br>
</p>
<p>Classes that expect to evolve over time may use
readObject/writeObject and/or readResolve/writeReplace to
customize the mapping between object state and bytestream. If
classes do not use this mechanism, serialization uses a default
schema evolution mechanism to adjust for changes in fields between
serialization and deserialization time; fields that are present in
the bytestream but not in the target class are ignored, and fields
that are present in the target class but not the bytestream get
default values (zero, null, etc.) <br>
</p>
<p>Anonymous classes follow the same approach and have access to the
same mechanisms (serialVersionUID, read/writeObject, etc), but
they have two additional sources of instability:<br>
</p>
<ul>
<li>The name is generated as EnclosingClass$nnn. Any change to
the set of anonymous classes in the enclosing class may cause
sequence numbers to change.</li>
<li>The number and type of fields (appears in bytecode but not
source code) are generated based on the set of captured values.
Any change to the set or order of captured values can cause
these signatures to change (in an unspecified way). </li>
</ul>
If the signatures remain stable, anonymous classes can use
serialization hooks to customize the serialized form, just like
named classes.
<p>The EG has observed that users have largely learned to deal with
the problems of serialization of inner classes, either by (a)
don't do it, or (b) ensure that essentially the same bits are
present on both sides of the pipe, preventing skew from causing
instability in either class names or signatures. <br>
</p>
<p>The EG has set, as a minimum bar, that lambda serialization be
"at least as good as" anonymous class serialization. (This is not
a high bar.) Further, the EG has concluded that gratuitous
deviations from anonymous class serialization are undesirable,
because, if users have to deal with an imperfect scheme, having
them deal with something that is basically the same as an
imperfect scheme they've already gotten used to is preferable to
dealing with a new and different scheme. <br>
</p>
<p>Further, the EG has rejected the idea of arbitrarily restricting
access to serialization just because it is dangerous; users who
have learned to use it safely should not be unduly encumbered. <br>
</p>
<p><b>Failure modes<br>
</b></p>
<p>For anonymous classes, one of two things will happen when
attempting to deserialize after things have changed "too much":<br>
</p>
<ol>
<li>A deserialization failure due to either the name or signature
not matching, resulting in NoSuchMethodError,
IncompatibleClassChangeError, etc. <br>
</li>
<li>Deserializing to the wrong thing, without any evidence of
error. <br>
</li>
</ol>
<p>Obviously, a type-2 failure is far worse than a type-1 failure,
because no error is raised and an unintended computation is
performed. Here are two examples of changes that are behaviorally
compatible but which will result in type-2 failures. The first
has to do with order-of-declaration. <br>
</p>
<table width="100%" border="1" cellpadding="2" cellspacing="2">
<tbody>
<tr>
<td valign="top"><b>Old code</b><b><br>
</b></td>
<td valign="top"><b>New code</b><b><br>
</b></td>
<td valign="top"><b>Result</b><b><br>
</b></td>
</tr>
<tr>
<td valign="top"><font color="#3333ff"><tt>Runnable r1 = new
Runnable() { </tt><tt><br>
</tt><tt> void run() {</tt><tt><br>
</tt><tt> System.out.println("one");</tt><tt><br>
</tt><tt> }</tt><tt><br>
</tt><tt>};</tt></font><tt><br>
</tt><tt>Runnable r2 = new Runnable() { </tt><tt><br>
</tt><tt> void run() {</tt><tt><br>
</tt><tt> System.out.println("two");</tt><tt><br>
</tt><tt> }</tt><tt><br>
</tt><tt> };</tt><tt><br>
</tt></td>
<td valign="top"><tt>Runnable r2 = new Runnable() { </tt><tt><br>
</tt><tt> void run() {</tt><tt><br>
</tt><tt> System.out.println("two");</tt><tt><br>
</tt><tt> }</tt><tt><br>
</tt><tt> };</tt><tt><br>
</tt><font color="#3333ff"><tt>Runnable r1 = new Runnable()
{ </tt><tt><br>
</tt><tt> void run() {</tt><tt><br>
</tt><tt> System.out.println("one");</tt><tt><br>
</tt><tt> }</tt><tt><br>
</tt><tt> };</tt></font><tt><br>
</tt> </td>
<td valign="top">Deserialized r1 (across skew) prints "two".<br>
</td>
</tr>
</tbody>
</table>
<p>This fails because in both cases, we get classes called Foo$1 and
Foo$2, but in the old code, these correspond to r1 and r2, but in
the new code, these correspond to r2 and r1. <br>
</p>
<p>The other failure has to do with order-of-capture. <br>
</p>
<table width="100%" border="1" cellpadding="2" cellspacing="2">
<tbody>
<tr>
<td valign="top"><b>Old code</b><b><br>
</b></td>
<td valign="top"><b>New code</b><b><br>
</b></td>
<td valign="top"><b>Result</b><b><br>
</b></td>
</tr>
<tr>
<td valign="top"><tt>String s1 = "foo";<br>
String s2 = "bar";<br>
Runnable r = new Runnable() { </tt><tt><br>
</tt><tt> void run() {</tt><tt><br>
</tt><tt> <font color="#3333ff">foo(s1, s2)</font>;</tt><tt><br>
</tt><tt> }</tt><tt><br>
</tt><tt>};</tt><tt><br>
</tt><tt><br>
</tt></td>
<td valign="top"><tt><tt>String s1 = "foo";<br>
String s2 = "bar";<br>
</tt>Runnable r = new Runnable() { </tt><tt><br>
</tt><tt> void run() {<br>
<font color="#cc0000"> String s = s2;<br>
</font></tt><font color="#cc0000"><tt> </tt></font><tt>
<font color="#cc0000">foo(s1, s)</font>;</tt><tt><br>
</tt><tt> }</tt><tt><br>
</tt><tt>};</tt><tt><br>
</tt><tt> </tt> </td>
<td valign="top">On deserialization, s1 and s2 are effectively
swapped.<br>
</td>
</tr>
</tbody>
</table>
<p>This fails because the order of arguments in the implicitly
generated constructor of the inner class changes due to the order
in which the compiler encounters captured variables. If the
reordered variables were of different types, this would cause a
type-1 failure, but if they are the same type, it causes a type-2
failure. <br>
</p>
<p><b>User expectations</b><br>
</p>
<p>While experienced users are quick to state the "same bits on both
sides" rule for reliable deserialization, a bit of investigation
reveals that user expectations are actually higher than that. For
example, if the compiler generated a <i>random</i> name for each
lambda at compile time, then recompiling the same source with the
same compiler, and using the result for deserialization, would
fail. This is too restrictive; user expectations are not tied to
"same bits", but to a vaguer notion of "I compiled essentially the
same source with essentially the same compiler, and therefore
didn't change anything significant." For example, users would
balk if adding a comment or changing whitespace were to affect
deserialization. Users likely expect (in part, due to behavior of
anonymous classes) changes to code that doesn't affect the lambda
directly or indirectly (e.g., add or remove a debugging println)
also would not affect the serialized form. <br>
</p>
<p>In the absence of the user being able to explicitly name the
lambda <i>and</i> its captures (as C++ does), there is no perfect
solution. Instead, our goal can only be to minimize type-2
failures while not unduly creating type-1 failures when "no
significant code change" happened. This means we have to put a
stake in the ground as to what constitutes "significant" code
change. <br>
</p>
<p>The de-facto (and likely accidental) definition of "significant"
used by inner classes here is: <br>
</p>
<ul>
<li>Adding, removing, or reordering inner class instances earlier
in the source file;</li>
<li>Changes to the number, order, or type of captured arguments</li>
</ul>
<p>This permits changes to code that has nothing to do with inner
classes, and many common refactorings as long as they do not
affect the order of inner class instances or their captures. <br>
</p>
<p><b>Current Lambda behavior</b><br>
</p>
<p>Lambda serialization currently behaves very similarly to
anonymous class serialization. Where anonymous classes have
stable method names but unstable class names, lambdas are the
dual; unstable method names but stable class names. But since
both are used together, the resulting naming stability is largely
the same. <br>
</p>
We do one thing to increase naming stability for lambdas: we hash
the name and signature of the enclosing method in the lambda name.
This insulates lambda naming from the addition, removal, or
reordering of methods within a class file, but naming stability
remains sensitive to the order of lambdas within the method.
Similarly, order-of-capture issues are largely similar to inner
classes. <br>
<p>Lambdas bodies are desugared to methods named in the following
form: lambda$<i>mmm</i>$<i>nnn</i>, where <i>mmm</i> is a hash of
the method name and signature, and <i>nnn</i> is a sequence
number of lambdas that have the same <i>mmm</i> hash. <br>
</p>
<p>Because lambdas are instantiated via invokedynamic rather than
invoking a constructor directly, there is also slightly more
leniency to changes to the <i>types</i> of captured argument;
changing a captured argument from, say, String to Object, would be
a breaking change for anonymous classes (it changes the
constructor signature) but not for lambdas. This leniency is
largely an accidental artifact of translation, rather than a
deliberate design decision.<br>
</p>
<p><b>Possible improvements</b><br>
</p>
<p>We can start by recognizing the role of the hash of the enclosing
method in the lambda method name. This reduces the set of lambdas
that could collide from "all the lambdas in the file" to "all the
lambdas in the method." This reduces the set of changes that
cause both type-1 and type-2 errors. <br>
</p>
<p>An additional observation is that there is a tension between
trying to <i>recover from</i> skew (rather than simply trying to
detect it, and failing deserialization) and complexity. So I
think we should focus primarily on detecting skew and failing
deserialization (turning type-2 failures into type-1) while at the
same time not unduly increasing the set of changes that cause
type-1 errors, with the goal of settling on an informal guideline
of what constitutes "too much" change. <br>
</p>
<p>We can do this by increasing the number of things that affect the
<i>mmm</i> hash, effectively constructing the lambda-equivalent of
the serialization version UID. The more context we add to this
hash, the smaller the set of lambdas that hash to the same bucket
gets, which reduces the space of possible collisions. The
following table shows possible candidates for inclusion, along
with examples of code that illustrate dependence on this item. <br>
</p>
<table width="100%" border="1" cellpadding="2" cellspacing="2">
<tbody>
<tr>
<td valign="top"><b>Item</b><b><br>
</b></td>
<td valign="top"><b>Old Code</b><b><br>
------------------------------<br>
</b></td>
<td valign="top"><b>New Code</b><b><br>
</b><b>------------------------------</b><br>
</td>
<td valign="top"><b>Effect</b><b><br>
</b></td>
<td valign="top"><b>Rationale</b><b><br>
</b></td>
</tr>
<tr>
<td valign="top">Names of captured arguments<br>
</td>
<td valign="top">int <font color="#3333ff">x</font> = ...<br>
f(() -> <font color="#3333ff">x</font>);<br>
</td>
<td valign="top">int <font color="#cc0000">y</font> = ...<br>
f(() -> <font color="#cc0000">y</font>);</td>
<td valign="top">Including the names of captured arguments in
the hash would cause rename-refactors of captured arguments
to be considered a serialization-breaking change. <br>
</td>
<td valign="top">While alpha-renaming is generally considered
to be semantic-preserving, serialization has always keyed
off of names (such as field names) as being clues to
developer intent. It seems reasonable to say "If you change
the names involved, we have to assume a semantic change
occurred." We cannot tell if a name change is a simple
alpha-rename or capturing a completely different variable,
so this is erring on the safe side.<br>
</td>
</tr>
<tr>
<td valign="top">Types of captured arguments<br>
</td>
<td valign="top"><font color="#3333ff">String </font>x = ...<br>
f(() -> x);</td>
<td valign="top"><font color="#cc0000">Object </font>x = ...<br>
f(() -> x);</td>
<td valign="top"><br>
</td>
<td valign="top">It seems reasonable to say that, if you
capture arguments of a different type, you've made a
semantic change. <br>
</td>
</tr>
<tr>
<td valign="top">Order of captured arguments<br>
</td>
<td valign="top">() -> { <br>
<font color="#3333ff"> int a = f(x);<br>
</font> int b = g(y);<br>
return h(a,b);<br>
};<br>
</td>
<td valign="top">() -> { <br>
int b = g(y);<br>
<font color="#3333ff"> int a = f(x);<br>
</font> return h(a,b);<br>
};</td>
<td valign="top">Changing the order of capture would become a
type-1 failure rather than possibly a type-2 failure. <br>
</td>
<td valign="top">Since we cannot detect whether the ordering
change is semantically meaningful or not, it is best to be
conservative and say: change to capture order is likely a
semantic change. <br>
</td>
</tr>
<tr>
<td valign="top">Variable assignment target (if present)<br>
</td>
<td valign="top"><font color="#3333ff">Runnable r1 = Foo::f;<br>
</font>Runnable r2 = Foo::g;<br>
</td>
<td valign="top">Runnable r2 = Foo::g;<br>
<font color="#3333ff"> Runnable r1 = Foo::f;</font><br>
<br>
</td>
<td valign="top">Including variable target name would render
this reordering recoverable and correct<br>
</td>
<td valign="top">If the user has gone to the effort of
providing a name, we can use this as a hint to the meaning
of the lambda. <br>
</td>
</tr>
<tr>
<td valign="top"><br>
</td>
<td valign="top">Runnable <font color="#3333ff">r</font> =
Foo::f;</td>
<td valign="top">Runnable <font color="#cc0000">runnable </font>=
Foo::f;</td>
<td valign="top">Including variable target name would render
this change (previously recoverable and correct) a
deserialiation failure<br>
</td>
<td valign="top">If the user has changed the name, it seems
reasonable to treat that as possibly meaning something else.<br>
</td>
</tr>
<tr>
<td valign="top">Target type<br>
</td>
<td valign="top"><font color="#3333ff">Predicate<String>
</font>p = String::isEmpty;<br>
</td>
<td valign="top"><font color="#cc0000">Function<String,
Boolean> </font>p = String::isEmpty;</td>
<td valign="top">Including target type reduces the space of
potential sequence number collisions.<br>
</td>
<td valign="top">If you've changed the target type, it is a
different lambda.<br>
</td>
</tr>
</tbody>
</table>
<p>This list is not exhaustive, and there are others we might
consider. (For example, for lambdas that appear in method
invocation context rather than assignment context, we might
include the hash of the invoked method name and signature, or even
the parameter index or name. This is where it starts to exhibit
diminishing returns and increasing brittleness.) <br>
</p>
<p>Taken in total, the effect is:<br>
</p>
<ul>
<li>All order-of-capture issues become type-1 failures, rather
than type-2 failures (modulo hash collisions). <br>
</li>
<li>Order of declaration issues are still present, but they are
dramatically reduced, turning many type-2 failures into type-1
failures.</li>
<li>Some new type-1 failures are introduced, mostly those deriving
from rename-refactors. </li>
</ul>
<p>The remaining type-2 failures could be dealt with if we added
named lambdas in the future. (They are also prevented if users
always assign lambdas to local variables whose names are unique
within the method; in this way, the local-variable trick becomes a
sort of poor-man's named lambda.)<br>
</p>
<p>We can reduce the probability of collision further by using a
different (and simpler) scheme for non-serializable lambdas
(lambda$nnn), so that serializable lambdas can only accidentally
collide with each other.<br>
</p>
<p>However, there are some transformations which we will still not
be able to avoid under this scheme. For example:<br>
</p>
<table width="100%" border="1" cellpadding="2" cellspacing="2">
<tbody>
<tr>
<td valign="top"><b>Old code</b><b><br>
</b></td>
<td valign="top"><b>New code</b><b><br>
</b></td>
<td valign="top"><b>Result</b><b><br>
</b></td>
</tr>
<tr>
<td valign="top"><tt>Supplier<Integer> s = <br>
<font color="#3333ff">foo </font>? <font
color="#3333ff">() -> 1 </font><br>
: () -> 2;<br>
</tt><tt> </tt></td>
<td valign="top"><tt>Supplier<Integer> s = <br>
<font color="#cc0000">!foo</font> ? () -> 2 <br>
: <font color="#3333ff">() -> 1</font>;</tt><tt>
</tt><tt> </tt> </td>
<td valign="top">This change is behaviorally compatible but
could result in type-2 failure, since both lambdas have the
same target type, capture arity, etc. <br>
</td>
</tr>
</tbody>
</table>
<ul>
</ul>
<p>However^2, we can still detect this risk and warn the user. If
for any <i>mmm</i>, we issue more than one sequence number <i>nnn</i>,
we are at risk for a type-2 failure, and can issue a lint warning
in that case, suggesting the user refactor to something more
stable. (Who knows what that diagnostic message will look like.)
With all the hash information above, it seems likely that the
number of potentially colliding lambdas will be small enough that
this warning would not come along too often. <br>
</p>
<p>The impact of this change in the implementation is surprisingly
small. It does not affect the serialized form
(java.lang.invoke.SerializedLambda), or the generated
deserialization code ($deserialize$). It only affects the code
which generates the lambda method name, which needs access to a
small additional bit of information -- the assignment target
name. Similarly, detecting the condition required for warning is
easy -- "sequence number != 1". <br>
</p>
<p>Qualitatively, the result is still similar in feel to inner
classes -- you can make "irrelevant" changes but we make no heroic
attempts to recover from things like changes in capture order --
but we do a better job of detecting them (and, if you follow some
coding discipline, you can avoid them entirely.) <br>
</p>
<p><br>
</p>
</body>
</html>