<4CBDA6E7.1050501@oracle.com> <4CBDBCC1.9020706@univ-mlv.fr> <4CBDBC13.8060601@oracle.com> <4CBDE118.4030103@univ-mlv.fr> <4CBF15BA.4020703@univ-mlv.fr> <4CBF36B3.8020203@univ-mlv.fr> <4CC80634.5000908@oracle.com> Message-ID: <4CDAA631.5040508@optrak.co.uk> On 10/11/2010 13:16, Llewellyn Falco wrote: > I just realized this is a mistake! > > Comparable will not work properly without the Equals method > being overridden. > > Namely, if you have two object a, b you will get > a.compareTo(b) == 0 > but > a.equals(b) == false From the JavaDoc for Comparable: "It is strongly recommended (though not required) that natural orderings be consistent with equals." Note it is a recommendation not a requirement. BigDecimal is an example where the natural ordering is not consistent with equals. Mark Thornton From alex.blewitt at gmail.com Wed Nov 10 06:29:11 2010 From: alex.blewitt at gmail.com (Alex Blewitt) Date: Wed, 10 Nov 2010 14:29:11 +0000 Subject: Lambdas and serialization In-Reply-To: References:

<4CE1B938.6050509@oracle.com> Message-ID: <201011160835.02197.peter.levart@marand.si> On 11/15/10, Alex Buckley wrote: > Hi Ali, > > On 11/15/2010 2:51 AM, Ali Ebrahimi wrote: > > if compiler can accept the following syntax: > > > >

> extension A max() default > > Trait.max; > > > > The aim is here to restrict calling of max only on CustomizedLists of > > Comparables, otherwise compiler raise compile time error. > > > > I know this is a new concept- a kind of conditional inheritance. but > > currently this can not be implemented in java, unless by introducing a new > > subinterface. then we have two interfaces. > > You and Steven have identified a valuable idiom that avoids new subtypes > - we can call it "conditional inheritance" if you like - but > unfortunately the generics mechanism cannot express it. Namely, the > bound of a type variable can be _either_ another type variable _or_ a > combination of classes and interfaces. (See JLS3 4.4.) > > The trouble is erasure: if a type variable's bound could be 'T & > Comparable<...>', then it would not be clear whether to erase max's > return type to Object (the erasure of T) or Comparable (the erasure of > Comparable<...>). I guess we could pick the former, but the Signature > attribute would want to record both T and Comparable<..> for later > typechecking of A's witness from the call site. (Does it subtype List's > type argument _and_ Comparable<..> ?) > > JLS3 4.4 arranged things to ensure a unique erasure, so it's not just a > matter of relaxing syntax to allow your desired type variable for max. > > I personally believe we will want to add extension methods that further > restrict the enclosing generic type's bounds, and that a new > subinterface (ComparableList) will often be undesirable. > But more experience is needed. > > Alex Why are extension methods so special? Wouldn't also other kinds of methods and class constructors benefit from such a feature? Take for example the interface SortedSet and one of it's implementations: TreeSet. It would be desirable that constructors not taking a Comparator restrict type variable to a subtype of Comparable. Regards, Peter > > From peter.levart at marand.si Tue Nov 16 00:43:23 2010 From: peter.levart at marand.si (Peter Levart) Date: Tue, 16 Nov 2010 09:43:23 +0100 Subject: Problem of defender methods with generic methods In-Reply-To: <4CE1E5A7.6050206@oracle.com> References: <4CE1D5BA.9080206@comp.lancs.ac.uk> <4CE1E5A7.6050206@oracle.com> Message-ID: <201011160943.23514.peter.levart@marand.si> On 11/16/10, Alex Buckley wrote: > On 11/15/2010 4:52 PM, Steven Simpson wrote: > > Could it be done like this?: > > > > interface List { > > ... > > > extension T max() default Collections.max; > > } I would expect that 'new' is actualy implicit in generic method declarations. Every type variable currently declared in a generic method is it's own instance (i.e. new). What we might need is a syntax that says "this": interface List { ... > extension T max() default Collections.max; } // which would mean that for a particular method, the type variable of the type instance should be bound accordingly or compile-time error is raised at a call site. > > > > The generic method signature as it appears in the bytecode would be > > almost the same, but with some extra flag on T, e.g. '!': > > > > ;>()TT > > It could, but don't forget to document the impact of additional bounds > on subtyping (notably, containment of type arguments) and inference. The > real question is whether library evolution commonly needs to so restrict > type variables in the first place. > > Alex > > P.S. A better use of the "new" token in the context of a bound is to > mandate that witnesses of a type variable have a no-args constructor, so > you can say "new T()" in the body. But that's another story. > > From alex.buckley at oracle.com Tue Nov 16 10:58:21 2010 From: alex.buckley at oracle.com (Alex Buckley) Date: Tue, 16 Nov 2010 10:58:21 -0800 Subject: Problem of defender methods with generic methods In-Reply-To: <201011160835.02197.peter.levart@marand.si> References:

<4CE1B938.6050509@oracle.com> <201011160835.02197.peter.levart@marand.si> Message-ID: <4CE2D44D.4010205@oracle.com> On 11/15/2010 11:35 PM, Peter Levart wrote: > Why are extension methods so special? Wouldn't also other kinds of > methods and class constructors benefit from such a feature? Take for > example the interface SortedSet and one of it's implementations: > TreeSet. It would be desirable that constructors not taking a > Comparator restrict type variable to a subtype of Comparable. JSR 14 designed generics in a particular way, such as the bound of a type variable having certain restrictions. We do not re-open the design of a former JSR lightly. Extension methods (or rather, the library evolution that needs extension methods) may provide a strong enough reason. Alex From vincent at sevel.eu Thu Nov 18 09:09:13 2010 From: vincent at sevel.eu (Vincent Sevel) Date: Thu, 18 Nov 2010 18:09:13 +0100 Subject: capturing (or not) mutable local variables Message-ID: Hi Brian, I came to you yesterday at devoxx to ask about the rationale behind the decision to not capture mutable local variables. I understand that among several arguments, there is a strong one about preventing people from doing unsafe things. you gave the analogy of the kid, the fence and the pool. if I continue on the analogy, we could say that if the kid disguised himself (eg: final int[] sum = new int[1]) then you would be probably happy, although it did not take much to workaround your fence, and actually people are used to it. of course your fear completely disappears if the variable is of type anything as long as it is final. so you do not have problem with that: class BigGuyThatDoesnotKnowHowToSwim implements Foreachable { int sum = 0; public void do(Object elt) { sum += (Integer) elt; } } final BigGuyThatDoesnotKnowHowToSwim bigguy = new BigGuyThatDoesnotKnowHowToSwim(); mylist.foreach(bigguy); really, from the thread safety perspective, the disguised kid and the bigguy are not better than the kid. asking if the guy is a kid is actually the wrong question. you should be asking yourself whether or not the guy actually knows how to swim, assuming he needs to be able to. because you have kids that swim better than you and me, and you have grown up that do not know. this thought leads to another one. in your example you are assuming that the foreach method will get into multithreaded stuff. myself, I did not expect it to be. but if that is the case, truly we want to be careful about it. but should I assume that any method that accepts a block can get into MT? and should I design around that assumption? the basic problem is that you can't figure out what a pool is. so you are saying there are pools out there, let's protect anything that is accessible by kids. so you will protect football fields and playgrounds too. if you are going that route I could argue you could go even further. if multithread safety is the issue, and I cannot rely on the designer to document the limitations of the api, or I cannot rely on the user to use the api the designer had anticipated, we are actually at risk on any method that accepts an argument with state. on that particular issue, it is not about final variables, it is about passing mutable state that supports the contract set by the api. If you had told me that your impl was using MT stuff under the cover, then I would have written a very different block: Object sync = new Object(); int sum = 0; list.forEach(#{ e -> synchronized(synch) { sum += e.size(); }}); or if I knew there was a pool I would have written a BigGuyThatKnowsHowToSwim. in conclusion your fence is not only overprotecting in cases where we do not need it, but actually it provides protection only in a minority of cases. so really what you want to do if you are not confident that people will read the documentation and take responsibility for their mistakes, you should help them make sure they thought about the issue when we know there is certainty about it. your foreach impl coud say: public void forEach(@Multithreaded Foreachable block) {...} then I would get a warning or an error as a client if I tried that: list.forEach(#{ e -> sum += e.size(); }); but the error would disappear if I did: list.forEach(@ThreadSafe #{ e -> synchronized(synch) { sum += e.size(); }}); the good thing about this is compiler would still complain about the following, and that, in spite of having the final keyword, which is in no way a guarantee of thread safety: final int[] sum = new int[1]; list.forEach(#{ e -> sum[0] += e.size(); }); I believe that if you do multithread your foreach, you will have not more (or less for that matter) problems with or without capturing mutable variables. because people will write blocks of code that do not access local variables, but still manipulate pojos in a non thread safe way. for that matter, you should probably name your loop method forEachMT. as a final point I would say that the issue is not about mutable variables, it is about carrying clearly the contract constraints to the user of the api. if the spec stays that way, I am hoping that this is not based on the pool paradigm. in the jdk we should learn more from dynamic languages about trust and responsibility taking. regards. vincent From neal at gafter.com Fri Nov 19 16:52:50 2010 From: neal at gafter.com (Neal Gafter) Date: Fri, 19 Nov 2010 16:52:50 -0800 Subject: capturing (or not) mutable local variables In-Reply-To: References: Message-ID: Vincent- I for one agree with you wholeheartedly. Designing a language feature around one particular category of use case generally results in a poor language feature and a need for further extensions. I've written about this phenomenon before < http://gafter.blogspot.com/2006/09/failure-of-imagination-in-language_17.html >. Cheers, Neal On Thu, Nov 18, 2010 at 9:09 AM, Vincent Sevel wrote: > Hi Brian, I came to you yesterday at devoxx to ask about the rationale > behind the decision to not capture mutable local variables. > I understand that among several arguments, there is a strong one about > preventing people from doing unsafe things. you gave the analogy of > the kid, the fence and the pool. > > if I continue on the analogy, we could say that if the kid disguised > himself (eg: final int[] sum = new int[1]) then you would be probably > happy, although it did not take much to workaround your fence, and > actually people are used to it. > > of course your fear completely disappears if the variable is of type > anything as long as it is final. so you do not have problem with that: > > class BigGuyThatDoesnotKnowHowToSwim implements Foreachable { > int sum = 0; > > public void do(Object elt) { > sum += (Integer) elt; > } > } > > final BigGuyThatDoesnotKnowHowToSwim bigguy = new > BigGuyThatDoesnotKnowHowToSwim(); > mylist.foreach(bigguy); > > really, from the thread safety perspective, the disguised kid and the > bigguy are not better than the kid. asking if the guy is a kid is > actually the wrong question. you should be asking yourself whether or > not the guy actually knows how to swim, assuming he needs to be able > to. because you have kids that swim better than you and me, and you > have grown up that do not know. > > this thought leads to another one. in your example you are assuming > that the foreach method will get into multithreaded stuff. myself, I > did not expect it to be. but if that is the case, truly we want to be > careful about it. but should I assume that any method that accepts a > block can get into MT? and should I design around that assumption? the > basic problem is that you can't figure out what a pool is. so you are > saying there are pools out there, let's protect anything that is > accessible by kids. so you will protect football fields and > playgrounds too. > > if you are going that route I could argue you could go even further. > if multithread safety is the issue, and I cannot rely on the designer > to document the limitations of the api, or I cannot rely on the user > to use the api the designer had anticipated, we are actually at risk > on any method that accepts an argument with state. > > on that particular issue, it is not about final variables, it is about > passing mutable state that supports the contract set by the api. If > you had told me that your impl was using MT stuff under the cover, > then I would have written a very different block: > > Object sync = new Object(); > int sum = 0; > list.forEach(#{ e -> synchronized(synch) { sum += e.size(); }}); > > or if I knew there was a pool I would have written a > BigGuyThatKnowsHowToSwim. > > in conclusion your fence is not only overprotecting in cases where we > do not need it, but actually it provides protection only in a minority > of cases. so really what you want to do if you are not confident that > people will read the documentation and take responsibility for their > mistakes, you should help them make sure they thought about the issue > when we know there is certainty about it. > > your foreach impl coud say: > > public void forEach(@Multithreaded Foreachable block) {...} > > then I would get a warning or an error as a client if I tried that: > > list.forEach(#{ e -> sum += e.size(); }); > > but the error would disappear if I did: > > list.forEach(@ThreadSafe #{ e -> synchronized(synch) { sum += e.size(); > }}); > > the good thing about this is compiler would still complain about the > following, and that, in spite of having the final keyword, which is in > no way a guarantee of thread safety: > > final int[] sum = new int[1]; > list.forEach(#{ e -> sum[0] += e.size(); }); > > I believe that if you do multithread your foreach, you will have not > more (or less for that matter) problems with or without capturing > mutable variables. because people will write blocks of code that do > not access local variables, but still manipulate pojos in a non thread > safe way. > > for that matter, you should probably name your loop method forEachMT. > > as a final point I would say that the issue is not about mutable > variables, it is about carrying clearly the contract constraints to > the user of the api. if the spec stays that way, I am hoping that this > is not based on the pool paradigm. in the jdk we should learn more > from dynamic languages about trust and responsibility taking. > > regards. > > vincent > > From jesse.sightler at gmail.com Fri Nov 19 17:12:08 2010 From: jesse.sightler at gmail.com (Jesse Sightler) Date: Fri, 19 Nov 2010 20:12:08 -0500 Subject: capturing (or not) mutable local variables In-Reply-To: References:

Message-ID: As a user of the Java language I completely agree that the benefits of local mutable variable access far outweigh the risks of misuse. On Nov 19, 2010 7:54 PM, "Neal Gafter" wrote: > Vincent- > > I for one agree with you wholeheartedly. Designing a language feature > around one particular category of use case generally results in a poor > language feature and a need for further extensions. I've written about this > phenomenon before < > http://gafter.blogspot.com/2006/09/failure-of-imagination-in-language_17.html >>. > > Cheers, > Neal > > On Thu, Nov 18, 2010 at 9:09 AM, Vincent Sevel wrote: > >> Hi Brian, I came to you yesterday at devoxx to ask about the rationale >> behind the decision to not capture mutable local variables. >> I understand that among several arguments, there is a strong one about >> preventing people from doing unsafe things. you gave the analogy of >> the kid, the fence and the pool. >> >> if I continue on the analogy, we could say that if the kid disguised >> himself (eg: final int[] sum = new int[1]) then you would be probably >> happy, although it did not take much to workaround your fence, and >> actually people are used to it. >> >> of course your fear completely disappears if the variable is of type >> anything as long as it is final. so you do not have problem with that: >> >> class BigGuyThatDoesnotKnowHowToSwim implements Foreachable { >> int sum = 0; >> >> public void do(Object elt) { >> sum += (Integer) elt; >> } >> } >> >> final BigGuyThatDoesnotKnowHowToSwim bigguy = new >> BigGuyThatDoesnotKnowHowToSwim(); >> mylist.foreach(bigguy); >> >> really, from the thread safety perspective, the disguised kid and the >> bigguy are not better than the kid. asking if the guy is a kid is >> actually the wrong question. you should be asking yourself whether or >> not the guy actually knows how to swim, assuming he needs to be able >> to. because you have kids that swim better than you and me, and you >> have grown up that do not know. >> >> this thought leads to another one. in your example you are assuming >> that the foreach method will get into multithreaded stuff. myself, I >> did not expect it to be. but if that is the case, truly we want to be >> careful about it. but should I assume that any method that accepts a >> block can get into MT? and should I design around that assumption? the >> basic problem is that you can't figure out what a pool is. so you are >> saying there are pools out there, let's protect anything that is >> accessible by kids. so you will protect football fields and >> playgrounds too. >> >> if you are going that route I could argue you could go even further. >> if multithread safety is the issue, and I cannot rely on the designer >> to document the limitations of the api, or I cannot rely on the user >> to use the api the designer had anticipated, we are actually at risk >> on any method that accepts an argument with state. >> >> on that particular issue, it is not about final variables, it is about >> passing mutable state that supports the contract set by the api. If >> you had told me that your impl was using MT stuff under the cover, >> then I would have written a very different block: >> >> Object sync = new Object(); >> int sum = 0; >> list.forEach(#{ e -> synchronized(synch) { sum += e.size(); }}); >> >> or if I knew there was a pool I would have written a >> BigGuyThatKnowsHowToSwim. >> >> in conclusion your fence is not only overprotecting in cases where we >> do not need it, but actually it provides protection only in a minority >> of cases. so really what you want to do if you are not confident that >> people will read the documentation and take responsibility for their >> mistakes, you should help them make sure they thought about the issue >> when we know there is certainty about it. >> >> your foreach impl coud say: >> >> public void forEach(@Multithreaded Foreachable block) {...} >> >> then I would get a warning or an error as a client if I tried that: >> >> list.forEach(#{ e -> sum += e.size(); }); >> >> but the error would disappear if I did: >> >> list.forEach(@ThreadSafe #{ e -> synchronized(synch) { sum += e.size(); >> }}); >> >> the good thing about this is compiler would still complain about the >> following, and that, in spite of having the final keyword, which is in >> no way a guarantee of thread safety: >> >> final int[] sum = new int[1]; >> list.forEach(#{ e -> sum[0] += e.size(); }); >> >> I believe that if you do multithread your foreach, you will have not >> more (or less for that matter) problems with or without capturing >> mutable variables. because people will write blocks of code that do >> not access local variables, but still manipulate pojos in a non thread >> safe way. >> >> for that matter, you should probably name your loop method forEachMT. >> >> as a final point I would say that the issue is not about mutable >> variables, it is about carrying clearly the contract constraints to >> the user of the api. if the spec stays that way, I am hoping that this >> is not based on the pool paradigm. in the jdk we should learn more >> from dynamic languages about trust and responsibility taking. >> >> regards. >> >> vincent >> >> > From brian.goetz at oracle.com Sat Nov 20 01:29:33 2010 From: brian.goetz at oracle.com (Brian Goetz) Date: Sat, 20 Nov 2010 10:29:33 +0100 Subject: capturing (or not) mutable local variables In-Reply-To: References: Message-ID: <6BD37934-7BC2-4B66-9001-BC40D3A5622D@oracle.com> > Hi Brian, I came to you yesterday at devoxx to ask about the rationale > behind the decision to not capture mutable local variables. I admire your passion and your persistence :) > I understand that among several arguments, there is a strong one about > preventing people from doing unsafe things. you gave the analogy of > the kid, the fence and the pool. Safety mechanisms do not need to be perfect to be useful. Your arguments seem to center around "I can climb the fence, so there's no point in having it." This argument is, to be blunt, silly. The design principles of the Java language stress safety, even if there are cases where such safety guards can be "casted away" by a sufficiently motivated user. If you want to capture a not-effectively-final local variable in a closure, then one of the following two things has to happen: a) The lifetime of the variable must be extended to the lifetime of the closure b) The lifetime of the closure must be shortened to the lifetime of the variable. In both cases, we would want to ensure the variable is only accessed from the thread capturing the closure. In (a), we would be effectively creating a whole new class of variables (in addition to the seven (or eight, depending on how you count) already defined by the JLS); these are no longer local variables and should not look like them. In both (a) and (b), we would be creating a kind of restricted closure, whose execution is restricted in space (confined to a specific thread) and/or time (confined to the lifetime of the scope in which the variable is declared.) For concreteness let's call these confined variables and confined lambdas. It is possible to create mechanisms to reify and enforce these various types of confinement (we've discussed them extensively internally.) However, they add nontrivial complexity to the language. In order to justify the complexity that these new features would generate, there needs to be a compelling use case. When I explored this issue, I asked a number of people to write down a use case for this. Every one wrote some form of: int sum = 0; list.forEach( #{ x -> sum += x.foo() } In a parellel world, this idiom is irretrievably broken. (See Guy Steele's presentation "Organizing Functional Code for Parallel Execution, or, foldl considered slightly harmful.") People will do this, they will do this without thinking, and their code will be broken. It is very hard, even for experts, to get this right. While Java has plenty of other opportunities to create non-thread-safe code, I am not going to create an entirely new and nearly irresistible vector for doing so. Iteration and side-effects are how we've been trained to do things in Java, but we have to learn to do better. In sum, adding this feature that you (and others) want so badly seems to add up to: - Lots of new complexity (e.g., confined variables and lambdas) - Nearly irresistible new areas for making errors - All to prop up a broken programming model (iterations + side effects) To quote a past president: "Wouldn't be prudent." At this point in time, it feels the risk and complexity outweighs the benefit. I would rather put the effort into supporting map/reduce-y idioms in the libraries. The above block is much better as: list.reduce( #{ x, y => x+y } ) or list.reduce(Reducers.SUM) or list.sum() > this thought leads to another one. in your example you are assuming > that the foreach method will get into multithreaded stuff. Not quite: s/will/may/ > your foreach impl coud say: > > public void forEach(@Multithreaded Foreachable block) {...} We explored these issues in the JSR-166 expert group several years ago. The basic problem is that we don't have a way of reliably expressing "reference to thread-safe object" in the type system (either statically or dynamically), so these things revert to being documentation rather than type assertions. > for that matter, you should probably name your loop method forEachMT. Under consideration. Though my intuition is that (a) such a convention will be hard to stick to and (b) in five years it will probably look silly. From mthornton at optrak.co.uk Sat Nov 20 01:34:39 2010 From: mthornton at optrak.co.uk (Mark Thornton) Date: Sat, 20 Nov 2010 09:34:39 +0000 Subject: capturing (or not) mutable local variables In-Reply-To: References:

Message-ID: On Sun, Nov 21, 2010 at 1:21 AM, Alex Blewitt wrote: > On 21 Nov 2010, at 02:08, Neal Gafter wrote: > > > The point of my question is that mutable versus immutable has nothing to > do > > with variable lifetime. Pretending they have anything to do with each > other > > is just unnecessarily muddying the the discussion. > > Except that this assumes it is the same variable inside and outside the > lambda. If it is semantically a different variable (with the lifetime of the > closure) which has been assigned a copy of the value from the enclosing > scope then mutability, or lack thereof, is key. > Is it semantically different? It appears in all ways to be a single variable throughout its scope. The "lifetime of the variable" isn't part of the semantics today, and if you distinguish one part of the variable's scope from another by introducing this concept, you haven't added anything useful to the description. Do you mean to intend to allow the lambda to change the value of its copy? If so, your description would have some meaning. If not, you're suggesting a difference without a distinctions. From brian.goetz at oracle.com Sun Nov 21 07:50:50 2010 From: brian.goetz at oracle.com (Brian Goetz) Date: Sun, 21 Nov 2010 10:50:50 -0500 Subject: capturing (or not) mutable local variables In-Reply-To: <86BC574E-205E-456D-89A3-2B94BDBFABEE@miginfocom.com> References: <6BD37934-7BC2-4B66-9001-BC40D3A5622D@oracle.com> <06C56FFC-A923-4D56-B999-1207F2E12169@oracle.com> <86BC574E-205E-456D-89A3-2B94BDBFABEE@miginfocom.com> Message-ID: <4CE93FDA.3050907@oracle.com> You can think of effectively final as a form of type inference. There are lots of aspects of type inference coming in Java (diamond from Coin, target typing for lambda formals) and inferring finality is yet another. On 11/21/2010 6:39 AM, Mikael Grev wrote: > Excellent Brian. > > It would mean that a single compiler error would be shown when this is broken: > > - A variable cannot be changed after it has been used in a closure initialization. > > Easy, clear and user friendly. No need for funky hacks in the non-mutable case. > > This kind of capture will not be seen as limited as the current effectively-final definition, which only real merit is that it saves six keystrokes. > > > And I think there's a big opportunity here as well. The spin doctor in me say that if it is carefully communicated by Oracle that 2) is a very conscious decision and that the way you should do capture of mutable variables is by using an AtomicXxx I think you can almost totally get away with it. Maybe even get out on top of other languages since this is very multi threading aware from the start. > It is very important that all code examples for Lamdas have at least one example of a good use of AtomicXxx for mutable variables. Then this becomes the standard way of implementing mutability in parallel closures. There might even be room for some syntax sweetening of AtomicXxx usage in the future, which makes this even more compelling. > > Cheers, > Mikael > > > On Nov 20, 2010, at 21:41 PM, Brian Goetz wrote: > >> (2) is an interesting idea and worth considering. >> >> On Nov 20, 2010, at 5:37 AM, Mikael Grev wrote: >> >>> Please don't make it possible to mutate the variable from within the closure. >>> >>> >>> However, I wouldn't mind if "effectively-final" was extended so that either: >>> >>> 1) The closure works on a copy, giving no restrictions on the initialization and reassignment of the captured variable >>> 2) Any variable is effectively-final as long as it isn't reassigned after the closure initialization. (reordering should be solvable) >>> >>> The reason for this is that the following isn't that uncommon and "i" needs to be reassigned as another variable to be effectively-final (ugly): >>> >>> int i = 0; >>> if (something) >>> i = xxx; >>> >>> capture of i >>> >>> Loosening the definition of effectively-final would effectively make the feature easier to use. >>> >>> Cheers, >>> Mikael >>> >>> >>> On Nov 20, 2010, at 10:29 AM, Brian Goetz wrote: >>> >>>>> Hi Brian, I came to you yesterday at devoxx to ask about the rationale >>>>> behind the decision to not capture mutable local variables. >>>> >>>> I admire your passion and your persistence :) >>>> >>>>> I understand that among several arguments, there is a strong one about >>>>> preventing people from doing unsafe things. you gave the analogy of >>>>> the kid, the fence and the pool. >>>> >>>> Safety mechanisms do not need to be perfect to be useful. Your arguments seem to center around "I can climb the fence, so there's no point in having it." This argument is, to be blunt, silly. The design principles of the Java language stress safety, even if there are cases where such safety guards can be "casted away" by a sufficiently motivated user. >>>> >>>> If you want to capture a not-effectively-final local variable in a closure, then one of the following two things has to happen: >>>> a) The lifetime of the variable must be extended to the lifetime of the closure >>>> b) The lifetime of the closure must be shortened to the lifetime of the variable. >>>> >>>> In both cases, we would want to ensure the variable is only accessed from the thread capturing the closure. In (a), we would be effectively creating a whole new class of variables (in addition to the seven (or eight, depending on how you count) already defined by the JLS); these are no longer local variables and should not look like them. In both (a) and (b), we would be creating a kind of restricted closure, whose execution is restricted in space (confined to a specific thread) and/or time (confined to the lifetime of the scope in which the variable is declared.) For concreteness let's call these confined variables and confined lambdas. >>>> >>>> It is possible to create mechanisms to reify and enforce these various types of confinement (we've discussed them extensively internally.) However, they add nontrivial complexity to the language. >>>> >>>> In order to justify the complexity that these new features would generate, there needs to be a compelling use case. When I explored this issue, I asked a number of people to write down a use case for this. Every one wrote some form of: >>>> >>>> int sum = 0; >>>> list.forEach( #{ x -> sum += x.foo() } >>>> >>>> In a parellel world, this idiom is irretrievably broken. (See Guy Steele's presentation "Organizing Functional Code for Parallel Execution, or, foldl considered slightly harmful.") People will do this, they will do this without thinking, and their code will be broken. It is very hard, even for experts, to get this right. While Java has plenty of other opportunities to create non-thread-safe code, I am not going to create an entirely new and nearly irresistible vector for doing so. Iteration and side-effects are how we've been trained to do things in Java, but we have to learn to do better. >>>> >>>> In sum, adding this feature that you (and others) want so badly seems to add up to: >>>> - Lots of new complexity (e.g., confined variables and lambdas) >>>> - Nearly irresistible new areas for making errors >>>> - All to prop up a broken programming model (iterations + side effects) >>>> >>>> To quote a past president: "Wouldn't be prudent." At this point in time, it feels the risk and complexity outweighs the benefit. I would rather put the effort into supporting map/reduce-y idioms in the libraries. The above block is much better as: >>>> >>>> list.reduce( #{ x, y => x+y } ) >>>> or >>>> list.reduce(Reducers.SUM) >>>> or >>>> list.sum() >>>> >>>>> this thought leads to another one. in your example you are assuming >>>>> that the foreach method will get into multithreaded stuff. >>>> >>>> Not quite: s/will/may/ >>>> >>>>> your foreach impl coud say: >>>>> >>>>> public void forEach(@Multithreaded Foreachable block) {...} >>>> >>>> We explored these issues in the JSR-166 expert group several years ago. The basic problem is that we don't have a way of reliably expressing "reference to thread-safe object" in the type system (either statically or dynamically), so these things revert to being documentation rather than type assertions. >>>> >>>>> for that matter, you should probably name your loop method forEachMT. >>>> >>>> Under consideration. Though my intuition is that (a) such a convention will be hard to stick to and (b) in five years it will probably look silly. >>> >>> > > From per at bothner.com Sun Nov 21 12:18:16 2010 From: per at bothner.com (Per Bothner) Date: Sun, 21 Nov 2010 12:18:16 -0800 Subject: capturing (or not) mutable local variables In-Reply-To: References: <6BD37934-7BC2-4B66-9001-BC40D3A5622D@oracle.com> <03D40340-3C94-40EF-AE18-C586D8EB42F8@gmail.com>

Message-ID: <4CE97E88.5010803@bothner.com> On 11/20/2010 12:44 PM, Brian Goetz wrote: >> I think I'm hearing that mutating a local variable from a closure is just a >> bad idea in general right? > > For a closure that might conceivably be executed in another thread, this is correct. Agreed - but how is that different from mutating a field or array element from a closure that might conceivably be executed in another thread? You're just forcing the user to do by hand to work around a restriction. Note this is not a restriction in any other language I know that has both closures and mutable local variables. The traditional concept of lexical scoping (as in Lisp and Scheme) is that closures capture local bindings. If you disallow closing over mutable locals, then you only provide a restricted subset of closures. Of course having multiple threads invoking a closure that can mutate a variable is Bad, but what I'm not getting how this is different from multiple threads invoking a method that mutate a field. Yes, it might be nice to have a language that protects against the latter - but that is different language than Java. > Single-element array is disastrous since it seeks only to subvert the rules and doesn't provide needed synchronization. Which leads to the question: Since the rules as so trivially subvertable, why have the rules, since that they complicate real use-cases? Java really isn't a language for true multi-threaded programming - it's a language where multiple threads can co-ordinate with each other (when using appropriate care). So making "multiple threads" the justification for this restriction doesn't really make multi-threaded programming easier or safer. -- --Per Bothner per at bothner.com http://per.bothner.com/ From peter.levart at gmail.com Sun Nov 21 12:46:29 2010 From: peter.levart at gmail.com (Peter Levart) Date: Sun, 21 Nov 2010 21:46:29 +0100 Subject: capturing (or not) mutable local variables In-Reply-To: References: <407E1F2F-84B2-4673-9ED6-C3B9F3D5B914@oracle.com> Message-ID: <201011212146.30228.peter.levart@marand.si> On Sunday, November 21, 2010 03:08:34 am Neal Gafter wrote: > Are you saying that if we restrict lambdas to only variables that don't > change, then it is OK if the variable lifetime is not extended to the > lifetime of the lambda? In that case, does the code blow up when you > invoke a lambda that outlasts the block in which it was created? > > The point of my question is that mutable versus immutable has nothing to do > with variable lifetime. Pretending they have anything to do with each > other is just unnecessarily muddying the the discussion. That's true. But not entirely. Currently any mutable variable that can be accessed from multiple threads (instance or static) can be marked as volatile. Local variables can't be marked as volatile. That "restriction" would have to be lifted. Regards, Peter > > And yes, I do realize that lambda expressions (and the resulting variable > lifetime extension - no matter if the variables are final or not) are a > departure from the experience of typical java developers. I also realize > that programming with immutable data is a departure from the experience of > typical java developers. > > On Sat, Nov 20, 2010 at 1:24 PM, Brian Goetz wrote: > > Surely you realize (a) would be a huge departure from the experience of > > typical java developers. Who do not read the jls. > > > > Sent from my iPhone > > > > On Nov 20, 2010, at 4:08 PM, Neal Gafter wrote: > > > > On Sat, Nov 20, 2010 at 1:29 AM, Brian Goetz < > > > > brian.goetz at oracle.com> wrote: > >> If you want to capture a not-effectively-final local variable in a > >> closure, then one of the following two things has to happen: > >> a) The lifetime of the variable must be extended to the lifetime of the > >> closure > >> b) The lifetime of the closure must be shortened to the lifetime of the > >> variable. > > > > The JLS doesn't talk about the lifetime of local variables because, like > > all variables and dynamically allocated objects, they simply cease to > > exist when they can no longer be referenced. There would be no need to > > change that. From jim at pentastich.org Sun Nov 21 14:41:09 2010 From: jim at pentastich.org (Jim Mayer) Date: Sun, 21 Nov 2010 17:41:09 -0500 Subject: capturing (or not) mutable local variables In-Reply-To: <4CE97E88.5010803@bothner.com> References: <6BD37934-7BC2-4B66-9001-BC40D3A5622D@oracle.com> <03D40340-3C94-40EF-AE18-C586D8EB42F8@gmail.com>

<4CE97E88.5010803@bothner.com> Message-ID: I completely agree with Mr. Bothner's analysis. If I understand correctly, the motivation for disallowing access to mutable local variables is to make libraries that transparently use multiple threads (e.g., parallel array operations) safer. I think the restriction unnecessarily limits the functionality of closures in the (far more common today) single-threaded cases and is insufficient to make the multi-threaded case actually safe. I like the idea of an annotation on methods (@Parallel?) that could be used by the compiler on a "best effort" basis to warn of common unsafe practices (e.g., any unsynchronized access to mutable state) would help. I assume that's been proposed and debated many times already, though :-) --- Jim On Sun, Nov 21, 2010 at 3:18 PM, Per Bothner wrote: > On 11/20/2010 12:44 PM, Brian Goetz wrote: > >> I think I'm hearing that mutating a local variable from a closure is > just a > >> bad idea in general right? > > > > For a closure that might conceivably be executed in another thread, this > is correct. > > Agreed - but how is that different from mutating a field or array element > from a closure that might conceivably be executed in another thread? > You're just forcing the user to do by hand to work around a restriction. > > Note this is not a restriction in any other language I know that has > both closures and mutable local variables. The traditional concept of > lexical scoping (as in Lisp and Scheme) is that closures capture > local bindings. If you disallow closing over mutable locals, > then you only provide a restricted subset of closures. > > Of course having multiple threads invoking a closure that can > mutate a variable is Bad, but what I'm not getting how this is > different from multiple threads invoking a method that mutate > a field. Yes, it might be nice to have a language that protects > against the latter - but that is different language than Java. > > > Single-element array is disastrous since it seeks only to subvert the > rules and doesn't provide needed synchronization. > > Which leads to the question: Since the rules as so trivially > subvertable, why have the > rules, since that they complicate real use-cases? Java really isn't a > language > for true multi-threaded programming - it's a language where multiple > threads > can co-ordinate with each other (when using appropriate care). So making > "multiple threads" the justification for this restriction doesn't really > make multi-threaded programming easier or safer. > -- > --Per Bothner > per at bothner.com http://per.bothner.com/ > > From forax at univ-mlv.fr Sun Nov 21 16:32:29 2010 From: forax at univ-mlv.fr (=?ISO-8859-1?Q?R=E9mi_Forax?=) Date: Mon, 22 Nov 2010 01:32:29 +0100 Subject: capturing (or not) mutable local variables In-Reply-To: References: <6BD37934-7BC2-4B66-9001-BC40D3A5622D@oracle.com> <03D40340-3C94-40EF-AE18-C586D8EB42F8@gmail.com>

Message-ID: I also want to point out that there seems to be an assumption that the consumer of the lambda has ANY idea that creator of the lambda USED a lambda. Remember that the signature for someone using a lambda will be like public List where(Func f) The fact that you created the Func SAM interface with a lambda is not known to the user of the lambda. Therefore, the idea that some of the creations might be thread safe, isn't very useful. Llewellyn From howard.lovatt at gmail.com Sun Nov 21 23:57:44 2010 From: howard.lovatt at gmail.com (Howard Lovatt) Date: Mon, 22 Nov 2010 18:57:44 +1100 Subject: capturing (or not) mutable local variables In-Reply-To: References:

Message-ID: @Jim & Llewellyn, You raise a number of points: 1. To me the proposed lambdas are consistent with Java, in particular with inner classes (which also only close over finals). The type of the lambda must be a SAM, which means it could be an instance of a class (including inner class) or a lambda, therefore it is important that something written to work with an inner class works with a lambda. 2. As others have pointed out fields can be marked volatile and locals can't, which helps in making a lambda thread safe when writing to a field (and it would be hard to add volatile to a local). 3. In Haskell a closure that interacts with mutable state via a monad can't, in general, be passed to a non-monodic function or vice versa (and this can be a pain and is an active area of research in the Haskell community to remove this restriction). In the proposed lambda model you can pass the lambdas freely to anything, that is a nice feature (you don't need to know the details of the function you call). 4. I agree that Java isn't the ideal parallel language, but lets learn from the mistakes of others and move away from mutable state and set Java up as better than C#, Scala, Lisp, etc. in the parallel domain. Tell people that if they want mutable state they should use AtomicXXX to guarantee thread safety. It is a pity that Java didn't make everything immutable in 95 and that you have to mark stuff as mutable using AtomicXXX (however this is looking back 15 years and it is very easy to spot mistakes in hindsight and the fact is that Java was much better than anything else in 95 and did provide state of the art parallel constructs for 95). Cheers, -- Howard. On 22 November 2010 17:39, Llewellyn Falco wrote: > I also want to point out that there seems to be an?assumption that the > consumer of the lambda has ANY idea that creator of the lambda USED a > lambda. > Remember that the signature for someone using a lambda will be like > > public List where(Func f) > > The fact that you created the Func SAM interface with a lambda is not known > to the user of the lambda. > Therefore, the idea that some of the?creations?might be thread safe, isn't > very useful. > > Llewellyn -- ? -- Howard. From forax at univ-mlv.fr Mon Nov 22 00:17:18 2010 From: forax at univ-mlv.fr (=?ISO-8859-1?Q?R=E9mi_Forax?=) Date: Mon, 22 Nov 2010 09:17:18 +0100 Subject: capturing (or not) mutable local variables In-Reply-To: References:

Message-ID: <4CEA270E.3050007@univ-mlv.fr> Le 22/11/2010 08:57, Howard Lovatt a ?crit : [...] > Tell people that if they want mutable state they should use AtomicXXX > to guarantee thread safety. It is a pity that Java didn't make Using AtomicXXX is not a good idea too. By example, if you want to sum a collection of integers using an atomic variable, this means the volatile variable (inside the atomic class) will be updated for each element. If you use a map/reduce pattern, you only need one synchronization point at the end of the calculation so the map/reduce code outperforms the solution with an atomic variable. > everything immutable in 95 and that you have to mark stuff as mutable > using AtomicXXX (however this is looking back 15 years and it is very > easy to spot mistakes in hindsight and the fact is that Java was much > better than anything else in 95 and did provide state of the art > parallel constructs for 95). > > Cheers, > > -- Howard. R?mi From schmidtfx at googlemail.com Mon Nov 22 01:13:37 2010 From: schmidtfx at googlemail.com (Felix Schmidt) Date: Mon, 22 Nov 2010 10:13:37 +0100 Subject: Compile Message-ID: <-5136260800772443964@unknownmsgid> I think this is a common question: how could I compile a simple example with closures from the lambda project? I have installed the standard compiler of open jdk 7. Thank you felix From maurizio.cimadamore at oracle.com Mon Nov 22 01:30:19 2010 From: maurizio.cimadamore at oracle.com (Maurizio Cimadamore) Date: Mon, 22 Nov 2010 09:30:19 +0000 Subject: Compile In-Reply-To: <-5136260800772443964@unknownmsgid> References: <-5136260800772443964@unknownmsgid> Message-ID: <4CEA382B.8090609@oracle.com> On 22/11/10 09:13, Felix Schmidt wrote: > I think this is a common question: how could I compile a simple > example with closures from the lambda project? > > I have installed the standard compiler of open jdk 7. > > Thank you > felix > > The instruction in this email [1] should help. Feel free to follow up if you have specific questions. [1] - http://mail.openjdk.java.net/pipermail/lambda-dev/2010-August/002179.html Thanks Maurizio From ss at comp.lancs.ac.uk Mon Nov 22 03:25:24 2010 From: ss at comp.lancs.ac.uk (Steven Simpson) Date: Mon, 22 Nov 2010 11:25:24 +0000 Subject: capturing (or not) mutable local variables In-Reply-To: References:

Message-ID: <4CEA5324.2040807@comp.lancs.ac.uk> On 22/11/10 07:57, Howard Lovatt wrote: > 2. As others have pointed out fields can be marked volatile and locals > can't, which helps in making a lambda thread safe when writing to a > field (and it would be hard to add volatile to a local). Would it be so hard? 'volatile' is currently meaningless on a local because it can only be accessed by one thread anyway, so it would be safe to ignore it if not referenced from a closure/inner class (right?). If it was referenced from a closure or inner class, how would it be made mutable? From what I've seen, the only proposals are to box it up in some object, e.g. array of 1, or anonymous class. Within that box, it could be declared volatile. Cheers! -- From alessiostalla at gmail.com Mon Nov 22 04:21:12 2010 From: alessiostalla at gmail.com (Alessio Stalla) Date: Mon, 22 Nov 2010 13:21:12 +0100 Subject: capturing (or not) mutable local variables In-Reply-To: <4CEA270E.3050007@univ-mlv.fr> References:

<4CEA270E.3050007@univ-mlv.fr> Message-ID: On Mon, Nov 22, 2010 at 9:17 AM, R?mi Forax wrote: > ?Le 22/11/2010 08:57, Howard Lovatt a ?crit : > > [...] > >> Tell people that if they want mutable state they should use AtomicXXX >> to guarantee thread safety. It is a pity that Java didn't make > > Using AtomicXXX is not a good idea too. > > By example, if you want to sum a collection of integers using an atomic > variable, this means the volatile variable (inside the atomic class) will be updated > for each element. > > If you use a map/reduce pattern, you only need one synchronization point > at the end of the calculation so the map/reduce code outperforms > the solution with an atomic variable. I'd like to point out that, in my experience with Lisp, the mutability of the closed-over variables is rarely if ever used to update a local to be used further down the same function/method; in other words, int i = 0; foo.foreach(#{ x -> i += x; }); use(i); is not a common code pattern, and it's generally considered bad style. Instead, mutable captured variables are used much more often as a way to give some private state to a closure, effectively turning it into a lightweight object (in the OO sense). So, while I do believe that "true" closures should close over their lexical environment without restrictions, including mutability, I think in Java the issue isn't/won't be that important; if you want private state you can still use a plain old inner class and explicitly capture (copy) the locals you're interested in - after all, closures are going to be "syntax sugar" for SAM types anyway - or you can use final/effectively final (+ maybe AtomicXxx if you need it). From peter.levart at marand.si Mon Nov 22 04:38:27 2010 From: peter.levart at marand.si (Peter Levart) Date: Mon, 22 Nov 2010 13:38:27 +0100 Subject: capturing (or not) mutable local variables In-Reply-To: <4CEA270E.3050007@univ-mlv.fr> References: <4CEA270E.3050007@univ-mlv.fr> Message-ID: <201011221338.27217.peter.levart@marand.si> On 11/22/10, R?mi Forax wrote: > Using AtomicXXX is not a good idea too. > > By example, if you want to sum a collection of integers using an atomic > variable, > this means the volatile variable (inside the atomic class) will be updated > for each element. > > If you use a map/reduce pattern, you only need one synchronization point > at the end of the calculation so the map/reduce code outperforms > the solution with an atomic variable. Or you can use Cliff Click's ConcurrentAutoTable from the http://sourceforge.net/projects/high-scale-lib/ Which is a single counter that automatically spreads the load (using hashing) to multiple independent counters and at each retrieval sums (reduces) them. The style you take to solve a problem depends on the problem. map/reduce is suitable for batch processing while a single counter is more suitable if you want concurrent monitoring of dynamic counter. Regards, Peter From opinali at gmail.com Mon Nov 22 05:35:41 2010 From: opinali at gmail.com (Osvaldo Pinali Doederlein) Date: Mon, 22 Nov 2010 08:35:41 -0500 Subject: capturing (or not) mutable local variables In-Reply-To: References:

<4CEA270E.3050007@univ-mlv.fr> Message-ID: <4CEA71AD.7040404@gmail.com> On 22/11/2010 07:21, Alessio Stalla wrote: > I'd like to point out that, in my experience with Lisp, the mutability > of the closed-over variables is rarely if ever used to update a local > to be used further down the same function/method; in other words, No much Lisp here, but in other langs like Smalltalk, full closures are essential for user/library-defined control structures. That is the big door that's being closed by the current closures spec; but we're already past this discussion, we're not going to have any special syntax for control abstractions. Still, I see this limitation as extra, artificial, unnecessary complexity. On the concurrency argument, I'm with Pavel and others - like it or not the Java language is deeply rooted in the "serial past", mutability is its main paradigm, and it's probably impossible to really fix that (even to reach a decent hybrid-paradigm language) while maintaining backwards compatibility. The current closures spec will make things worse for serial code, while not necessarily making anything better for concurrent code. That could be different if Java8 had a broader revision towards functional behavior - for example, extra syntax like const qualifiers, inference to detect immutability and function pureness etc. (The late JavaFX Script was already making some nice progress in these directions, hopefully this will continue in Visage and other Java-like JVM langs). But such revision is not in the plans for Java8 (although the Checker Framework is). A+ Osvaldo > int i = 0; > foo.foreach(#{ x -> i += x; }); > use(i); > > is not a common code pattern, and it's generally considered bad style. > Instead, mutable captured variables are used much more often as a way > to give some private state to a closure, effectively turning it into a > lightweight object (in the OO sense). So, while I do believe that > "true" closures should close over their lexical environment without > restrictions, including mutability, I think in Java the issue > isn't/won't be that important; if you want private state you can still > use a plain old inner class and explicitly capture (copy) the locals > you're interested in - after all, closures are going to be "syntax > sugar" for SAM types anyway - or you can use final/effectively final > (+ maybe AtomicXxx if you need it). > From per at bothner.com Mon Nov 22 11:18:41 2010 From: per at bothner.com (Per Bothner) Date: Mon, 22 Nov 2010 11:18:41 -0800 Subject: capturing (or not) mutable local variables In-Reply-To: <4CEA5324.2040807@comp.lancs.ac.uk> References:

<4CEA5324.2040807@comp.lancs.ac.uk> Message-ID: <4CEAC211.50205@bothner.com> On 11/22/2010 03:25 AM, Steven Simpson wrote: > If it was referenced from a closure or inner class, how would it be made > mutable? From what I've seen, the only proposals are to box it up in > some object, e.g. array of 1, or anonymous class. Within that box, it > could be declared volatile. If a lambda is implemented as an inner class, then any captured mutable local variables can be implemented as fields of the closure class. I.e. you don't need to allocate any extra objects or create new classes. That means all accesses to the variable (even the ones outside the closure) get translated to access of the field in the closure. But, you say, the variable is defined earlier than the closure, and possibly in an outer scope! The solution: Pre-allocate the closure when entering the scope of the variable. It gets more complicated if the closure captures mutable variable from multiple different scopes, especially if the inner scope is in a loop - in that case, I think you do have to allocate a helper object. I did this for Kawa *before* Java had inner classes - i.e. in the jdk-1.0 dawn of Java. Look for "Old closure implementation" in: http://www.gnu.org/software/kawa/internals/procedures.html -- --Per Bothner per at bothner.com http://per.bothner.com/ From schmidtfx at googlemail.com Mon Nov 22 12:49:44 2010 From: schmidtfx at googlemail.com (Felix Schmidt) Date: Mon, 22 Nov 2010 21:49:44 +0100 Subject: Compile In-Reply-To: <4CEA382B.8090609@oracle.com> References: <-5136260800772443964@unknownmsgid> <4CEA382B.8090609@oracle.com> Message-ID: <4CEAD768.4060608@googlemail.com> Maurizio, thank you for your reply. I compiled the langtools :) But how can I compile my application with the closures feature? langtools/dist/bin/javac CTest.java ? Thank you felix On 11/22/2010 10:30 AM, Maurizio Cimadamore wrote: > On 22/11/10 09:13, Felix Schmidt wrote: >> I think this is a common question: how could I compile a simple >> example with closures from the lambda project? >> >> I have installed the standard compiler of open jdk 7. >> >> Thank you >> felix >> > The instruction in this email [1] should help. Feel free to follow up > if you have specific questions. > > [1] - > http://mail.openjdk.java.net/pipermail/lambda-dev/2010-August/002179.html > > Thanks > Maurizio From forax at univ-mlv.fr Mon Nov 22 13:46:13 2010 From: forax at univ-mlv.fr (=?ISO-8859-1?Q?R=E9mi_Forax?=) Date: Mon, 22 Nov 2010 22:46:13 +0100 Subject: Compile In-Reply-To: <4CEAD768.4060608@googlemail.com> References: <-5136260800772443964@unknownmsgid> <4CEA382B.8090609@oracle.com> <4CEAD768.4060608@googlemail.com> Message-ID: <4CEAE4A5.2010808@univ-mlv.fr> On 11/22/2010 09:49 PM, Felix Schmidt wrote: > Maurizio, > > thank you for your reply. I compiled the langtools :) > > But how can I compile my application with the closures feature? > > langtools/dist/bin/javac CTest.java ? > > Thank you > felix java -cp classes.jar:. com.sun.tools.javac.Main CTest.java R?mi From schmidtfx at googlemail.com Mon Nov 22 14:03:52 2010 From: schmidtfx at googlemail.com (Felix Schmidt) Date: Mon, 22 Nov 2010 23:03:52 +0100 Subject: Compile In-Reply-To: <4CEAE4A5.2010808@univ-mlv.fr> References: <-5136260800772443964@unknownmsgid> <4CEA382B.8090609@oracle.com> <4CEAD768.4060608@googlemail.com> <4CEAE4A5.2010808@univ-mlv.fr> Message-ID: <4CEAE8C8.1040403@googlemail.com> I got the following error CTest.java:4: '{' expected #int() fortyTwo = #()(42); ^ CTest.java:4: expected #int() fortyTwo = #()(42); ^ CTest.java:4: ';' expected #int() fortyTwo = #()(42); ^ CTest.java:4: '{' expected #int() fortyTwo = #()(42); ^ CTest.java:4: ';' expected #int() fortyTwo = #()(42); ^ CTest.java:4: not a statement #int() fortyTwo = #()(42); ^ 6 errors On 11/22/2010 10:46 PM, R?mi Forax wrote: > On 11/22/2010 09:49 PM, Felix Schmidt wrote: > >> Maurizio, >> >> thank you for your reply. I compiled the langtools :) >> >> But how can I compile my application with the closures feature? >> >> langtools/dist/bin/javac CTest.java ? >> >> Thank you >> felix >> > java -cp classes.jar:. com.sun.tools.javac.Main CTest.java > > R?mi > > > From jim at pentastich.org Mon Nov 22 14:25:17 2010 From: jim at pentastich.org (Jim Mayer) Date: Mon, 22 Nov 2010 17:25:17 -0500 Subject: capturing (or not) mutable local variables In-Reply-To: References:

Message-ID: proposal for local variable capture: There are many cases in which local variables are useful in the scope of lambdas, but the basically break down into 2 cases 1: need to read a local variable 2: need to change a local variable The purpose of the this proposal is to suggest a way that is but clear and easy to write, but also follows "the pit of success"; namely, the code does not act in a Surprising manner to the average, uninformed developer. Let's look at the first case: read a local variable. // create filters for the numbers 1 - 10 for(int i = 0; i < 10; i++) { filter(#{f -> f == i}); } Of course this doesn't work in many languages, and won't compile in java. The work around is of course to // create filters for the numbers 1 - 10 for(int i = 0; i < 10; i++) { final int i2 = i; filter(#{f -> f == i2}); } I would like to suggest that java lambda's do this. Namely, if a local variable is *only read* create a second reference to the local variable, and transparently use it instead. Doing this would create the "pit of success" that main other languages have missed. Namely, the code would do as expected, even to the uninformed java programmer. The second case is where you need to track some common variable. int i = 0; createUniqueIds(#{i++}); Here, it's important to be able to have access to i otherwise it would conflict. this also will not compile in java. the workaround of course is int i = 0; final int[] i2 = {i}; createUniqueIds(#{i2[0]++}); This of course becomes an issue inside of a loop int j = 0; for(int i = 0; i < 10; i++) { createUniqueIds(#{ j++}); } does this create 10 different j's. NO, again the "pit of success" would imply no duplicate Id's across all 10 calls to createUniqueIds so the result should be int j = 0; final int j2[] = {j}; for(int i = 0; i < 10; i++) { createUniqueIds(#{ j2[0]++}); } So I am suggesting that the way local variables are handled differs based on whether or not the use of the variable is Read only access or Write access. To be clear, assume the following int j = 0; for(int i = i; i < 10; i++) { createUniqueIds(#{ j+= i}); } would pre-compile to int j = 0; final int j2[] = {j}; for(int i = i; i < 10; i++) { final int i2 = i; createUniqueIds(#{ j2[0]+= i2}); } I believe handling these 2 cases separately would be the least surprising to everyone, avoid many common mistakes made in other languages, and yet keep the syntax clean and simple. Achieving the "pit of success" If there is a situation I haven't thought of, that would create a puzzling pit of failure, please let me know (I'm not sure I've gotten everything...) Thanks, Llewellyn Falco http://bit.ly/lambdas From alex.buckley at oracle.com Tue Nov 23 17:13:41 2010 From: alex.buckley at oracle.com (Alex Buckley) Date: Tue, 23 Nov 2010 17:13:41 -0800 Subject: Defender methods and compatibility In-Reply-To: References: <4CEC169B.3040300@oracle.com> <4CEC5D4E.4060903@oracle.com> Message-ID: <4CEC66C5.3040505@oracle.com> On 11/23/2010 4:55 PM, Neal Gafter wrote: > On Tue, Nov 23, 2010 at 4:33 PM, Alex Buckley You're misreading the asterisks. mod-extn is binary compatible always. > > Not with the currently published specification. Right; Brian's mail stated there are details yet to be published. > In any case, my comment holds true for source compatibility as well: > the language is currently designed so that the change of an > implementation detail in one place does not break source > compatibility elsewhere. The current proposal undermines that design > goal. As does any proposal to have method bodies directly in interfaces. Let's cut to the heart of the issue: > In other words: start with a class whose superinterfaces have no > ambiguity about multiple extension methods; now recompile one of > those superinterfaces so there's an ambiguity; the class cannot now > be recompiled without change. > > Indeed, it should not be possible to "recompile" to introduce an > ambiguity. Only changes to the public API should be capable of > affecting binary or source compatibility. I agree with the first sentence; an ambiguity results in a compile-time error. As for the second sentence, the sense of an extension method is precisely that its default (whether given by name or by value in the interface) becomes part of the method's logical "signature". How could it be otherwise, given declaration-site extensions and separate compilation? Alex From collin.fagan at gmail.com Tue Nov 23 18:22:20 2010 From: collin.fagan at gmail.com (Collin Fagan) Date: Tue, 23 Nov 2010 20:22:20 -0600 Subject: capturing (or not) mutable local variables In-Reply-To: References:

Message-ID: Okay, that's it, look it's over people, you lose, no mutable variable capture for you. It's been stated by the actual people *doing the work* that this is *not in scope*. Dead. Done. Over. As software engineers we all have to manage scope in our projects and sometimes that entails telling people they don't get what they want. This is one of those times. You can be right and still be wrong, this is one of those situations. I'm sorry, Collin On Tue, Nov 23, 2010 at 7:13 PM, Llewellyn Falco wrote: > proposal for local variable capture: > > There are many cases in which local variables are useful in the scope of > lambdas, but the basically break down into 2 cases > > 1: need to read a local variable > 2: need to change a local variable > > The purpose of the this proposal is to suggest a way that is but clear and > easy to write, but also follows "the pit of success"; namely, the code does > not act in a Surprising manner to the average, uninformed developer. > > Let's look at the first case: read a local variable. > > // create filters for the numbers 1 - 10 > for(int i = 0; i < 10; i++) > { > filter(#{f -> f == i}); > } > > Of course this doesn't work in many languages, and won't compile in java. > > The work around is of course to > // create filters for the numbers 1 - 10 > for(int i = 0; i < 10; i++) > { > final int i2 = i; > filter(#{f -> f == i2}); > } > > I would like to suggest that java lambda's do this. Namely, if a local > variable is *only read* create a second reference to the local variable, > and > transparently use it instead. > > Doing this would create the "pit of success" that main other languages have > missed. Namely, the code would do as expected, even to the uninformed java > programmer. > > > The second case is where you need to track some common variable. > > int i = 0; > createUniqueIds(#{i++}); > > Here, it's important to be able to have access to i otherwise it would > conflict. > this also will not compile in java. > > the workaround of course is > int i = 0; > final int[] i2 = {i}; > createUniqueIds(#{i2[0]++}); > > > This of course becomes an issue inside of a loop > int j = 0; > for(int i = 0; i < 10; i++) > { > createUniqueIds(#{ j++}); > } > does this create 10 different j's. NO, again the "pit of success" would > imply no duplicate Id's across all 10 calls to createUniqueIds > > so the result should be > int j = 0; > final int j2[] = {j}; > for(int i = 0; i < 10; i++) > { > createUniqueIds(#{ j2[0]++}); > } > > So I am suggesting that the way local variables are handled differs based > on > whether or not the use of the variable is Read only access or Write access. > > To be clear, assume the following > int j = 0; > for(int i = i; i < 10; i++) > { > createUniqueIds(#{ j+= i}); > } > > would pre-compile to > > int j = 0; > final int j2[] = {j}; > for(int i = i; i < 10; i++) > { > final int i2 = i; > createUniqueIds(#{ j2[0]+= i2}); > } > > > I believe handling these 2 cases separately would be the least surprising > to > everyone, avoid many common mistakes made in other languages, and yet keep > the syntax clean and simple. Achieving the "pit of success" > > If there is a situation I haven't thought of, that would create a puzzling > pit of failure, please let me know (I'm not sure I've gotten everything...) > > > > Thanks, > > Llewellyn Falco > http://bit.ly/lambdas > > From neal at gafter.com Tue Nov 23 19:44:35 2010 From: neal at gafter.com (Neal Gafter) Date: Tue, 23 Nov 2010 19:44:35 -0800 Subject: Defender methods and compatibility In-Reply-To: <4CEC66C5.3040505@oracle.com> References: <4CEC169B.3040300@oracle.com> <4CEC5D4E.4060903@oracle.com> <4CEC66C5.3040505@oracle.com> Message-ID: On Tue, Nov 23, 2010 at 5:13 PM, Alex Buckley wrote: > On 11/23/2010 4:55 PM, Neal Gafter wrote:In any case, my comment holds true > for source compatibility as well: > >> the language is currently designed so that the change of an >> implementation detail in one place does not break source >> compatibility elsewhere. The current proposal undermines that design >> goal. >> > > As does any proposal to have method bodies directly in interfaces. > I don't believe that is true. > Let's cut to the heart of the issue: > > > In other words: start with a class whose superinterfaces have no >> ambiguity about multiple extension methods; now recompile one of >> those superinterfaces so there's an ambiguity; the class cannot now >> be recompiled without change. >> >> Indeed, it should not be possible to "recompile" to introduce an >> ambiguity. Only changes to the public API should be capable of affecting >> binary or source compatibility. >> > > I agree with the first sentence; an ambiguity results in a compile-time > error. As for the second sentence, the sense of an extension method is > precisely that its default (whether given by name or by value in the > interface) becomes part of the method's logical "signature". How could it be > otherwise, given declaration-site extensions and separate compilation? It could easily be otherwise: place the extension method bodies in the interfaces and treat every method body as distinct from every other method body. From neal at gafter.com Tue Nov 23 20:13:31 2010 From: neal at gafter.com (Neal Gafter) Date: Tue, 23 Nov 2010 20:13:31 -0800 Subject: capturing (or not) mutable local variables In-Reply-To: <4CEBD20B.50902@oracle.com> References: <6BD37934-7BC2-4B66-9001-BC40D3A5622D@oracle.com> <82sjysnnkk.fsf@mid.bfk.de> <4CEBD20B.50902@oracle.com> Message-ID: On Tue, Nov 23, 2010 at 6:39 AM, Brian Goetz wrote: > >> In a parellel world, this idiom is irretrievably broken. > > > > I fear that such an extreme focus on concurrent programming is rather > > unhealthy for a general-purpose language. > > How can you possibly (with a straight face) describe the choice to *not* > add a > controversial feature as being an "unhealthily extreme focus"? Seems to me > the rhetoric in this discussion has left the realm of reality. > "capture of mutable variables" is not a feature. Capture of variables (i.e. lexical scoping) is a feature. Mutable variables is a feature. The intersection of the two is just a confluence of two orthogonal features. Independently designing the intersection of two otherwise orthogonal language features on the basis of use cases is a classic sign of amateur language design. Failure of imagination - that is, failure to make language constructs orthogonal because you aren't quite sure how people will need to use them together - is one of the most common and severe errors you can make in programming language design. Cheers, Neal From alex.buckley at oracle.com Wed Nov 24 12:00:51 2010 From: alex.buckley at oracle.com (Alex Buckley) Date: Wed, 24 Nov 2010 12:00:51 -0800 Subject: Defender methods and compatibility In-Reply-To: References: <4CEC169B.3040300@oracle.com> <4CEC5D4E.4060903@oracle.com> <4CEC66C5.3040505@oracle.com> Message-ID: <4CED6EF3.5070408@oracle.com> On 11/23/2010 7:44 PM, Neal Gafter wrote: > I agree with the first sentence; an ambiguity results in a > compile-time error. As for the second sentence, the sense of an > extension method is precisely that its default (whether given by > name or by value in the interface) becomes part of the method's > logical "signature". How could it be otherwise, given > declaration-site extensions and separate compilation? > > It could easily be otherwise: place the extension method bodies in the > interfaces and treat every method body as distinct from every other > method body. I don't understand how this preserves source compatibility. Start with the following types that compile and run: interface I {} interface J {} class C implements I,J {} Now modify and recompile I only: interface I { extension void m() { BODY_I } } It turns out to be source-compatible because C would still compile. Now modify and recompile J only: interface J { extension void m() { BODY_J } } Now C would not compile. We all agree C must be changed so it declares m and (probably) delegates to I.m or J.m. But the add-extn to J remains a source-incompatible change. Alex P.S. add-extn and mod-extn on an interface are SC if _all_ classes that implement the interface define (by declaration or by class inheritance) the method that gained/changed extension. But there's no way to check it, of course. From neal at gafter.com Wed Nov 24 14:38:25 2010 From: neal at gafter.com (Neal Gafter) Date: Wed, 24 Nov 2010 14:38:25 -0800 Subject: Defender methods and compatibility In-Reply-To: <4CED6EF3.5070408@oracle.com> References: <4CEC169B.3040300@oracle.com> <4CEC5D4E.4060903@oracle.com> <4CEC66C5.3040505@oracle.com> <4CED6EF3.5070408@oracle.com> Message-ID: On Wed, Nov 24, 2010 at 12:00 PM, Alex Buckley wrote: > Now C would not compile. We all agree C must be changed so it declares m > and (probably) delegates to I.m or J.m. But the add-extn to J remains a > source-incompatible change. > Right. We agree that add-extn is a source-incompatible change. My point was that mod-extn should be a source-compatible change, but you've specified it to be a source-incompatible change. Cheers, Neal From alex.buckley at oracle.com Wed Nov 24 15:29:57 2010 From: alex.buckley at oracle.com (Alex Buckley) Date: Wed, 24 Nov 2010 15:29:57 -0800 Subject: Defender methods and compatibility In-Reply-To: References: <4CEC169B.3040300@oracle.com> <4CEC5D4E.4060903@oracle.com> <4CEC66C5.3040505@oracle.com> <4CED6EF3.5070408@oracle.com> Message-ID: <4CED9FF5.2080308@oracle.com> On 11/24/2010 2:38 PM, Neal Gafter wrote: > Now C would not compile. We all agree C must be changed so it > declares m and (probably) delegates to I.m or J.m. But the add-extn > to J remains a source-incompatible change. > > Right. We agree that add-extn is a source-incompatible change. My > point was that mod-extn should be a source-compatible change, but you've > specified it to be a source-incompatible change. There has been some confusion - you started by quoting Brian's comment on add-meth, which should be compared to add-extn, but you raised a point about mod-extn and its relationship to mod-meth. The initial summary mail should have compared add-extn specifically to add-meth, and mod-extn specifically to mod-meth. When I wrote earlier about "changing an extension method's default", I was mainly thinking of adding a default, which we all agree is not guaranteed SC. Specifically, I was thinking of changing an interface method into an extension method by adding a default. That's one case of add-extn. (The other is adding an extension method signature from scratch.) Anyway, the good news is that mod-extn _is_ SC. That's true whether a default is specified in the interface via a method body or a method name. If specified via a method name, then independent mod-extn actions (or add-extn - there it is again!) on different interfaces _may_ still be SC for an implementation class. Some people see this as an optimization we can leave out; that may be so, but it is not a prima facie reason for preferring method bodies. Alex From alex.buckley at oracle.com Wed Nov 24 15:48:07 2010 From: alex.buckley at oracle.com (Alex Buckley) Date: Wed, 24 Nov 2010 15:48:07 -0800 Subject: Defender methods and compatibility In-Reply-To: <4CED9FF5.2080308@oracle.com> References: <4CEC169B.3040300@oracle.com> <4CEC5D4E.4060903@oracle.com> <4CEC66C5.3040505@oracle.com> <4CED6EF3.5070408@oracle.com> <4CED9FF5.2080308@oracle.com> Message-ID: <4CEDA437.1070904@oracle.com> On 11/24/2010 3:29 PM, Alex Buckley wrote: > Anyway, the good news is that mod-extn _is_ SC. That's true whether a > default is specified in the interface via a method body or a method name. Should have said: mod-extn _may be_ SC. When an implementation class doesn't override the extension, it's possible for previously-identical extension methods to diverge (i.e. one changes its default) and break SC. Alex From neal at gafter.com Wed Nov 24 17:22:41 2010 From: neal at gafter.com (Neal Gafter) Date: Wed, 24 Nov 2010 17:22:41 -0800 Subject: Defender methods and compatibility In-Reply-To: <4CEDA437.1070904@oracle.com> References: <4CEC169B.3040300@oracle.com> <4CEC5D4E.4060903@oracle.com> <4CEC66C5.3040505@oracle.com> <4CED6EF3.5070408@oracle.com> <4CED9FF5.2080308@oracle.com> <4CEDA437.1070904@oracle.com> Message-ID: On Wed, Nov 24, 2010 at 3:48 PM, Alex Buckley wrote: > On 11/24/2010 3:29 PM, Alex Buckley wrote: > >> Anyway, the good news is that mod-extn _is_ SC. That's true whether a >> default is specified in the interface via a method body or a method name. >> > > Should have said: mod-extn _may be_ SC. When an implementation class > doesn't override the extension, it's possible for previously-identical > extension methods to diverge (i.e. one changes its default) and break SC. > Understood. That is the particular part of the specification that I take issue with. mod-extn (changing the code of a default method) should (in my opinion) always be source compatible, as the code of a method is a prime example of an implementation detail. From dl at cs.oswego.edu Fri Nov 26 02:58:23 2010 From: dl at cs.oswego.edu (Doug Lea) Date: Fri, 26 Nov 2010 05:58:23 -0500 Subject: capturing (or not) mutable local variables In-Reply-To: <4CE97E88.5010803@bothner.com> References: <6BD37934-7BC2-4B66-9001-BC40D3A5622D@oracle.com> <03D40340-3C94-40EF-AE18-C586D8EB42F8@gmail.com>

<4CE97E88.5010803@bothner.com> Message-ID: <4CEF92CF.8090600@cs.oswego.edu> Finally reading this discussion... On 11/21/10 15:18, Per Bothner wrote: > On 11/20/2010 12:44 PM, Brian Goetz wrote: >>> I think I'm hearing that mutating a local variable from a closure is just a >>> bad idea in general right? >> >> For a closure that might conceivably be executed in another thread, this is correct. > > Agreed - but how is that different from mutating a field or array element > from a closure that might conceivably be executed in another thread? > You're just forcing the user to do by hand to work around a restriction. I think the main goal here is to preserve the "shallow data race freedom of locals" property of Java: Writes to any variable syntactically declared as a local (or argument) can never be subject to a data race. In conjunction with the other main property of Java locals that they are definitely assigned, this can be used to provide a simple partial correctness guarantee about the integrity of reads. Which can then be used as a base step for more thorough reasoning about safety. The shallowness arises in case a local is a reference to an object and is dereferenced (accessed), in which case programmers have more work to do proving race-freedom of the accesses. The limitation to shallow safety seems to be pragmatic decision: Deeper guarantees would require non-modular checks (as in, trying to verify that data-race freedom held across calls to possible virtual methods) as well as concurrency awareness by front-end compilers, plus some means of programmers annotating the shapes of data structures and partitioning, as in those in DPJ (http://dpj.cs.uiuc.edu/DPJ/Home.html) that can help check that all in-place updates of elements of arrays or collections are fully partitioned across threads. (Aside: Many efficient multicore algorithms entail in-place updates -- the main advantages of their design (versus other forms of multiprocessors) stem from their typically faster access to shared memory.) The workarounds like using 1-slot arrays add a level of indirection to subvert the shallow-only provision. Which seems tolerable. It would be even better to come up with capture rules that guaranteed this property while still also allowing more flexible usages. The suggestion to extend "could be final" to something akin to having a unique SSA form is one step. Going even a bit further than this would be hard because front-end compilers are not thread-aware. So among other things they cannot detect the canonically bad idea of using accumulator variables in parallel loops. However, perhaps the folks arguing for more flexible usage could discover more powerful yet practical techniques that would still preserve the shallow data-race freedom property in more cases, and thus allow some of their target usages. -Doug From dl at cs.oswego.edu Fri Nov 26 04:45:57 2010 From: dl at cs.oswego.edu (Doug Lea) Date: Fri, 26 Nov 2010 07:45:57 -0500 Subject: capturing (or not) mutable local variables In-Reply-To: <4CEF92CF.8090600@cs.oswego.edu> References: <6BD37934-7BC2-4B66-9001-BC40D3A5622D@oracle.com> <03D40340-3C94-40EF-AE18-C586D8EB42F8@gmail.com>

<4CE97E88.5010803@bothner.com> <4CEF92CF.8090600@cs.oswego.edu> Message-ID: <4CEFAC05.9050402@cs.oswego.edu> On 11/26/10 05:58, Doug Lea wrote: > However, perhaps the folks arguing for more flexible > usage could discover more powerful yet practical techniques > that would still preserve the shallow data-race freedom > property in more cases, and thus allow some of their target usages. > One way is to simply preserve this property, period: Suppose the translation mechanics for mutable variables were in terms of ThreadLocal boxed objects, rather than those possibly shared across threads. In other words class C { void f() { int i = 0; ... #{ ... ++i; ... } } } becomes translated to something like: class C { void f() { ThreadLocal ti = ...; ... #{ ... ti.increment(); ... } } } It would be challenging to come up good rules to maximize chances that this is done only when people really meant to do it. The various "mistake" cases (like for-loop indices) seem to outnumber the intentional ones. And further to teach front-end compilers, AOT bytecode optimizers, and/or JITs to understand these constructions well enough to avoid all the overhead when it is statically determinable that lambda can be inlined/macro-expanded in a way that allows variables to just remain as locals. It would also provide some fodder for pushing ThreadLocal support deeper into the language/compilers/runtimes to help improve performance. And the net impact on constructions like accumulators in parallel loops would be that you'd always get a safe but wrong answer. All in all, while this may be the best possible compromise (providing both mutability and guaranteed locality), I don't see a good reason to do it. However, it does suggest that if there were better syntax support for safe workarounds, people might be happier to use them. Maybe @ThreadLocal could trigger this transformation? class C { void f() { @ThreadLocal int i = 0; ... #{ ... ++i; ... } } } -Doug From markmahieu at gmail.com Fri Nov 26 09:13:44 2010 From: markmahieu at gmail.com (Mark Mahieu) Date: Fri, 26 Nov 2010 17:13:44 +0000 Subject: capturing (or not) mutable local variables In-Reply-To: <4CEF92CF.8090600@cs.oswego.edu> References: <6BD37934-7BC2-4B66-9001-BC40D3A5622D@oracle.com> <03D40340-3C94-40EF-AE18-C586D8EB42F8@gmail.com>

<4CE97E88.5010803@bothner.com> <4CEF92CF.8090600@cs.oswego.edu> Message-ID: On 26 Nov 2010, at 10:58, Doug Lea wrote: > > The workarounds like using 1-slot arrays add a level of > indirection to subvert the shallow-only provision. > Which seems tolerable. It's tolerable if the workaround is employed with a good understanding of its limitations, but it has become a common approach to simple inconveniences with final variables used in relatively innocuous contexts. My fear is that it's a workaround which will find itself used in far less tolerant situations through the simple virtues of being well known and long established. So the following kind of approach... > However, it does suggest > that if there were better syntax support for safe > workarounds, people might be happier to use them. > Maybe @ThreadLocal could trigger this transformation? > > class C { > void f() { > @ThreadLocal int i = 0; > ... #{ ... ++i; ... } > } > } > > -Doug ... is one I see as a (conceptually) neat alternative to the two extremes this discussion has polarised around so far. Lambda does present a very rare opportunity to offer a different approach and have it noticed by a wide audience. Cheers, Mark From jim at pentastich.org Fri Nov 26 18:39:37 2010 From: jim at pentastich.org (Jim Mayer) Date: Fri, 26 Nov 2010 21:39:37 -0500 Subject: pure functions and the parallel evaluation of reduce-like operations Message-ID: Hi all, In the recent debate on capturing mutable local variables the "reduce" function was used as an example application of functional programming, sometimes with an implication that the form could be evaluated in parallel. It seems to me, though, that a parallel evaluation scheme is only applicable if the operator is associative (or, better, commutative and associative). There's no question that it is easier and more efficient to do parallel operations with "pure" functions, but a functional style is not sufficient to enable parallelism. In fact, in Java, even addition and multiplication of most numeric types are not associative because of overflow and loss of precision. For "well behaved" data sets things usually work well enough, but it really matters in some calculations. My question is, as we consider use cases for lambda expressions and whether to put effort into (as Brian Goetz said) "supporting map/reduce-y idioms in the libraries", do we need to consider how functions (like 'reduce') declare the constraints? How we would use functional programming idioms to enable parallelism in Java? Do we define annotations to declare intent (e.g, @Associative)? Do we have multiple reduce operations (e.g., leftReduce, rightReduce, associativeReduce, unorderedReduce)? Do we pass in the function's constraints as a parameter (e.g., List.reduce(Operation.ASSOCIATIVE, #{s v -> s + v}))? Are the mechanisms available sufficient to enable wide-spread, safe, use of the map/reduce-y idioms? --- Jim From scolebourne at joda.org Sat Nov 27 03:54:08 2010 From: scolebourne at joda.org (Stephen Colebourne) Date: Sat, 27 Nov 2010 11:54:08 +0000 Subject: capturing (or not) mutable local variables In-Reply-To: References: <6BD37934-7BC2-4B66-9001-BC40D3A5622D@oracle.com> <03D40340-3C94-40EF-AE18-C586D8EB42F8@gmail.com>

<4CE97E88.5010803@bothner.com> <4CEF92CF.8090600@cs.oswego.edu> Message-ID: On 26 November 2010 17:13, Mark Mahieu wrote: > On 26 Nov 2010, at 10:58, Doug Lea wrote: >> The workarounds like using 1-slot arrays add a level of >> indirection to subvert the shallow-only provision. >> Which seems tolerable. > > It's tolerable if the workaround is employed with a good understanding of its limitations, but it has become a common approach to simple inconveniences with final variables used in relatively innocuous contexts. My fear is that it's a workaround which will find itself used in far less tolerant situations through the simple virtues of being well known and long established. Agreed. And there is a connection to generics wildcards too. In both cases, restrictions were/are added to the language for good reasons - to keep the language safe. (For generics I'm thinking of casting a List to a List, and many other cases as in the huge FAQ). What happens is: - the developer tries to perform the "bad" thing - the compiler stops them - the developer curses the compiler - the developer finds the quickest hack In my experience, most don't worry about WHY the thing was "bad" they just learn the hacks (and aren't worried if the hack is safe). For generics, that might be using a raw type. For no access to mutable variables, its the length one array. While I can entirely see why the mutable variable restriction is desirable from a correctness POV, I'm far from convinced that it is beneficial from a practical POV, and may actually be harmful (because only a fraction of developers understand the restriction). As such, allowing mutable access may well be better overall. Although, some kind of middle ground like Doug suggests would be good. (Note that it could be argued based on the above, that mutable access via arrays should be blocked to force an object based workaround like AtomicRefs) Stephen From thomas.andreas.jung at googlemail.com Sat Nov 27 07:45:50 2010 From: thomas.andreas.jung at googlemail.com (Thomas Jung) Date: Sat, 27 Nov 2010 16:45:50 +0100 Subject: Artifact URL changes due to move to Kenai Message-ID: Hi, the URLs of downloaded source artifacts seems to changed due to the move to Kenai. Building with "make all" works with the URLs below. Thomas diff -r 3f61197b38c8 jaxws.properties --- a/jaxws.properties Tue Oct 19 11:07:54 2010 +0100 +++ b/jaxws.properties Sat Nov 27 16:38:07 2010 +0100 @@ -28,12 +28,12 @@ jaxws_src.bundle.name= jdk7-jaxws2_2-2010_08_19.zip jaxws_src.bundle.md5.checksum=8775ccefd3b4fa2dde5155ec4b7e4ceb jaxws_src.master.bundle.dir=${drops.master.copy.base} -jaxws_src.master.bundle.url.base=https://jax-ws.dev.java.net/files/documents/4202/152532 +jaxws_src.master.bundle.url.base=http://java.net/downloads/jax-ws/JDK7/ jaf_src.bundle.name=jdk7-jaf-2010_08_19.zip jaf_src.bundle.md5.checksum=18d15dfd71117daadb332af003d08212 jaf_src.master.bundle.dir=${drops.master.copy.base} -jaf_src.master.bundle.url.base=https://jax-ws.dev.java.net/files/documents/4202/152336 +jaf_src.master.bundle.url.base=http://java.net/downloads/jax-ws/JDK7/ #jaxws_tests.bundle.name=jdk7-jaxws-tests-2009_08_28.zip #jaxws_tests.master.bundle.dir=${drops.master.copy.base} diff -r bfeed086a9c7 jaxp.properties --- a/jaxp.properties Tue Oct 19 11:07:46 2010 +0100 +++ b/jaxp.properties Sat Nov 27 16:39:41 2010 +0100 @@ -28,7 +28,7 @@ jaxp_src.bundle.name=jaxp-1_4_4.zip jaxp_src.bundle.md5.checksum=2c40a758392c4abf2d59f355240df46a jaxp_src.master.bundle.dir=${drops.master.copy.base} -jaxp_src.master.bundle.url.base=https://jaxp.dev.java.net/files/documents/913/152561 +jaxp_src.master.bundle.url.base=http://java.net/downloads/jaxp/jdk7 #jaxp_tests.bundle.name=jaxp-unittests-1_4_4.zip #jaxp_tests.bundle.md5.checksum=51845e38b02920cf5374d0331ab3a4ee From reinier at zwitserloot.com Sat Nov 27 08:44:24 2010 From: reinier at zwitserloot.com (Reinier Zwitserloot) Date: Sat, 27 Nov 2010 17:44:24 +0100 Subject: capturing (or not) mutable local variables In-Reply-To: References: <6BD37934-7BC2-4B66-9001-BC40D3A5622D@oracle.com> <03D40340-3C94-40EF-AE18-C586D8EB42F8@gmail.com>

<4CE97E88.5010803@bothner.com> <4CEF92CF.8090600@cs.oswego.edu>

<4CE97E88.5010803@bothner.com> <4CEF92CF.8090600@cs.oswego.edu>

Message-ID: <4CF14FA0.8040302@cs.oswego.edu> On 11/27/10 06:54, Stephen Colebourne wrote: > While I can entirely see why the mutable variable restriction is > desirable from a correctness POV, I'm far from convinced that it is > beneficial from a practical POV. When those of us writing underlying core library support cannot rely on locality properties, it means that all the application code that uses it is also suspect, which might be a practical problem :-) > > (Note that it could be argued based on the above, that mutable access > via arrays should be blocked to force an object based workaround like > AtomicRefs) Mostly as an aside: While the borderlines are fuzzy, concurrent programming is mainly about synchronization, and parallel programming is mainly about isolation. AtomicRefs and the like should be rarely applicable in constructions oriented toward parallel execution. As I mentioned, annotations causing translation into ThreadLocals seems plausible for scalars, but for arrays you'd probably prefer some sort of wrapper that causes the program to blow up if any element is ever accessed by any thread other than its creator. -Doug From serge.boulay at gmail.com Sat Nov 27 11:26:12 2010 From: serge.boulay at gmail.com (Serge Boulay) Date: Sat, 27 Nov 2010 14:26:12 -0500 Subject: capturing (or not) mutable local variables In-Reply-To: References: <6BD37934-7BC2-4B66-9001-BC40D3A5622D@oracle.com> <03D40340-3C94-40EF-AE18-C586D8EB42F8@gmail.com>

<4CE97E88.5010803@bothner.com> <4CEF92CF.8090600@cs.oswego.edu>

Message-ID: At some point I'd like to see a still visible but less convoluted way of doing this, for example by adding a keyword that can be applied to a local" What about the @Shared annotation in "closures for Java" specification? A mandatory warning is required if some local variable or parameter from an enclosing scope is used, unless one of the following conditions holds: 1. The variable is declared final, or 2. The variable is not the target of any assignment (i.e., it is only initialized in its declaration), or 3. The variable is annotated @Shared, or 4. some enclosing construct is annotated @SuppressWarnings("shared"). RE: Reiner Zwitserloot The difference between warning and error is virtually non-existent. As under the hood there will be an enormous difference, keyword is a better fit. Also, with the annotation you can't mark the local 'volatile'. On Nov 27, 2010 6:23 PM, "Serge Boulay" wrote: > "At some point I'd like to see a still visible but less convoluted way of > doing this, for example by adding a keyword that can be applied to a local" > > What about the @Shared annotation in "closures for Java" specification? > > > A mandatory warning is required if some local variable or parameter from an > enclosing scope is used, unless one of the following conditions holds: > > 1. The variable is declared final, or > 2. The variable is not the target of any assignment (i.e., it is only > initialized in its declaration), or > 3. The variable is annotated @Shared, or > 4. some enclosing construct is annotated @SuppressWarnings("shared"). - Show quoted text - On Sat, Nov 27, 2010 at 11:44 AM, Reinier Zwitserloot < reinier at zwitserloot.com> wrote: > That sounds a bit defeatist! > > Just because a sizable chunk of programmers casually write code that > includes an active bug or is extremely fragile doesn't mean we should just > throw in the towel. > > Work should perhaps be done on the warning / error messages. For example, > include a URL to a page that serves as a tutorial for the more complex java > rules, such as why you can't assign a List to a List. But, > that's not a job for lambda-dev, and doesn't have to be done right now. The > exact wording of error messages aren't part of the java spec, fortunately. > > I think PHP shows what happens when one always takes the most direct > pragmatic solution to any problem. I assume I don't have to elaborate much > on why PHP indicates this is not a particularly wise move. > > There's also a decent chance someone with absolutely no understanding as to > why mutables in (non-thread-local) closures are a bad idea will > nevertheless > be coached into doing the right thing. The error message will doubtlessly > reference the notion that _final_ locals _can_ be accessed in the closure. > Making a copy final is a lot simpler than the array/AtomicX workaround, and > so happens to be the superior solution too. > > A second benefit is that, while the one that wrote the code may well be > clueless about what multicore implies, someone else reviewing this code > will > have an easier time identifying what's going on. This is quite java-esque - > a field ref can't possibly invoke code. A local variable can't possibly be > involved in a race condition. It's the basis upon which more complex > reasoning can be performed. An 1-size array/AtomicX is an easily visible > red > flag. > > At some point I'd like to see a still visible but less convoluted way of > doing this, for example by adding a keyword that can be applied to a local, > but there's no particular need to do this for the first release of closures > in java. An advantage of doing it later is that there will be far more > experience to base further language changes on. > > --Reinier Zwitserloot > > > > On Sat, Nov 27, 2010 at 12:54 PM, Stephen Colebourne > wrote: > > > On 26 November 2010 17:13, Mark Mahieu wrote: > > > On 26 Nov 2010, at 10:58, Doug Lea wrote: > > >> The workarounds like using 1-slot arrays add a level of > > >> indirection to subvert the shallow-only provision. > > >> Which seems tolerable. > > > > > > It's tolerable if the workaround is employed with a good understanding > of > > its limitations, but it has become a common approach to simple > > inconveniences with final variables used in relatively innocuous > contexts. > > My fear is that it's a workaround which will find itself used in far > less > > tolerant situations through the simple virtues of being well known and > long > > established. > > > > Agreed. And there is a connection to generics wildcards too. In both > > cases, restrictions were/are added to the language for good reasons - > > to keep the language safe. (For generics I'm thinking of casting a > > List to a List, and many other cases as in the huge > > FAQ). What happens is: > > - the developer tries to perform the "bad" thing > > - the compiler stops them > > - the developer curses the compiler > > - the developer finds the quickest hack > > In my experience, most don't worry about WHY the thing was "bad" they > > just learn the hacks (and aren't worried if the hack is safe). For > > generics, that might be using a raw type. For no access to mutable > > variables, its the length one array. > > > > While I can entirely see why the mutable variable restriction is > > desirable from a correctness POV, I'm far from convinced that it is > > beneficial from a practical POV, and may actually be harmful (because > > only a fraction of developers understand the restriction). As such, > > allowing mutable access may well be better overall. Although, some > > kind of middle ground like Doug suggests would be good. > > > > (Note that it could be argued based on the above, that mutable access > > via arrays should be blocked to force an object based workaround like > > AtomicRefs) > > > > Stephen > > > > > > From jim at pentastich.org Sat Nov 27 14:10:17 2010 From: jim at pentastich.org (Jim Mayer) Date: Sat, 27 Nov 2010 17:10:17 -0500 Subject: pure functions and the parallel evaluation of reduce-like operations In-Reply-To: References: <4CF13F12.8090400@cs.oswego.edu> Message-ID: Doug, Well, I suspect that 99 percent of programmers will neither know not care how their floating point calculations are ordered. I'm also quite sure that anyone trying to invert an ill conditioned matrix will care a lot. But that's just for number crunching, and addition and multiplication are wonderfully tractable operations. What about functions that are not associative at all? Then it is not a matter of determinism but of correctness. Considered List.reduce (#{s, v -> s * s + v}). This is a pure function but is not associative. (1^ 2 +2)^ 2 +3 is 12, but 1^ 2 +(2^ 2 + 3) is 8. I also suspect that some operations will be slowed down by parallel execution because they are gated by the memory subsystem and not the CPU. My point is that we won't be able to just turn existing functions just into parallel versions of themselves, and that for operations like reduce/fold/etc., which are incredibly useful even in single threaded environments, we may well end up with multiple versions with differing degrees of allowed parallelism. -- Jim On Nov 27, 2010 12:28 PM, "Doug Lea"