From vladimir.kirichenko at gmail.com  Fri Nov 14 17:25:10 2008
From: vladimir.kirichenko at gmail.com (Vladimir Kirichenko)
Date: Sat, 15 Nov 2008 03:25:10 +0200
Subject: why unrestricted closures are always void?
Message-ID: <491E24F6.2000005@gmail.com>

This makes impossible to implement functional-style control abstractions 
like generators:

Iterable<Integer> squares = yield(int i : someIterable()) { i * i }

int[] sqa =  for collect(int i : array) { i * i }

It looks very inconsistent to be able to use control abstraction with 
"void" code and to be unable to use it with "code with result".  It 
places major limitation to usage of control abstractions. Is this really 
necessary?

-- 
Best Regards,
Vladimir Kirichenko

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 254 bytes
Desc: OpenPGP digital signature
Url : http://mail.openjdk.java.net/pipermail/closures-dev/attachments/20081115/6b3b1abe/attachment.bin 

From neal at gafter.com  Sat Nov 15 13:26:08 2008
From: neal at gafter.com (Neal Gafter)
Date: Sat, 15 Nov 2008 14:26:08 -0700
Subject: why unrestricted closures are always void?
In-Reply-To: <491E24F6.2000005@gmail.com>
References: <491E24F6.2000005@gmail.com>
Message-ID: <15e8b9d20811151326t4627999fodce5770df9941993@mail.gmail.com>

Unrestricted closures are not so constrained: they can have any return type.

On Fri, Nov 14, 2008 at 6:25 PM, Vladimir Kirichenko <
vladimir.kirichenko at gmail.com> wrote:

> This makes impossible to implement functional-style control abstractions
> like generators:
>
> Iterable<Integer> squares = yield(int i : someIterable()) { i * i }
>
> int[] sqa =  for collect(int i : array) { i * i }
>
> It looks very inconsistent to be able to use control abstraction with
> "void" code and to be unable to use it with "code with result".  It places
> major limitation to usage of control abstractions. Is this really necessary?
>
> --
> Best Regards,
> Vladimir Kirichenko
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/closures-dev/attachments/20081115/cd8d719b/attachment.html 

From tronicek at fel.cvut.cz  Sat Nov 15 15:07:45 2008
From: tronicek at fel.cvut.cz (Zdenek Tronicek)
Date: Sun, 16 Nov 2008 00:07:45 +0100
Subject: why unrestricted closures are always void?
In-Reply-To: <15e8b9d20811151326t4627999fodce5770df9941993@mail.gmail.com>
References: <491E24F6.2000005@gmail.com>
	<15e8b9d20811151326t4627999fodce5770df9941993@mail.gmail.com>
Message-ID: <20081116000745.20566v26zzgygcox@wimap.feld.cvut.cz>

Neal, he means control invocation syntax. For example:

   public static int control(int x, { int ==> void } block) {
     block.invoke(x);
     return 42;
   }

   public static void main(String[] args) {
     // this is ok
     int answer = control(1, { int i ==> System.out.println(i); });
     // but this does not compile
     answer = control(int i : 2) {
       System.out.println(i);
     }
   }

Z.
-- 
Zdenek Tronicek
Department of Computer Science and Engineering
Prague                   tel: +420 2 2435 7410
http://cs.felk.cvut.cz/~tronicek


Quoting Neal Gafter <neal at gafter.com>:

> Unrestricted closures are not so constrained: they can have any return type.
>
> On Fri, Nov 14, 2008 at 6:25 PM, Vladimir Kirichenko <
> vladimir.kirichenko at gmail.com> wrote:
>
>> This makes impossible to implement functional-style control abstractions
>> like generators:
>>
>> Iterable<Integer> squares = yield(int i : someIterable()) { i * i }
>>
>> int[] sqa =  for collect(int i : array) { i * i }
>>
>> It looks very inconsistent to be able to use control abstraction with
>> "void" code and to be unable to use it with "code with result".  It places
>> major limitation to usage of control abstractions. Is this really necessary?
>>
>> --
>> Best Regards,
>> Vladimir Kirichenko
>>
>>
>




From vladimir.kirichenko at gmail.com  Sat Nov 15 17:21:45 2008
From: vladimir.kirichenko at gmail.com (Vladimir Kirichenko)
Date: Sun, 16 Nov 2008 03:21:45 +0200
Subject: why unrestricted closures are always void?
In-Reply-To: <20081116000745.20566v26zzgygcox@wimap.feld.cvut.cz>
References: <491E24F6.2000005@gmail.com>	<15e8b9d20811151326t4627999fodce5770df9941993@mail.gmail.com>
	<20081116000745.20566v26zzgygcox@wimap.feld.cvut.cz>
Message-ID: <491F75A9.6020809@gmail.com>

Zdenek Tronicek wrote:
> Neal, he means control invocation syntax. For example:

Yeah. In ruby which have similar facility I can write something like:

def _for(x, &f) yield x end

and than invoke it:

res = _for(5) do |x| x*2 end

It will be good thing to have ability to write control abstractions 
those could return results, i.e.:

Iterable<Integer> squares = yield(int i : someIterable()) { i * i }

will construct generator - similar to one that exists in bunch of other 
languages.

And it's just looks very strange:

//compiles fine
public static int f(int i, {int ==> int} block) { return block.invoke(i); }

//does fine
System.out.println(f(3, {int i ==> i*i }));

but:

f(int i : 3) { return i * i; }

reports error:

cannot return a value from method whose result type is void
            f(int i : 3) { return i * i; }

It's not void neither in closure nor in method.

-- 
Best Regards,
Vladimir Kirichenko

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 254 bytes
Desc: OpenPGP digital signature
Url : http://mail.openjdk.java.net/pipermail/closures-dev/attachments/20081116/48892fb4/attachment.bin 

From mark at twistedbanana.demon.co.uk  Sat Nov 15 18:53:05 2008
From: mark at twistedbanana.demon.co.uk (Mark Mahieu)
Date: Sun, 16 Nov 2008 02:53:05 +0000
Subject: why unrestricted closures are always void?
In-Reply-To: <491F75A9.6020809@gmail.com>
References: <491E24F6.2000005@gmail.com>	<15e8b9d20811151326t4627999fodce5770df9941993@mail.gmail.com>
	<20081116000745.20566v26zzgygcox@wimap.feld.cvut.cz>
	<491F75A9.6020809@gmail.com>
Message-ID: <995F13C7-D3B4-4FF4-B04D-F8604DABC13C@twistedbanana.demon.co.uk>

For your amusement...

Several months ago I realised that it's possible to use Neal's  
closures prototype to get code like the following to work:

for each (Employee empl : findEmployees())
     where (empl.yearsInService() > 5)
     orderBy (empl.getLastName()) {

	// do something with empl
}

In this example, it's the control invocation syntax which allows the  
chaining of several 'clauses' into a more sophisticated statement,  
with the 'arguments' to the 'where' and 'orderBy' clauses actually  
being closures which are evaluated on each iteration.

There are caveats to doing this with BGGA as it stands, but exploring  
the idea further I came to the conclusion that the technique could  
form the basis for another language feature, which would allow the  
creation of many different types of 'internal DSL' in Java.

I even started working on a spec for such a language feature, with  
the intention of following up with an implementation (just for my own  
amusement really, I don't see any chance of something like that  
ending up in Java).

One of the more interesting problems to solve relates directly to  
your examples, since many DSLs are best expressed in a form which  
returns a value.  For example, rephrasing the above example in a more  
LINQ-like style would allow the query (the each...where...orderBy  
part) to be assigned to something like an Iterable<Employee>, which  
could then be reused, passed around etc.  As you've noted, that's not  
possible when using the control invocation syntax.

It's a fun one to think about.

Regards,

Mark


On 16 Nov 2008, at 01:21, Vladimir Kirichenko wrote:

> Zdenek Tronicek wrote:
>> Neal, he means control invocation syntax. For example:
>
> Yeah. In ruby which have similar facility I can write something like:
>
> def _for(x, &f) yield x end
>
> and than invoke it:
>
> res = _for(5) do |x| x*2 end
>
> It will be good thing to have ability to write control abstractions  
> those could return results, i.e.:
>
> Iterable<Integer> squares = yield(int i : someIterable()) { i * i }
>
> will construct generator - similar to one that exists in bunch of  
> other languages.
>
> And it's just looks very strange:
>
> //compiles fine
> public static int f(int i, {int ==> int} block) { return  
> block.invoke(i); }
>
> //does fine
> System.out.println(f(3, {int i ==> i*i }));
>
> but:
>
> f(int i : 3) { return i * i; }
>
> reports error:
>
> cannot return a value from method whose result type is void
>            f(int i : 3) { return i * i; }
>
> It's not void neither in closure nor in method.
>
> -- 
> Best Regards,
> Vladimir Kirichenko
>



From vladimir.kirichenko at gmail.com  Sat Nov 15 18:53:49 2008
From: vladimir.kirichenko at gmail.com (Vladimir Kirichenko)
Date: Sun, 16 Nov 2008 04:53:49 +0200
Subject: why unrestricted closures are always void?
In-Reply-To: <491F75A9.6020809@gmail.com>
References: <491E24F6.2000005@gmail.com>	<15e8b9d20811151326t4627999fodce5770df9941993@mail.gmail.com>
	<20081116000745.20566v26zzgygcox@wimap.feld.cvut.cz>
	<491F75A9.6020809@gmail.com>
Message-ID: <491F8B3D.3020407@gmail.com>

Vladimir Kirichenko wrote:

> It's not void neither in closure nor in method.

Aha, have got my mistake about return meaning in this case. But in this 
case:

  method f in class test.Test cannot be applied to given types
required: int,{int ==> int}
found: int,{int ==> void}
            f(int i : 3) { out.println(i);}

there is no way to satisfy "required" type with control syntax.

Another one is for abstraction with return type:

     static for int eachEntry(int[] values, { int ==> void } block) {
                   for (int p : values)
                       block.invoke(p);
                 return 1;
           }

this one will not compile:
int y = for eachEntry(int i : v) {
       System.out.println(i);
     }

So there is mutually exclusive cases:

1. I can use "break" or "continue" only with "for xxx" invocation - but 
in this case return value is unaccessible.

2. I can access return value - but cannot use "continue" and "break" 
features of unrestricted closure.

Test.java:101: continue outside of loop
         int x =  eachEntry(v, {int i ==> if (i > 3) continue; 
out.println(i);});

but this will compile and work:

         int x =  eachEntry(v, {int i ==> if (i > 3) return; 
out.println(i);});

but in this notation there are 2 things:
1. code after this loop will not be executed.
2. return value will never be acquired.

Another example:

//compilation ok
  {int ==> void} t = {int i ==> if (i >3) return; out.println(i);};

//compilation ok
  static {int ==> void} terminator() {
         return {int i ==> out.println(i);};
}

  static {int ==> void} terminator() {
                 return {int i ==> if (i > 3) return; out.println(i);};
}

// oops
Test.java:86: missing return value
                 return {int i ==> if (i > 3) return; out.println(i);};

what is the difference? I can define local variable but cannot define a 
value?

Ok, lets hack it:
  //just to return argument
  static <T> T id(T t) {return t;}
  //compiles and works.
  {int ==> void} t = {int i ==> if (i >3) return; out.println(i);};
  {int ==> void} t2 = id(t);
  int x =  eachEntry(v, t2);

But this looks crazy - return defined elsewhere (possibly in different 
class) terminates local method. Imagine how collections handling method 
will look like, one writing the iteration code which applies closure 
passed from elsewhere and could not even imagine will the execution be 
continued after iteration or not:

    void m ({int ===> void} t) {

       for(int i: v) t.invoke(i);

       System.out.println("who knows if we ever reach here...");

    }



Looking to the previous examples I'm thinking - it there sense to be 
able to use unrestricted closures as an argument at all?

Or am I missing something?

-- 
Best Regards,
Vladimir Kirichenko

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 254 bytes
Desc: OpenPGP digital signature
Url : http://mail.openjdk.java.net/pipermail/closures-dev/attachments/20081116/46d9e2e4/attachment.bin 

From vladimir.kirichenko at gmail.com  Sat Nov 15 19:26:29 2008
From: vladimir.kirichenko at gmail.com (Vladimir Kirichenko)
Date: Sun, 16 Nov 2008 05:26:29 +0200
Subject: unrestricted closures
Message-ID: <491F92E5.7060609@gmail.com>

More accurate problematic example:

public class Test {

          static {int ==> int} temp;

          static void term() {
                  {int==>int} t = {int i ==> if (i > 5) return; i};
                  temp = t;

          }

          static int m(int i, {int ==> int} f) {
                  int y = f.invoke(i + 5);
                  return y;
          }

          static int caller(int i) {
                  return m(i,temp);
          }
          public static void main(String... args) {
                  term();
                  int x = caller(5);
                  System.out.println(x);
          }
}

Exception in thread "main" java.lang.UnmatchedTransfer: late transfer
          at java.lang.Jump.transfer(Jump.java:48)
          at test.Test$3.+invoke(Test.java:12)
          at test.Test$3.invoke(Test.java:12)
          at test.Test.m(Test.java:18)
          at test.Test.caller(Test.java:23)
          at test.Test.main(Test.java:29)
Caused by: test.Test$2

And it's not the bug - code like this could appear in different cases,
it just shows what unrestricted closures passing could raise.


-- 
Best Regards,
Vladimir Kirichenko
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 254 bytes
Desc: not available
Url : http://mail.openjdk.java.net/pipermail/closures-dev/attachments/20081116/8159e6fd/attachment.bin 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 254 bytes
Desc: OpenPGP digital signature
Url : http://mail.openjdk.java.net/pipermail/closures-dev/attachments/20081116/8159e6fd/attachment-0001.bin 

From tronicek at fel.cvut.cz  Sat Nov 15 23:59:18 2008
From: tronicek at fel.cvut.cz (Zdenek Tronicek)
Date: Sun, 16 Nov 2008 08:59:18 +0100
Subject: unrestricted closures
In-Reply-To: <491F92E5.7060609@gmail.com>
References: <491F92E5.7060609@gmail.com>
Message-ID: <20081116085918.12961skav9w9405y@wimap.feld.cvut.cz>

Quoting Vladimir Kirichenko <vladimir.kirichenko at gmail.com>:

> More accurate problematic example:
>
> public class Test {
>
>          static {int ==> int} temp;
>
>          static void term() {
>                  {int==>int} t = {int i ==> if (i > 5) return; i};
>                  temp = t;
>
>          }
>
>          static int m(int i, {int ==> int} f) {
>                  int y = f.invoke(i + 5);
>                  return y;
>          }
>
>          static int caller(int i) {
>                  return m(i,temp);
>          }
>          public static void main(String... args) {
>                  term();
>                  int x = caller(5);
>                  System.out.println(x);
>          }
> }
>
> Exception in thread "main" java.lang.UnmatchedTransfer: late transfer
>          at java.lang.Jump.transfer(Jump.java:48)
>          at test.Test$3.+invoke(Test.java:12)
>          at test.Test$3.invoke(Test.java:12)
>          at test.Test.m(Test.java:18)
>          at test.Test.caller(Test.java:23)
>          at test.Test.main(Test.java:29)
> Caused by: test.Test$2
>
> And it's not the bug - code like this could appear in different cases,
> it just shows what unrestricted closures passing could raise.

Right. This is not bug but a feature because the return statement, as  
well as continue and break, are lexically bound.

Z.
-- 
Zdenek Tronicek
Department of Computer Science and Engineering
Prague                   tel: +420 2 2435 7410
http://cs.felk.cvut.cz/~tronicek



From neal at gafter.com  Sun Nov 16 10:10:23 2008
From: neal at gafter.com (Neal Gafter)
Date: Sun, 16 Nov 2008 11:10:23 -0700
Subject: unrestricted closures
In-Reply-To: <491F92E5.7060609@gmail.com>
References: <491F92E5.7060609@gmail.com>
Message-ID: <15e8b9d20811161010gcc60819ibf6d4db794b477a1@mail.gmail.com>

All of the examples I've seen where you could get into this kind of
"trouble" involve making locally obvious errors.  The exception points you
precisely to where you made the mistake, too.  In this case the temp()
method clearly constructs a closure that should not escape its scope (it has
a return in it) yet is assigned to a static variable.

On Sat, Nov 15, 2008 at 8:26 PM, Vladimir Kirichenko <
vladimir.kirichenko at gmail.com> wrote:

> More accurate problematic example:
>
> public class Test {
>
>         static {int ==> int} temp;
>
>         static void term() {
>                 {int==>int} t = {int i ==> if (i > 5) return; i};
>                 temp = t;
>
>         }
>
>         static int m(int i, {int ==> int} f) {
>                 int y = f.invoke(i + 5);
>                 return y;
>         }
>
>         static int caller(int i) {
>                 return m(i,temp);
>         }
>         public static void main(String... args) {
>                 term();
>                 int x = caller(5);
>                 System.out.println(x);
>         }
> }
>
> Exception in thread "main" java.lang.UnmatchedTransfer: late transfer
>         at java.lang.Jump.transfer(Jump.java:48)
>         at test.Test$3.+invoke(Test.java:12)
>         at test.Test$3.invoke(Test.java:12)
>         at test.Test.m(Test.java:18)
>         at test.Test.caller(Test.java:23)
>         at test.Test.main(Test.java:29)
> Caused by: test.Test$2
>
> And it's not the bug - code like this could appear in different cases,
> it just shows what unrestricted closures passing could raise.
>
>
> --
> Best Regards,
> Vladimir Kirichenko
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.4-svn0 (GNU/Linux)
> Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org
>
> iD8DBQFJH5KD2HRuYYrY4NQRAlR8AJ0dClhBvckn4NMiF14oBjTIijZ6BACdFgF3
> Zd8fWLKRU1XGF0ob9vEmTqo=
> =uZP4
> -----END PGP SIGNATURE-----
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/closures-dev/attachments/20081116/d63457a0/attachment.html 

From vladimir.kirichenko at gmail.com  Sun Nov 16 14:44:07 2008
From: vladimir.kirichenko at gmail.com (Vladimir Kirichenko)
Date: Mon, 17 Nov 2008 00:44:07 +0200
Subject: unrestricted closures
In-Reply-To: <15e8b9d20811161010gcc60819ibf6d4db794b477a1@mail.gmail.com>
References: <491F92E5.7060609@gmail.com>
	<15e8b9d20811161010gcc60819ibf6d4db794b477a1@mail.gmail.com>
Message-ID: <4920A237.1020009@gmail.com>

Neal Gafter wrote:
> All of the examples I've seen where you could get into this kind of 
> "trouble" involve making locally obvious errors.  The exception points 
> you precisely to where you made the mistake, too.  In this case the 
> temp() method clearly constructs a closure that should not escape its 
> scope (it has a return in it) yet is assigned to a static variable.

The main point was to show that after the

unrestrictedClosure.invoke()

there are to guaranty that the following statements will execute.

so there will be no exception but the code will not be executed:

void m ({int ===> void} t) {

       t.invoke(i);

       System.out.println("who knows if we ever get here...");

    }

Doesn't it look strange?

-- 
Best Regards,
Vladimir Kirichenko

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 254 bytes
Desc: OpenPGP digital signature
Url : http://mail.openjdk.java.net/pipermail/closures-dev/attachments/20081117/c3dda7f4/attachment.bin 

From neal at gafter.com  Sun Nov 16 15:50:46 2008
From: neal at gafter.com (Neal Gafter)
Date: Sun, 16 Nov 2008 16:50:46 -0700
Subject: unrestricted closures
In-Reply-To: <4920A237.1020009@gmail.com>
References: <491F92E5.7060609@gmail.com>
	<15e8b9d20811161010gcc60819ibf6d4db794b477a1@mail.gmail.com>
	<4920A237.1020009@gmail.com>
Message-ID: <15e8b9d20811161550m2dd0bb75vc3bc15e4c56bdec@mail.gmail.com>

Right: (with or without closures) there is no guarantee that any particular
method completes normally unless it is specified to do so.

On Sun, Nov 16, 2008 at 3:44 PM, Vladimir Kirichenko <
vladimir.kirichenko at gmail.com> wrote:

> Neal Gafter wrote:
>
>> All of the examples I've seen where you could get into this kind of
>> "trouble" involve making locally obvious errors.  The exception points you
>> precisely to where you made the mistake, too.  In this case the temp()
>> method clearly constructs a closure that should not escape its scope (it has
>> a return in it) yet is assigned to a static variable.
>>
>
> The main point was to show that after the
>
> unrestrictedClosure.invoke()
>
> there are to guaranty that the following statements will execute.
>
> so there will be no exception but the code will not be executed:
>
> void m ({int ===> void} t) {
>
>      t.invoke(i);
>
>      System.out.println("who knows if we ever get here...");
>
>   }
>
> Doesn't it look strange?
>
> --
> Best Regards,
> Vladimir Kirichenko
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/closures-dev/attachments/20081116/fcd56609/attachment.html 

From jrl at staffmail.ed.ac.uk  Thu Nov 20 09:27:53 2008
From: jrl at staffmail.ed.ac.uk (John Longley)
Date: Thu, 20 Nov 2008 17:27:53 +0000
Subject: Static analysis of non-local transfers: a proposal
Message-ID: <20081120172753.vphjeep04kosco4k@www.staffmail.ed.ac.uk>


Dear Java closures people,

I very much appreciate your interesting and enterprising work on Javac.
As a theoretically minded researcher in programming languages, I have a
suggestion to offer regarding a possible scheme for static analysis of
programs involving return/break/continue, so as to pre-empt
UnmatchedNonlocalTransfer exceptions at runtime.

My colleague Phil Wadler at Edinburgh University recently drew my attention
to your work, suggesting that some of my own research ideas might find
applications there. Having studied your proposal (v0.5), it does indeed seem
plausible to me that this may be the case, so this message is a first attempt
at trying to convey my thoughts in Java terminology. Of course, it's highly
possible that you have thought through this kind of thing already - or that
I've misunderstood or overlooked something important! - but in any case,
I'd be very interested in discussing the issues further.

If I've got it right, my proposal has the following characteristics:

* It consists simply of a compile-time checkable restriction on programs of
   the Javac language exactly as proposed in the spec for v0.5. No additional
   annotations by the programmer are required.
* Programs obeying the restriction will never throw an
   UnmatchedNonlocalTransfer exception at runtime.
* The restriction doesn't rule out any standard Java programs (i.e. programs
   not involving closures).
* The restriction doesn't seem to exclude any of the motivating examples
   of Java programs with closures that I have seen so far.
* More tentatively, if the restrictions were adopted as part of the language
   spec, I believe this might also permit a simplification of the currently
   proposed system for tracking exceptions, yielding the same safety
   properties but requiring fewer exception-related annotations by the
   programmer. However, I won't go into details of this in the present note.

I'll describe the idea here using Java terminology. However, I'm also hoping
(soon!) to produce a draft paper presenting a toy language with formal typing
rules etc. which illustrates the essential issues, along with a proof of the
desired safety property for this language.

So here goes. I'll concentrate here on the restriction insofar as it affects
the use of "return", and then briefly comment on "break" and "continue".

The restriction consists of two aspects:

* A restriction on assignment expressions where the variable being
   assigned to is of function type. (Note however that *initializers* for
   variables of function type are not restricted, except as prescribed by
   the ordinary rules of Java, e.g. we can't have "return" if we're not
   within a method or constructor body.)
* A restriction on "return" statements themselves.

To describe the restrictions, the following notions are useful.
First, we suppose that in some preprocessing phase, the compiler has
annotated each occurrence of a method or constructor declaration with a
unique *target label*, and moreover has annotated each occurrence
of "return" with the appropriate target label as determined by lexical
scoping (i.e. with the target label of the closest enclosing method or
constructor declaration).

By a *dynamic element*, we shall mean either a variable of function type
or a target label. I don't think this is an especially good name (other
suggestions welcome!), but the intuition is that a dynamic element can be
"invoked" - by being applied in the case of a variable of function type,
or by being jumped to in the case of a target label - causing a computation
which might trigger (or itself be) an attempted transfer of control.
By contrast, a variable of primitive type is not considered "dynamic" since
there is nothing to "invoke". (It might seem odd that variables containing
references to objects are *not* deemed to be dynamic - roughly, this  
is because
a method invocation cannot *in itself* be responsible for a non-local  
transfer,
except indirectly via some other dynamic element such as a field in the
target object.)

Every dynamic element has a *scope*: the scope of a variable is
defined as usual, while the scope of a target label is deemed to be
the whole of the corresponding method or constructor declaration.
We define a partial order relation on dynamic elements as follows: y << x if
the scope of x strictly contains that of y. (The important intuition here is
"x may outlive y".) We also write y <<= x if the scope of x contains or is
equal to that of y.

We also have the usual notion of a variable appearing *free* in an
expression e. In addition, we say a target label t appears *free* in e if
e contains a return statement annotated with t and not "bound" by a
method/constructor declaration annotated by t within e itself.

We can now give an easy-to-understand but slightly crude first cut at our
restrictions as follows (I'll improve on this below):

* In an assignment expression x = e where x is of function type,
   no dynamic element y << x appears free in e.
* In a statement return e annotated with target label t,
   no dynamic element y <<= t appears free in e.

Thus, for example, the following four code fragments are forbidden:

1.   {=> int} x ;  // a field declaration, say
      void m () {
          x = {=> return 5 ;}
      }
      // scope of target label narrower than that of x

2.   {=> int} x ;
      void m () {
          {=> int} y = {=> return 5 ;} ;
          x = y ;
      }
      // scope of y narrower than that of x

3.   return {=> return 5 ;}
      // both returns will have the same target label here

4.   {=> int} x = {=> return 5 ;} ;
      return x ;
      // scope of x is narrower than that of target label

Whilst this version of our restriction suffices to guarantee safety of
returns, it imposes some possibly annoying restrictions on the way the
programmer has to write things. For example, under the above rules
one cannot write

      {int => {=> int}} K = {n:int => {=> n}} ;
      {=> int} f ;
      {  {=> int} g = {=> 5} ;
         f = K.invoke (g.invoke()) ;
      }

even though it is OK to write

      {int => {=> int}} K = {n:int => {=> n}} ;
      {=> int} f ;
      {  {=> int} g = {=> 5} ;
         int n = g.invoke() ;
         f = K.invoke(n) ;
      }

which is not essentially different. Thus, the programmer would sometimes
have to rewrite code in an unnatural style in order to conform to the
above rules.

The point about this example is that in the assignment

      f = K.invoke (g.invoke()) ;

the occurrence of g is not "dangerous", because the subexpression
g.invoke() is evaluated to a ground value (5) before the assignment is
performed, so there is no way for the dynamic behaviour of g to find
its way into that of f. One can take account of this via a slightly more
fine-grained version of our restrictions as follows.

Let's say a syntactic subexpression e' of e is *normal* if e' does not
appear inside a closure literal in e (i.e. underneath a lambda).
And say a dynamic element y *critically appears in* an expression e
if there is an occurrence y_0 of y which is not contained in any normal
subexpression e' of non-function type.

Thus, in the above example, g appears in K.invoke (g.invoke()),
though not critically. However, the appearance of g in {=> g.invoke()}
*is* critical, because the invocation of the dynamic behaviour of g
is delayed.

With these notions in place, we may now relax our restrictions to the
following (just by replacing "appears" by "critically appears"):

* In an assignment expression x = e where x is of function type,
   no dynamic element y << x critically appears free in e.
* In a statement return e annotated with target label t,
   no dynamic element y <<= t critically appears free in e.

One further simple relaxation is worth mentioning. For an assignment x = e,
the appearance of some y << x isn't dangerous if y is a variable and there
is no intervening target label between y and x. So we could in general define
y <<< x if for some target label t we have y <<= t << x, and then replace
"y << x" by "y <<< x" in the restriction on assignment expressions.

So that's the idea. It should now be reasonably clear how to extend this
scheme to deal with "break" and "continue", by annotating loop constructs
with target labels at appropriate points.

Of course, a few adjustments might be needed to take account of all aspects
of Java (e.g. I guess some care is also needed over where to put the target
labels in try-finally statements), but I'm hopeful that in essence the scheme
should be able to do everything one wants, without sacrificing much of
importance in terms of programming power or convenience.

Thanks for reading this far! I would be interested to know what you think,
so any reactions/queries/problematic examples gratefully received.

Cheers,
John Longley
(Lecturer, School of Informatics, University of Edinburgh, Scotland)


[PS: Please reply to my email address, as I guess I'm not a recipient of
this mailing list.]


-- 
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.




From mark at twistedbanana.demon.co.uk  Fri Nov 21 19:54:29 2008
From: mark at twistedbanana.demon.co.uk (Mark Mahieu)
Date: Sat, 22 Nov 2008 03:54:29 +0000
Subject: Static analysis of non-local transfers: a proposal
In-Reply-To: <20081120172753.vphjeep04kosco4k@www.staffmail.ed.ac.uk>
References: <20081120172753.vphjeep04kosco4k@www.staffmail.ed.ac.uk>
Message-ID: <A5CBDDF9-8D0C-4322-BE1A-0F62FCBF71C8@twistedbanana.demon.co.uk>

Hi John,

This is very interesting work - thanks for taking the time to  
describe your ideas!

One initial question:

If I've understood correctly, the restrictions you've described rely  
on detecting that the target of an assignment is of function type.   
However, the closures specification allows a closure literal such as  
{==> return;} to be converted to a 'compatible' interface type, and  
I'm wondering to what extent your proposal would be able to handle that?

Here's a concrete example of the kind I have in mind, which currently  
throws an UnmatchedTransfer:


class Conversion {
	
	interface Task {
		void exec();
	}

	static Task task;

	static void store(Task t) {
		task = t;
	}
	
	static void m() {
		store({==> return;});
	}

	public static void main(String[] args) {
		m();
		task.exec();
	}
}


As I see it, the only restrictions that could be applied here would  
be on the code in m(), since everything else is standard Java code.


Kind regards,

Mark Mahieu



On 20 Nov 2008, at 17:27, John Longley wrote:

>
> Dear Java closures people,
>
> I very much appreciate your interesting and enterprising work on  
> Javac.
> As a theoretically minded researcher in programming languages, I  
> have a
> suggestion to offer regarding a possible scheme for static analysis of
> programs involving return/break/continue, so as to pre-empt
> UnmatchedNonlocalTransfer exceptions at runtime.
>
> My colleague Phil Wadler at Edinburgh University recently drew my  
> attention
> to your work, suggesting that some of my own research ideas might find
> applications there. Having studied your proposal (v0.5), it does  
> indeed seem
> plausible to me that this may be the case, so this message is a  
> first attempt
> at trying to convey my thoughts in Java terminology. Of course,  
> it's highly
> possible that you have thought through this kind of thing already -  
> or that
> I've misunderstood or overlooked something important! - but in any  
> case,
> I'd be very interested in discussing the issues further.
>
> If I've got it right, my proposal has the following characteristics:
>
> * It consists simply of a compile-time checkable restriction on  
> programs of
>   the Javac language exactly as proposed in the spec for v0.5. No  
> additional
>   annotations by the programmer are required.
> * Programs obeying the restriction will never throw an
>   UnmatchedNonlocalTransfer exception at runtime.
> * The restriction doesn't rule out any standard Java programs (i.e.  
> programs
>   not involving closures).
> * The restriction doesn't seem to exclude any of the motivating  
> examples
>   of Java programs with closures that I have seen so far.
> * More tentatively, if the restrictions were adopted as part of the  
> language
>   spec, I believe this might also permit a simplification of the  
> currently
>   proposed system for tracking exceptions, yielding the same safety
>   properties but requiring fewer exception-related annotations by the
>   programmer. However, I won't go into details of this in the  
> present note.
>
> I'll describe the idea here using Java terminology. However, I'm  
> also hoping
> (soon!) to produce a draft paper presenting a toy language with  
> formal typing
> rules etc. which illustrates the essential issues, along with a  
> proof of the
> desired safety property for this language.
>
> So here goes. I'll concentrate here on the restriction insofar as  
> it affects
> the use of "return", and then briefly comment on "break" and  
> "continue".
>
> The restriction consists of two aspects:
>
> * A restriction on assignment expressions where the variable being
>   assigned to is of function type. (Note however that  
> *initializers* for
>   variables of function type are not restricted, except as  
> prescribed by
>   the ordinary rules of Java, e.g. we can't have "return" if we're not
>   within a method or constructor body.)
> * A restriction on "return" statements themselves.
>
> To describe the restrictions, the following notions are useful.
> First, we suppose that in some preprocessing phase, the compiler has
> annotated each occurrence of a method or constructor declaration  
> with a
> unique *target label*, and moreover has annotated each occurrence
> of "return" with the appropriate target label as determined by lexical
> scoping (i.e. with the target label of the closest enclosing method or
> constructor declaration).
>
> By a *dynamic element*, we shall mean either a variable of function  
> type
> or a target label. I don't think this is an especially good name  
> (other
> suggestions welcome!), but the intuition is that a dynamic element  
> can be
> "invoked" - by being applied in the case of a variable of function  
> type,
> or by being jumped to in the case of a target label - causing a  
> computation
> which might trigger (or itself be) an attempted transfer of control.
> By contrast, a variable of primitive type is not considered  
> "dynamic" since
> there is nothing to "invoke". (It might seem odd that variables  
> containing
> references to objects are *not* deemed to be dynamic - roughly,  
> this is because
> a method invocation cannot *in itself* be responsible for a non- 
> local transfer,
> except indirectly via some other dynamic element such as a field in  
> the
> target object.)
>
> Every dynamic element has a *scope*: the scope of a variable is
> defined as usual, while the scope of a target label is deemed to be
> the whole of the corresponding method or constructor declaration.
> We define a partial order relation on dynamic elements as follows:  
> y << x if
> the scope of x strictly contains that of y. (The important  
> intuition here is
> "x may outlive y".) We also write y <<= x if the scope of x  
> contains or is
> equal to that of y.
>
> We also have the usual notion of a variable appearing *free* in an
> expression e. In addition, we say a target label t appears *free*  
> in e if
> e contains a return statement annotated with t and not "bound" by a
> method/constructor declaration annotated by t within e itself.
>
> We can now give an easy-to-understand but slightly crude first cut  
> at our
> restrictions as follows (I'll improve on this below):
>
> * In an assignment expression x = e where x is of function type,
>   no dynamic element y << x appears free in e.
> * In a statement return e annotated with target label t,
>   no dynamic element y <<= t appears free in e.
>
> Thus, for example, the following four code fragments are forbidden:
>
> 1.   {=> int} x ;  // a field declaration, say
>      void m () {
>          x = {=> return 5 ;}
>      }
>      // scope of target label narrower than that of x
>
> 2.   {=> int} x ;
>      void m () {
>          {=> int} y = {=> return 5 ;} ;
>          x = y ;
>      }
>      // scope of y narrower than that of x
>
> 3.   return {=> return 5 ;}
>      // both returns will have the same target label here
>
> 4.   {=> int} x = {=> return 5 ;} ;
>      return x ;
>      // scope of x is narrower than that of target label
>
> Whilst this version of our restriction suffices to guarantee safety of
> returns, it imposes some possibly annoying restrictions on the way the
> programmer has to write things. For example, under the above rules
> one cannot write
>
>      {int => {=> int}} K = {n:int => {=> n}} ;
>      {=> int} f ;
>      {  {=> int} g = {=> 5} ;
>         f = K.invoke (g.invoke()) ;
>      }
>
> even though it is OK to write
>
>      {int => {=> int}} K = {n:int => {=> n}} ;
>      {=> int} f ;
>      {  {=> int} g = {=> 5} ;
>         int n = g.invoke() ;
>         f = K.invoke(n) ;
>      }
>
> which is not essentially different. Thus, the programmer would  
> sometimes
> have to rewrite code in an unnatural style in order to conform to the
> above rules.
>
> The point about this example is that in the assignment
>
>      f = K.invoke (g.invoke()) ;
>
> the occurrence of g is not "dangerous", because the subexpression
> g.invoke() is evaluated to a ground value (5) before the assignment is
> performed, so there is no way for the dynamic behaviour of g to find
> its way into that of f. One can take account of this via a slightly  
> more
> fine-grained version of our restrictions as follows.
>
> Let's say a syntactic subexpression e' of e is *normal* if e' does not
> appear inside a closure literal in e (i.e. underneath a lambda).
> And say a dynamic element y *critically appears in* an expression e
> if there is an occurrence y_0 of y which is not contained in any  
> normal
> subexpression e' of non-function type.
>
> Thus, in the above example, g appears in K.invoke (g.invoke()),
> though not critically. However, the appearance of g in {=> g.invoke()}
> *is* critical, because the invocation of the dynamic behaviour of g
> is delayed.
>
> With these notions in place, we may now relax our restrictions to the
> following (just by replacing "appears" by "critically appears"):
>
> * In an assignment expression x = e where x is of function type,
>   no dynamic element y << x critically appears free in e.
> * In a statement return e annotated with target label t,
>   no dynamic element y <<= t critically appears free in e.
>
> One further simple relaxation is worth mentioning. For an  
> assignment x = e,
> the appearance of some y << x isn't dangerous if y is a variable  
> and there
> is no intervening target label between y and x. So we could in  
> general define
> y <<< x if for some target label t we have y <<= t << x, and then  
> replace
> "y << x" by "y <<< x" in the restriction on assignment expressions.
>
> So that's the idea. It should now be reasonably clear how to extend  
> this
> scheme to deal with "break" and "continue", by annotating loop  
> constructs
> with target labels at appropriate points.
>
> Of course, a few adjustments might be needed to take account of all  
> aspects
> of Java (e.g. I guess some care is also needed over where to put  
> the target
> labels in try-finally statements), but I'm hopeful that in essence  
> the scheme
> should be able to do everything one wants, without sacrificing much of
> importance in terms of programming power or convenience.
>
> Thanks for reading this far! I would be interested to know what you  
> think,
> so any reactions/queries/problematic examples gratefully received.
>
> Cheers,
> John Longley
> (Lecturer, School of Informatics, University of Edinburgh, Scotland)
>
>
> [PS: Please reply to my email address, as I guess I'm not a  
> recipient of
> this mailing list.]
>
>
> -- 
> The University of Edinburgh is a charitable body, registered in
> Scotland, with registration number SC005336.
>
>



From jrl at staffmail.ed.ac.uk  Sat Nov 22 15:24:54 2008
From: jrl at staffmail.ed.ac.uk (John Longley)
Date: Sat, 22 Nov 2008 23:24:54 +0000
Subject: Static analysis of non-local transfers: a proposal
In-Reply-To: <A5CBDDF9-8D0C-4322-BE1A-0F62FCBF71C8@twistedbanana.demon.co.uk>
References: <20081120172753.vphjeep04kosco4k@www.staffmail.ed.ac.uk>
	<A5CBDDF9-8D0C-4322-BE1A-0F62FCBF71C8@twistedbanana.demon.co.uk>
Message-ID: <20081122232454.inydevxfk08osg0g@www.staffmail.ed.ac.uk>


Hi Mark,

Many thanks for the speedy and encouraging response!

> If I've understood correctly, the restrictions you've described rely on
> detecting that the target of an assignment is of function type.
> However, the closures specification allows a closure literal such as
> {==> return;} to be converted to a 'compatible' interface type, and I'm
> wondering to what extent your proposal would be able to handle that?

I can see that this is an important issue. My proposal here would be that
a function type should be at least conceptually distinguished from the  
corresponding interface type, to the extent that we can think in terms  
of
a type conversion being inserted by the compiler, e.g.:

> 	static void m() {
> 		store (CONVERT {==> return;});
> 	}

I would then propose that such type-converted expressions should be subject
to a restriction similar in spirit to the others (but a bit stronger):

* In any expression CONVERT e (where CONVERT is an implicit conversion from
   function to interface type), no *shadowed* dynamic element y (i.e. one for
   which there is some target label l <<= y) may critically appear free in e.

As I see it, I don't really have any other option here, since once  
something is
treated as having an interface type, it can (under standard Java) be  
stored unrestrictedly and one loses all control of it.

[Of course, one should take care to set things up so that function application
is always permitted and isn't considered as requiring a type conversion,
even if the chosen syntax looks like method invocation on an object: e.g.

   {==> return;}.exec()

should certainly be allowed.]

I would be very interested to know whether the idea that not all closure
expressions are convertible to interface types appears at all acceptable
in terms of current thinking on type compatibility. It seems to me that
there is at least a coherent point of view which makes sense of this idea.
To wit: objects and closures clearly overlap in what they can express, but
they offer different things: objects can be stored in variables without
restriction but don't give you the non-local transfer features; while
closures give you these interesting features but (under my proposal) at the
cost of some restrictions on how they are stored. So, depending on what you
are trying to do, you can have either of these but not both at once - the
point being precisely that the combination of non-local transfer with  
unrestricted store is inherently unsafe. Type conversion thus makes  
sense
as long as one is in the intersection of the two regimes. As I say, I think
one is forced to take something like this view if one hopes to pre-empt
unmatched transfers via a static analysis.

While I'm here: there is a small loophole in what I described in my first
email, which I discovered in the course of my formal analysis. I said that
initializers needn't be subject to any special restriction - but that's not
quite true, as the following obscure scenario shows. Suppose one has a local
class declaration appearing within the declaration of e.g. a method m, and
suppose the local class has a field f of function type. Then an initializer
for f must be prevented from including a return statement that binds to
the target label of m (and likewise for assignments to f within initializer
blocks, I guess). E.g.

     {==> int} m() {
         class c {
            {==> int} f = {==> return ;}
         } ;
         return new c().f
     }

     // later:
     m().exec() ;  // throws unmatched transfer

I can certainly give appropriate restrictions to fix this problem - but on
the other hand, the issues only arises in such bizarre contexts that one
might prefer a simpler solution, such as simply forbidding fields of function
type in local class declarations appearing within the scope of a target label.

I will be much more confident of my scheme as a whole once my formal analysis
is completed! I expect to have a rough outline available within the next
couple of days.

Best wishes,
John


-- 
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.




From neal at gafter.com  Sat Nov 22 15:46:24 2008
From: neal at gafter.com (Neal Gafter)
Date: Sat, 22 Nov 2008 16:46:24 -0700
Subject: Static analysis of non-local transfers: a proposal
In-Reply-To: <20081120172753.vphjeep04kosco4k@www.staffmail.ed.ac.uk>
References: <20081120172753.vphjeep04kosco4k@www.staffmail.ed.ac.uk>
Message-ID: <15e8b9d20811221546v32e3f028nc245a4be0b2ac0e9@mail.gmail.com>

John-

I haven't looked at this in detail, but I should point out that one
important class of use cases requires the presence of these exceptions.  See
http://gafter.blogspot.com/2006/10/concurrent-loops-using-java-closures.html.
This kind of API is implemented by catching the unmatched nonlocal transfer
exception and retrying the transfer in the correct thread.

Also, as Mark Mahieu has pointed out, function types are merely a syntactic
shorthand for interface types.  In order to take the approach you suggest,
they would have to be something fundamentally different.

Regards,
Neal

On Thu, Nov 20, 2008 at 10:27 AM, John Longley <jrl at staffmail.ed.ac.uk>wrote:

>
> Dear Java closures people,
>
> I very much appreciate your interesting and enterprising work on Javac.
> As a theoretically minded researcher in programming languages, I have a
> suggestion to offer regarding a possible scheme for static analysis of
> programs involving return/break/continue, so as to pre-empt
> UnmatchedNonlocalTransfer exceptions at runtime.
>
> My colleague Phil Wadler at Edinburgh University recently drew my attention
> to your work, suggesting that some of my own research ideas might find
> applications there. Having studied your proposal (v0.5), it does indeed
> seem
> plausible to me that this may be the case, so this message is a first
> attempt
> at trying to convey my thoughts in Java terminology. Of course, it's highly
> possible that you have thought through this kind of thing already - or that
> I've misunderstood or overlooked something important! - but in any case,
> I'd be very interested in discussing the issues further.
>
> If I've got it right, my proposal has the following characteristics:
>
> * It consists simply of a compile-time checkable restriction on programs of
>  the Javac language exactly as proposed in the spec for v0.5. No additional
>  annotations by the programmer are required.
> * Programs obeying the restriction will never throw an
>  UnmatchedNonlocalTransfer exception at runtime.
> * The restriction doesn't rule out any standard Java programs (i.e.
> programs
>  not involving closures).
> * The restriction doesn't seem to exclude any of the motivating examples
>  of Java programs with closures that I have seen so far.
> * More tentatively, if the restrictions were adopted as part of the
> language
>  spec, I believe this might also permit a simplification of the currently
>  proposed system for tracking exceptions, yielding the same safety
>  properties but requiring fewer exception-related annotations by the
>  programmer. However, I won't go into details of this in the present note.
>
> I'll describe the idea here using Java terminology. However, I'm also
> hoping
> (soon!) to produce a draft paper presenting a toy language with formal
> typing
> rules etc. which illustrates the essential issues, along with a proof of
> the
> desired safety property for this language.
>
> So here goes. I'll concentrate here on the restriction insofar as it
> affects
> the use of "return", and then briefly comment on "break" and "continue".
>
> The restriction consists of two aspects:
>
> * A restriction on assignment expressions where the variable being
>  assigned to is of function type. (Note however that *initializers* for
>  variables of function type are not restricted, except as prescribed by
>  the ordinary rules of Java, e.g. we can't have "return" if we're not
>  within a method or constructor body.)
> * A restriction on "return" statements themselves.
>
> To describe the restrictions, the following notions are useful.
> First, we suppose that in some preprocessing phase, the compiler has
> annotated each occurrence of a method or constructor declaration with a
> unique *target label*, and moreover has annotated each occurrence
> of "return" with the appropriate target label as determined by lexical
> scoping (i.e. with the target label of the closest enclosing method or
> constructor declaration).
>
> By a *dynamic element*, we shall mean either a variable of function type
> or a target label. I don't think this is an especially good name (other
> suggestions welcome!), but the intuition is that a dynamic element can be
> "invoked" - by being applied in the case of a variable of function type,
> or by being jumped to in the case of a target label - causing a computation
> which might trigger (or itself be) an attempted transfer of control.
> By contrast, a variable of primitive type is not considered "dynamic" since
> there is nothing to "invoke". (It might seem odd that variables containing
> references to objects are *not* deemed to be dynamic - roughly, this is
> because
> a method invocation cannot *in itself* be responsible for a non-local
> transfer,
> except indirectly via some other dynamic element such as a field in the
> target object.)
>
> Every dynamic element has a *scope*: the scope of a variable is
> defined as usual, while the scope of a target label is deemed to be
> the whole of the corresponding method or constructor declaration.
> We define a partial order relation on dynamic elements as follows: y << x
> if
> the scope of x strictly contains that of y. (The important intuition here
> is
> "x may outlive y".) We also write y <<= x if the scope of x contains or is
> equal to that of y.
>
> We also have the usual notion of a variable appearing *free* in an
> expression e. In addition, we say a target label t appears *free* in e if
> e contains a return statement annotated with t and not "bound" by a
> method/constructor declaration annotated by t within e itself.
>
> We can now give an easy-to-understand but slightly crude first cut at our
> restrictions as follows (I'll improve on this below):
>
> * In an assignment expression x = e where x is of function type,
>  no dynamic element y << x appears free in e.
> * In a statement return e annotated with target label t,
>  no dynamic element y <<= t appears free in e.
>
> Thus, for example, the following four code fragments are forbidden:
>
> 1.   {=> int} x ;  // a field declaration, say
>     void m () {
>         x = {=> return 5 ;}
>     }
>     // scope of target label narrower than that of x
>
> 2.   {=> int} x ;
>     void m () {
>         {=> int} y = {=> return 5 ;} ;
>         x = y ;
>     }
>     // scope of y narrower than that of x
>
> 3.   return {=> return 5 ;}
>     // both returns will have the same target label here
>
> 4.   {=> int} x = {=> return 5 ;} ;
>     return x ;
>     // scope of x is narrower than that of target label
>
> Whilst this version of our restriction suffices to guarantee safety of
> returns, it imposes some possibly annoying restrictions on the way the
> programmer has to write things. For example, under the above rules
> one cannot write
>
>     {int => {=> int}} K = {n:int => {=> n}} ;
>     {=> int} f ;
>     {  {=> int} g = {=> 5} ;
>        f = K.invoke (g.invoke()) ;
>     }
>
> even though it is OK to write
>
>     {int => {=> int}} K = {n:int => {=> n}} ;
>     {=> int} f ;
>     {  {=> int} g = {=> 5} ;
>        int n = g.invoke() ;
>        f = K.invoke(n) ;
>     }
>
> which is not essentially different. Thus, the programmer would sometimes
> have to rewrite code in an unnatural style in order to conform to the
> above rules.
>
> The point about this example is that in the assignment
>
>     f = K.invoke (g.invoke()) ;
>
> the occurrence of g is not "dangerous", because the subexpression
> g.invoke() is evaluated to a ground value (5) before the assignment is
> performed, so there is no way for the dynamic behaviour of g to find
> its way into that of f. One can take account of this via a slightly more
> fine-grained version of our restrictions as follows.
>
> Let's say a syntactic subexpression e' of e is *normal* if e' does not
> appear inside a closure literal in e (i.e. underneath a lambda).
> And say a dynamic element y *critically appears in* an expression e
> if there is an occurrence y_0 of y which is not contained in any normal
> subexpression e' of non-function type.
>
> Thus, in the above example, g appears in K.invoke (g.invoke()),
> though not critically. However, the appearance of g in {=> g.invoke()}
> *is* critical, because the invocation of the dynamic behaviour of g
> is delayed.
>
> With these notions in place, we may now relax our restrictions to the
> following (just by replacing "appears" by "critically appears"):
>
> * In an assignment expression x = e where x is of function type,
>  no dynamic element y << x critically appears free in e.
> * In a statement return e annotated with target label t,
>  no dynamic element y <<= t critically appears free in e.
>
> One further simple relaxation is worth mentioning. For an assignment x = e,
> the appearance of some y << x isn't dangerous if y is a variable and there
> is no intervening target label between y and x. So we could in general
> define
> y <<< x if for some target label t we have y <<= t << x, and then replace
> "y << x" by "y <<< x" in the restriction on assignment expressions.
>
> So that's the idea. It should now be reasonably clear how to extend this
> scheme to deal with "break" and "continue", by annotating loop constructs
> with target labels at appropriate points.
>
> Of course, a few adjustments might be needed to take account of all aspects
> of Java (e.g. I guess some care is also needed over where to put the target
> labels in try-finally statements), but I'm hopeful that in essence the
> scheme
> should be able to do everything one wants, without sacrificing much of
> importance in terms of programming power or convenience.
>
> Thanks for reading this far! I would be interested to know what you think,
> so any reactions/queries/problematic examples gratefully received.
>
> Cheers,
> John Longley
> (Lecturer, School of Informatics, University of Edinburgh, Scotland)
>
>
> [PS: Please reply to my email address, as I guess I'm not a recipient of
> this mailing list.]
>
>
> --
> The University of Edinburgh is a charitable body, registered in
> Scotland, with registration number SC005336.
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/closures-dev/attachments/20081122/058accf3/attachment.html 

From mark at twistedbanana.demon.co.uk  Sat Nov 22 18:17:54 2008
From: mark at twistedbanana.demon.co.uk (Mark Mahieu)
Date: Sun, 23 Nov 2008 02:17:54 +0000
Subject: Static analysis of non-local transfers: a proposal
In-Reply-To: <15e8b9d20811221546v32e3f028nc245a4be0b2ac0e9@mail.gmail.com>
References: <20081120172753.vphjeep04kosco4k@www.staffmail.ed.ac.uk>
	<15e8b9d20811221546v32e3f028nc245a4be0b2ac0e9@mail.gmail.com>
Message-ID: <9D638248-E45A-43AD-9490-554BFF9936BC@twistedbanana.demon.co.uk>

I interpreted John's proposal as really being targeted at the 'late  
transfer' rather than 'wrong thread' form, and as such the  
restrictions shouldn't prevent the deliberate use of  
UnmatchedTransfer for the creation of APIs like 'for eachConcurrently'.

Regards,

Mark


On 22 Nov 2008, at 23:46, Neal Gafter wrote:

> John-
>
> I haven't looked at this in detail, but I should point out that one  
> important class of use cases requires the presence of these  
> exceptions.  See http://gafter.blogspot.com/2006/10/concurrent- 
> loops-using-java-closures.html.  This kind of API is implemented by  
> catching the unmatched nonlocal transfer exception and retrying the  
> transfer in the correct thread.
>
> Also, as Mark Mahieu has pointed out, function types are merely a  
> syntactic shorthand for interface types.  In order to take the  
> approach you suggest, they would have to be something fundamentally  
> different.
>
> Regards,
> Neal
>
> On Thu, Nov 20, 2008 at 10:27 AM, John Longley  
> <jrl at staffmail.ed.ac.uk> wrote:
>
> Dear Java closures people,
>
> I very much appreciate your interesting and enterprising work on  
> Javac.
> As a theoretically minded researcher in programming languages, I  
> have a
> suggestion to offer regarding a possible scheme for static analysis of
> programs involving return/break/continue, so as to pre-empt
> UnmatchedNonlocalTransfer exceptions at runtime.
>
> My colleague Phil Wadler at Edinburgh University recently drew my  
> attention
> to your work, suggesting that some of my own research ideas might find
> applications there. Having studied your proposal (v0.5), it does  
> indeed seem
> plausible to me that this may be the case, so this message is a  
> first attempt
> at trying to convey my thoughts in Java terminology. Of course,  
> it's highly
> possible that you have thought through this kind of thing already -  
> or that
> I've misunderstood or overlooked something important! - but in any  
> case,
> I'd be very interested in discussing the issues further.
>
> If I've got it right, my proposal has the following characteristics:
>
> * It consists simply of a compile-time checkable restriction on  
> programs of
>  the Javac language exactly as proposed in the spec for v0.5. No  
> additional
>  annotations by the programmer are required.
> * Programs obeying the restriction will never throw an
>  UnmatchedNonlocalTransfer exception at runtime.
> * The restriction doesn't rule out any standard Java programs (i.e.  
> programs
>  not involving closures).
> * The restriction doesn't seem to exclude any of the motivating  
> examples
>  of Java programs with closures that I have seen so far.
> * More tentatively, if the restrictions were adopted as part of the  
> language
>  spec, I believe this might also permit a simplification of the  
> currently
>  proposed system for tracking exceptions, yielding the same safety
>  properties but requiring fewer exception-related annotations by the
>  programmer. However, I won't go into details of this in the  
> present note.
>
> I'll describe the idea here using Java terminology. However, I'm  
> also hoping
> (soon!) to produce a draft paper presenting a toy language with  
> formal typing
> rules etc. which illustrates the essential issues, along with a  
> proof of the
> desired safety property for this language.
>
> So here goes. I'll concentrate here on the restriction insofar as  
> it affects
> the use of "return", and then briefly comment on "break" and  
> "continue".
>
> The restriction consists of two aspects:
>
> * A restriction on assignment expressions where the variable being
>  assigned to is of function type. (Note however that *initializers*  
> for
>  variables of function type are not restricted, except as  
> prescribed by
>  the ordinary rules of Java, e.g. we can't have "return" if we're not
>  within a method or constructor body.)
> * A restriction on "return" statements themselves.
>
> To describe the restrictions, the following notions are useful.
> First, we suppose that in some preprocessing phase, the compiler has
> annotated each occurrence of a method or constructor declaration  
> with a
> unique *target label*, and moreover has annotated each occurrence
> of "return" with the appropriate target label as determined by lexical
> scoping (i.e. with the target label of the closest enclosing method or
> constructor declaration).
>
> By a *dynamic element*, we shall mean either a variable of function  
> type
> or a target label. I don't think this is an especially good name  
> (other
> suggestions welcome!), but the intuition is that a dynamic element  
> can be
> "invoked" - by being applied in the case of a variable of function  
> type,
> or by being jumped to in the case of a target label - causing a  
> computation
> which might trigger (or itself be) an attempted transfer of control.
> By contrast, a variable of primitive type is not considered  
> "dynamic" since
> there is nothing to "invoke". (It might seem odd that variables  
> containing
> references to objects are *not* deemed to be dynamic - roughly,  
> this is because
> a method invocation cannot *in itself* be responsible for a non- 
> local transfer,
> except indirectly via some other dynamic element such as a field in  
> the
> target object.)
>
> Every dynamic element has a *scope*: the scope of a variable is
> defined as usual, while the scope of a target label is deemed to be
> the whole of the corresponding method or constructor declaration.
> We define a partial order relation on dynamic elements as follows:  
> y << x if
> the scope of x strictly contains that of y. (The important  
> intuition here is
> "x may outlive y".) We also write y <<= x if the scope of x  
> contains or is
> equal to that of y.
>
> We also have the usual notion of a variable appearing *free* in an
> expression e. In addition, we say a target label t appears *free*  
> in e if
> e contains a return statement annotated with t and not "bound" by a
> method/constructor declaration annotated by t within e itself.
>
> We can now give an easy-to-understand but slightly crude first cut  
> at our
> restrictions as follows (I'll improve on this below):
>
> * In an assignment expression x = e where x is of function type,
>  no dynamic element y << x appears free in e.
> * In a statement return e annotated with target label t,
>  no dynamic element y <<= t appears free in e.
>
> Thus, for example, the following four code fragments are forbidden:
>
> 1.   {=> int} x ;  // a field declaration, say
>     void m () {
>         x = {=> return 5 ;}
>     }
>     // scope of target label narrower than that of x
>
> 2.   {=> int} x ;
>     void m () {
>         {=> int} y = {=> return 5 ;} ;
>         x = y ;
>     }
>     // scope of y narrower than that of x
>
> 3.   return {=> return 5 ;}
>     // both returns will have the same target label here
>
> 4.   {=> int} x = {=> return 5 ;} ;
>     return x ;
>     // scope of x is narrower than that of target label
>
> Whilst this version of our restriction suffices to guarantee safety of
> returns, it imposes some possibly annoying restrictions on the way the
> programmer has to write things. For example, under the above rules
> one cannot write
>
>     {int => {=> int}} K = {n:int => {=> n}} ;
>     {=> int} f ;
>     {  {=> int} g = {=> 5} ;
>        f = K.invoke (g.invoke()) ;
>     }
>
> even though it is OK to write
>
>     {int => {=> int}} K = {n:int => {=> n}} ;
>     {=> int} f ;
>     {  {=> int} g = {=> 5} ;
>        int n = g.invoke() ;
>        f = K.invoke(n) ;
>     }
>
> which is not essentially different. Thus, the programmer would  
> sometimes
> have to rewrite code in an unnatural style in order to conform to the
> above rules.
>
> The point about this example is that in the assignment
>
>     f = K.invoke (g.invoke()) ;
>
> the occurrence of g is not "dangerous", because the subexpression
> g.invoke() is evaluated to a ground value (5) before the assignment is
> performed, so there is no way for the dynamic behaviour of g to find
> its way into that of f. One can take account of this via a slightly  
> more
> fine-grained version of our restrictions as follows.
>
> Let's say a syntactic subexpression e' of e is *normal* if e' does not
> appear inside a closure literal in e (i.e. underneath a lambda).
> And say a dynamic element y *critically appears in* an expression e
> if there is an occurrence y_0 of y which is not contained in any  
> normal
> subexpression e' of non-function type.
>
> Thus, in the above example, g appears in K.invoke (g.invoke()),
> though not critically. However, the appearance of g in {=> g.invoke()}
> *is* critical, because the invocation of the dynamic behaviour of g
> is delayed.
>
> With these notions in place, we may now relax our restrictions to the
> following (just by replacing "appears" by "critically appears"):
>
> * In an assignment expression x = e where x is of function type,
>  no dynamic element y << x critically appears free in e.
> * In a statement return e annotated with target label t,
>  no dynamic element y <<= t critically appears free in e.
>
> One further simple relaxation is worth mentioning. For an  
> assignment x = e,
> the appearance of some y << x isn't dangerous if y is a variable  
> and there
> is no intervening target label between y and x. So we could in  
> general define
> y <<< x if for some target label t we have y <<= t << x, and then  
> replace
> "y << x" by "y <<< x" in the restriction on assignment expressions.
>
> So that's the idea. It should now be reasonably clear how to extend  
> this
> scheme to deal with "break" and "continue", by annotating loop  
> constructs
> with target labels at appropriate points.
>
> Of course, a few adjustments might be needed to take account of all  
> aspects
> of Java (e.g. I guess some care is also needed over where to put  
> the target
> labels in try-finally statements), but I'm hopeful that in essence  
> the scheme
> should be able to do everything one wants, without sacrificing much of
> importance in terms of programming power or convenience.
>
> Thanks for reading this far! I would be interested to know what you  
> think,
> so any reactions/queries/problematic examples gratefully received.
>
> Cheers,
> John Longley
> (Lecturer, School of Informatics, University of Edinburgh, Scotland)
>
>
> [PS: Please reply to my email address, as I guess I'm not a  
> recipient of
> this mailing list.]
>
>
> -- 
> The University of Edinburgh is a charitable body, registered in
> Scotland, with registration number SC005336.
>
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/closures-dev/attachments/20081123/4a9f3e66/attachment.html 

From tronicek at fel.cvut.cz  Sun Nov 23 01:55:25 2008
From: tronicek at fel.cvut.cz (Zdenek Tronicek)
Date: Sun, 23 Nov 2008 10:55:25 +0100
Subject: Static analysis of non-local transfers: a proposal
In-Reply-To: <20081122232454.inydevxfk08osg0g@www.staffmail.ed.ac.uk>
References: <20081120172753.vphjeep04kosco4k@www.staffmail.ed.ac.uk>
	<A5CBDDF9-8D0C-4322-BE1A-0F62FCBF71C8@twistedbanana.demon.co.uk>
	<20081122232454.inydevxfk08osg0g@www.staffmail.ed.ac.uk>
Message-ID: <20081123105525.97826e8ouah0oozx@wimap.feld.cvut.cz>

Quoting John Longley <jrl at staffmail.ed.ac.uk>:

>
>     {==> int} m() {
>         class c {
>            {==> int} f = {==> return ;}
>         } ;
>         return new c().f
>     }
>

Hi John,

such restriction already exists under the current prototype. For example,
this

   interface X {
     int methodX();
   }

   public static X makeX() {
     X p = new X() {
       { => void } e = { => return null; };
       public int methodX() {
         e.invoke();
         return 42;
       }
     };
     return p;
   }

does not compile.

Zdenek
-- 
Zdenek Tronicek
Department of Computer Science and Engineering
Prague                   tel: +420 2 2435 7410
http://cs.felk.cvut.cz/~tronicek





From jrl at staffmail.ed.ac.uk  Mon Nov 24 04:58:21 2008
From: jrl at staffmail.ed.ac.uk (John Longley)
Date: Mon, 24 Nov 2008 12:58:21 +0000
Subject: Static analysis of non-local transfers: a proposal
In-Reply-To: <9D638248-E45A-43AD-9490-554BFF9936BC@twistedbanana.demon.co.uk>
References: <20081120172753.vphjeep04kosco4k@www.staffmail.ed.ac.uk>
	<15e8b9d20811221546v32e3f028nc245a4be0b2ac0e9@mail.gmail.com>
	<9D638248-E45A-43AD-9490-554BFF9936BC@twistedbanana.demon.co.uk>
Message-ID: <20081124125821.jngv8pihfk40040g@www.staffmail.ed.ac.uk>


Thanks to all for the interesting replies.

It's true that my proposal depends on function types and closure types  
being distinguished at an early stage of processing. However, once my  
proposed conditions have been checked, the types can of course be  
identified.

The concurrency examples are very interesting. Mark wrote:

> I interpreted John's proposal as really being targeted at the 'late
> transfer' rather than 'wrong thread' form, and as such the
> restrictions shouldn't prevent the deliberate use of  UnmatchedTransfer
> for the creation of APIs like 'for eachConcurrently'.

Yes, that's how I see it. One possible approach might be simply to  
accept that my proposal doesn't exclude UnmatchedTransfer in  
concurrent situations. This mightn't seem so bad if the idea is that  
the clever concurrent stuff is written once-for-all by an expert who  
knows what they're doing and then stuck in an API.

The crucial lines of code for handling these transfers (taken from  
Mark's blog) are

       if (throwable instanceof UnmatchedTransfer)
            throw ((UnmatchedTransfer) throwable).transfer();

Although I don't understand exactly how this works, I imagine that  
returning control to the correct thread is pretty much *always* what  
one wants to do in practice. So perhaps there is also a more  
sophisticated approach which would build in this behaviour somehow -  
e.g. any concurrent code involving closures is treated as implicitly  
having a handler like the above, so that it's still guaranteed that  
the programmer never sees an UnmatchedTransfer? (I haven't thought  
this through in detail.)

As a further point, I'd also like to describe how I think my proposal  
can simplify the tracking of exceptions, but I'll save this for  
another message.

Cheers, John


-- 
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.




From mark at twistedbanana.demon.co.uk  Tue Nov 25 03:24:56 2008
From: mark at twistedbanana.demon.co.uk (Mark Mahieu)
Date: Tue, 25 Nov 2008 11:24:56 +0000
Subject: Static analysis of non-local transfers: a proposal
In-Reply-To: <20081124125821.jngv8pihfk40040g@www.staffmail.ed.ac.uk>
References: <20081120172753.vphjeep04kosco4k@www.staffmail.ed.ac.uk>
	<15e8b9d20811221546v32e3f028nc245a4be0b2ac0e9@mail.gmail.com>
	<9D638248-E45A-43AD-9490-554BFF9936BC@twistedbanana.demon.co.uk>
	<20081124125821.jngv8pihfk40040g@www.staffmail.ed.ac.uk>
Message-ID: <5F2139C8-1413-422E-9A9D-356F97929E25@twistedbanana.demon.co.uk>


On 24 Nov 2008, at 12:58, John Longley wrote:

> The concurrency examples are very interesting. Mark wrote:
>
>> I interpreted John's proposal as really being targeted at the 'late
>> transfer' rather than 'wrong thread' form, and as such the
>> restrictions shouldn't prevent the deliberate use of   
>> UnmatchedTransfer
>> for the creation of APIs like 'for eachConcurrently'.
>
> Yes, that's how I see it. One possible approach might be simply to  
> accept that my proposal doesn't exclude UnmatchedTransfer in  
> concurrent situations. This mightn't seem so bad if the idea is  
> that the clever concurrent stuff is written once-for-all by an  
> expert who knows what they're doing and then stuck in an API.

That's the idea.  The two uses of UnmatchedTransfer are very  
different in my view; the 'late transfer' form is almost certainly  
due to programmer error, whilst the 'wrong thread' form is actually  
useful to the implementation of these concurrent APIs, and provides a  
method to help do the right thing.  In some ways, it might be clearer  
if they were different classes, perhaps with the 'late' form  
manifested as an Error rather than an Exception.

So I think that excluding the 'wrong thread' forms is precisely  
what's required, rather than something that should be accepted as a  
limitation.

> The crucial lines of code for handling these transfers (taken from  
> Mark's blog) are
>
>       if (throwable instanceof UnmatchedTransfer)
>            throw ((UnmatchedTransfer) throwable).transfer();
>
> Although I don't understand exactly how this works, I imagine that  
> returning control to the correct thread is pretty much *always*  
> what one wants to do in practice.

Almost; to be pedantic, you'd either choose to attempt to restart the  
transfer in the correct thread (which may still fail due to it being  
a 'late transfer' of course), or you'd choose to discard it,  
typically because you may encounter many UnmatchedTransfers but can't  
restart more than one.

> So perhaps there is also a more sophisticated approach which would  
> build in this behaviour somehow - e.g. any concurrent code  
> involving closures is treated as implicitly having a handler like  
> the above, so that it's still guaranteed that the programmer never  
> sees an UnmatchedTransfer? (I haven't thought this through in detail.)

There probably is, but I found it works very well as it is now.  I'm  
sure my example implementation can be improved upon, but when  
implementing such APIs you'd usually also want to handle other  
Exceptions that are raised anyway by rethrowing them in the correct  
thread.  Once you've got the code written to deal with that (which  
accounts for almost all of the complexity), handling non-local  
transfers via UnmatchedTransfer just slots right in, with the only  
difference being that you call the transfer() method on it, rather  
than rethrowing.

> As a further point, I'd also like to describe how I think my  
> proposal can simplify the tracking of exceptions, but I'll save  
> this for another message.

Sounds intriguing; looking forward to hearing about it!

>
> Cheers, John
>
>
> -- 
> The University of Edinburgh is a charitable body, registered in
> Scotland, with registration number SC005336.
>
>

Best regards,

Mark