From steshaw at gmail.com Sat Apr 5 09:22:36 2008 From: steshaw at gmail.com (Steven Shaw) Date: Sun, 6 Apr 2008 02:22:36 +1000 Subject: stack manipulation APIs In-Reply-To: <47EAD978.1030506@jku.at> References: <47EAD978.1030506@jku.at> Message-ID: <1ba9d4a00804050922s4b58501bycb5d2ec495cba484@mail.gmail.com> On 27/03/2008, Lukas Stadler wrote: > This could lead to all sorts of harmful behavior, like exiting a > monitor twice, etc. Should this be possible, or will only a > restricted case be implemented? There would have to be a big red > sign (and possibly some kind of verifier) with all the things that > aren't allowed in such code. > I recently looked at the apache commons Javaflow library - they > implement storing the current stack state using only bytecode > instrumentation and a small and unintrusive runtime framework. (I > can write a short summary of how they're doing this if anyone is > interested.) I think that, as their implementation is inherently > safe, we could partly adopt its behaviour. Hi Lukas, I'd like to hear how the Javaflow works at the byte level and what they do with monitor entry/exit. I assume they implement multi-shot continuations? Cheers, Steve. From lukas.stadler at jku.at Mon Apr 7 04:20:58 2008 From: lukas.stadler at jku.at (Lukas Stadler) Date: Mon, 07 Apr 2008 13:20:58 +0200 Subject: stack manipulation APIs In-Reply-To: <1ba9d4a00804050922s4b58501bycb5d2ec495cba484@mail.gmail.com> References: <47EAD978.1030506@jku.at> <1ba9d4a00804050922s4b58501bycb5d2ec495cba484@mail.gmail.com> Message-ID: <47FA039A.9000104@jku.at> Steven Shaw wrote: > On 27/03/2008, Lukas Stadler wrote: > >> This could lead to all sorts of harmful behavior, like exiting a >> monitor twice, etc. Should this be possible, or will only a >> restricted case be implemented? There would have to be a big red >> sign (and possibly some kind of verifier) with all the things that >> aren't allowed in such code. >> I recently looked at the apache commons Javaflow library - they >> implement storing the current stack state using only bytecode >> instrumentation and a small and unintrusive runtime framework. (I >> can write a short summary of how they're doing this if anyone is >> interested.) I think that, as their implementation is inherently >> safe, we could partly adopt its behaviour. >> > > Hi Lukas, I'd like to hear how the Javaflow works at the byte level > and what they do with monitor entry/exit. I assume they implement > multi-shot continuations? > > Cheers, > Steve. > Hi! Yes, they implement multi-shot continuations. The javaflow library is a prime example of the power of bytecode-rewriting. It uses a thread-local object "StackRecorder" to control the behavior of the modified methods. A thread can be in one of three states: execute, capture or restore. Depending on this state the methods behave completely different: execute: normal execution with no special behavior. "finally" - clauses will only be executed in this state. capture: whenever a method call returns the current method stores its local variables to the StackRecorder and returns immediately. restore: when a method is called it restores its local variables from the StackRecorder and continues at the old location. This also makes it clear how javaflow handles monitor entry/exit - it doesn't. It implicitly releases and locks the monitors. Javaflow also doesn't provide any way to implement a dynamic-wind structure - but this could be implemented relatively easy. I assume it just wasn't needed. BTW I met with Peter Feigl last week who is an expert for Scheme implementations. While talking about continuations and Scheme implementations in general and it became clear that if (big if!) a scheme implementation supports fully-fledged continuations then it does so by copying the contents of the stack. Some implementations use a segmented stack approach that copies stack segments on-demand. This could also be achieved in the MLVM by inserting handlers into the stack return addresses (like deoptimization does) every x frames or every ~y bytes. Interestingly most uses of continuations can be replaced by simple non-local exits - and a good compiler can detect these most of the time by escape analysis! One thing that came up was how continuations behave if they are called in different threads (maybe at the same time?). Scheme itself doesn't define any behavior for this case. Another interesting problem are delimited continuations - part of the stack is turned into a function that returns a value. A good library of research papers for Scheme implementations: http://library.readscheme.org/page6.html Regards, Lukas From p.thamarai at gmail.com Thu Apr 10 07:43:30 2008 From: p.thamarai at gmail.com (Thamaraiselvan Poomalai) Date: Thu, 10 Apr 2008 20:13:30 +0530 Subject: MLVM Talk at Jax India 2008 Message-ID: <5def42a80804100743t779b4eb4lf12cf06803eef892@mail.gmail.com> FYI. Yesterday, I spoke at Jax India 2008, Bangalore. My session was on "The Da Vinci Machine - Multi Language Environment for Java Virtual Machine". There were around 70 to 90 participants. My session went fine with number of good questions from participants. Started my talk with "why we need other languages support on JVM" then pitched into "Introduction to porting new languages on JVM" and ended my talk with discussion on various "MLVM features (Light-weight byte code loading, light weight method objects, Dynamic Invocation, Symbolic freedom, Continuations and Tail recursion)". Here is the link to presentation slides. http://docs.google.com/Present?docid=dgmr9kck_448zkrssdt&skipauth=true#0 Thanks & Regards, selvan From John.Rose at Sun.COM Thu Apr 10 10:23:11 2008 From: John.Rose at Sun.COM (John Rose) Date: Thu, 10 Apr 2008 10:23:11 -0700 Subject: MLVM Talk at Jax India 2008 In-Reply-To: <5def42a80804100743t779b4eb4lf12cf06803eef892@mail.gmail.com> References: <5def42a80804100743t779b4eb4lf12cf06803eef892@mail.gmail.com> Message-ID: <942FFF24-8EDD-4C0C-903C-AC9B91079AED@sun.com> On Apr 10, 2008, at 7:43 AM, Thamaraiselvan Poomalai wrote: > http://docs.google.com/Present? > docid=dgmr9kck_448zkrssdt&skipauth=true#0 That's a good presentation! I'm glad there was a good session at Jax on the DVM project. I will put a link to it from the MLVM pages too. -- John From pdoubleya at gmail.com Thu Apr 10 10:28:23 2008 From: pdoubleya at gmail.com (Patrick Wright) Date: Thu, 10 Apr 2008 19:28:23 +0200 Subject: MLVM Talk at Jax India 2008 In-Reply-To: <5def42a80804100743t779b4eb4lf12cf06803eef892@mail.gmail.com> References: <5def42a80804100743t779b4eb4lf12cf06803eef892@mail.gmail.com> Message-ID: <64efa1ba0804101028p3a2ce963p853ff6d4f1760a69@mail.gmail.com> Selvan I just cross-posted the link to your presentation and blog over on the JVM Languages group [1]; I hope that's OK (I assumed it was, as you posted to a public list). If you're not already following the languages group, I imagine your experiments with the AnonymousClassLoader might be of interest to the JVM language implementors that follow that mailing list. Regards, and thanks for the presentation! Patrick [1] http://groups.google.com/group/jvm-languages From asouma3 at yahoo.fr Sat Apr 12 00:22:21 2008 From: asouma3 at yahoo.fr (asma belhadj) Date: Sat, 12 Apr 2008 07:22:21 +0000 (GMT) Subject: No subject Message-ID: <11654.26645.qm@web28106.mail.ukl.yahoo.com> asouma3 at yahoo.fr _____________________________________________________________________________ Envoyez avec Yahoo! Mail. Une boite mail plus intelligente http://mail.yahoo.fr -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/mlvm-dev/attachments/20080412/47bf674a/attachment.html From john.rose at sun.com Tue Apr 15 01:31:17 2008 From: john.rose at sun.com (john.rose at sun.com) Date: Tue, 15 Apr 2008 08:31:17 +0000 Subject: hg: mlvm/mlvm: initial load of empty patch repository Message-ID: <20080415083117.3DF4E27D50@hg.openjdk.java.net> Changeset: 1e070e27ffd7 Author: jrose Date: 2008-04-15 01:13 -0700 URL: http://hg.openjdk.java.net/mlvm/mlvm/rev/1e070e27ffd7 initial load of empty patch repository Reviewed-by: jrose + README.txt + make/current-release.sh + make/each-patch-repo.sh + make/link-patch-dirs.sh From john.rose at sun.com Tue Apr 15 01:33:04 2008 From: john.rose at sun.com (john.rose at sun.com) Date: Tue, 15 Apr 2008 08:33:04 +0000 Subject: hg: mlvm/mlvm/hotspot: 3 new changesets Message-ID: <20080415083305.0A3CC27D57@hg.openjdk.java.net> Changeset: 0c5d7f106a00 Author: jrose Date: 2008-04-15 01:16 -0700 URL: http://hg.openjdk.java.net/mlvm/mlvm/hotspot/rev/0c5d7f106a00 initial load of empty patch repository Reviewed-by: jrose + .hgignore + series Changeset: d4a18d7dbe5c Author: jrose Date: 2008-04-15 01:22 -0700 URL: http://hg.openjdk.java.net/mlvm/mlvm/hotspot/rev/d4a18d7dbe5c anonk: 6653858: dynamic languages need to be able to load anonymous classes Summary: first working revision Reviewed-by: jrose + anonk.patch + anonk.txt ! series Changeset: 716775c05842 Author: jrose Date: 2008-04-15 01:26 -0700 URL: http://hg.openjdk.java.net/mlvm/mlvm/hotspot/rev/716775c05842 anonk: 6655643: some dynamic languages need stack reification Summary: first working revision Reviewed-by: jrose + callcc.patch + callcc.txt ! series From john.rose at sun.com Tue Apr 15 01:33:30 2008 From: john.rose at sun.com (john.rose at sun.com) Date: Tue, 15 Apr 2008 08:33:30 +0000 Subject: hg: mlvm/mlvm/jdk: 3 new changesets Message-ID: <20080415083330.3A26627D5C@hg.openjdk.java.net> Changeset: a30739934482 Author: jrose Date: 2008-04-15 01:18 -0700 URL: http://hg.openjdk.java.net/mlvm/mlvm/jdk/rev/a30739934482 initial load of empty patch repository Reviewed-by: jrose + .hgignore + series Changeset: 0ac4312c0830 Author: jrose Date: 2008-04-15 01:22 -0700 URL: http://hg.openjdk.java.net/mlvm/mlvm/jdk/rev/0ac4312c0830 anonk: 6653858: dynamic languages need to be able to load anonymous classes Summary: first working revision Reviewed-by: jrose + anonk.patch ! series Changeset: 349442ac41d1 Author: jrose Date: 2008-04-15 01:26 -0700 URL: http://hg.openjdk.java.net/mlvm/mlvm/jdk/rev/349442ac41d1 anonk: 6655643: some dynamic languages need stack reification Summary: first working revision Reviewed-by: jrose + callcc.patch ! series From John.Rose at Sun.COM Tue Apr 15 02:45:34 2008 From: John.Rose at Sun.COM (John Rose) Date: Tue, 15 Apr 2008 02:45:34 -0700 Subject: welcome to the Da Vinci archipelago In-Reply-To: <20080415083330.3A26627D5C@hg.openjdk.java.net> References: <20080415083330.3A26627D5C@hg.openjdk.java.net> Message-ID: <94006E28-9323-4539-8B46-DEB3429EF3B1@sun.com> As you can see from the Mercurial changeset notifications that just went out, we have a repository for the Da Vinci Machine project. The repository does not hold a separate copy of the JVM. Rather, it is a Mercurial patch repository[0], a system of version-controlled patch files, each of which applies to the OpenJDK master source repository. At this point, there are three parts, three islands in what I trust will someday be a thriving cyber-polynesia: 0. The rules under which patches will co-exist are embodied in a README, assisted by a few shell scripts. 1. The anonk patch set, an implementation of VM-native anonymous classes, as previously blogged by me. 2. The callcc patch set, a first cut at VM-native continuations. All of these are works in progress. Please try out the setup, and if you find something that doesn't work right or is unclear, please propose a fix and send me a patch. At first, I'll apply the patches, with suitable Contributed-by lines.[1] To start, pull http://hg.openjdk.java.net/mlvm/mlvm and read the README. Or just browse: http://hg.openjdk.java.net/mlvm/mlvm/rev/ 1e070e27ffd7 The anonk patch set is a proposed fast path[2] for loading bytecodes, such as JRuby invokers. It currently needs a pre-JDK7 emulation package, which takes the bytes, applies the CP patches the hard way, chooses a strange name (instead of no name), probably creates a fresh class loader, and loads it up. The callcc patch set is a proof of concept of the copyStack/ restoreStack API. It is an early snapshot of something Lukas Stadler [3] is now working on. Another island which will soon rise out of the sea will be the home of method handles.[4] Unlike the first two projects, method handles are likely to be included in the first version of JSR 292, as an essential underpinning of invokedynamic. (Thus, it will be the first civilized island, since it will evolve under JCP oversight.) Best wishes, -- John Rose [0] http://hgbook.red-bean.com/hgbookch12.html , http:// www.selenic.com/mercurial/wiki/index.cgi/MqExtension [1] http://openjdk.java.net/guide/producingChangeset.html [2] http://blogs.sun.com/jrose/entry/anonymous_classes_in_the_vm [3] http://mail.openjdk.java.net/pipermail/mlvm-dev/2008-April/ 000037.html [4] http://openjdk.java.net/projects/mlvm/subprojects.html#MethodHandles From charles.nutter at sun.com Tue Apr 15 08:17:05 2008 From: charles.nutter at sun.com (Charles Oliver Nutter) Date: Tue, 15 Apr 2008 08:17:05 -0700 Subject: welcome to the Da Vinci archipelago In-Reply-To: <94006E28-9323-4539-8B46-DEB3429EF3B1@sun.com> References: <20080415083330.3A26627D5C@hg.openjdk.java.net> <94006E28-9323-4539-8B46-DEB3429EF3B1@sun.com> Message-ID: <4804C6F1.7040000@sun.com> John Rose wrote: > As you can see from the Mercurial changeset notifications that just > went out, we have a repository for the Da Vinci Machine project. > > The repository does not hold a separate copy of the JVM. Rather, it > is a Mercurial patch repository[0], a system of version-controlled > patch files, each of which applies to the OpenJDK master source > repository. > > At this point, there are three parts, three islands in what I trust > will someday be a thriving cyber-polynesia: > > 0. The rules under which patches will co-exist are embodied in a > README, assisted by a few shell scripts. > 1. The anonk patch set, an implementation of VM-native anonymous > classes, as previously blogged by me. > 2. The callcc patch set, a first cut at VM-native continuations. > > All of these are works in progress. Please try out the setup, and if > you find something that doesn't work right or is unclear, please > propose a fix and send me a patch. At first, I'll apply the patches, > with suitable Contributed-by lines.[1] > > To start, pull http://hg.openjdk.java.net/mlvm/mlvm and read the > README. Or just browse: http://hg.openjdk.java.net/mlvm/mlvm/rev/ > 1e070e27ffd7 > > The anonk patch set is a proposed fast path[2] for loading bytecodes, > such as JRuby invokers. It currently needs a pre-JDK7 emulation > package, which takes the bytes, applies the CP patches the hard way, > chooses a strange name (instead of no name), probably creates a fresh > class loader, and loads it up. > > The callcc patch set is a proof of concept of the copyStack/ > restoreStack API. It is an early snapshot of something Lukas Stadler > [3] is now working on. > > Another island which will soon rise out of the sea will be the home > of method handles.[4] Unlike the first two projects, method handles > are likely to be included in the first version of JSR 292, as an > essential underpinning of invokedynamic. (Thus, it will be the first > civilized island, since it will evolve under JCP oversight.) Excellent. I now also have a machine I should be able to get a build up and running on, so hopefully this week I can pull the bits together and give them a try. I'm also interested in what we can do to help languages on pre-JDK7 JVMs, since the vast majority of real users of JRuby (for example) could use some real help now. But I will look at your patches and try to formulate some strategy myself as well... - Charlie From John.Rose at Sun.COM Tue Apr 15 14:11:14 2008 From: John.Rose at Sun.COM (John Rose) Date: Tue, 15 Apr 2008 14:11:14 -0700 Subject: welcome to the Da Vinci archipelago In-Reply-To: <4804C6F1.7040000@sun.com> References: <20080415083330.3A26627D5C@hg.openjdk.java.net> <94006E28-9323-4539-8B46-DEB3429EF3B1@sun.com> <4804C6F1.7040000@sun.com> Message-ID: On Apr 15, 2008, at 8:17 AM, Charles Oliver Nutter wrote: > I'm also interested in what we can do to help languages on pre-JDK7 > JVMs, since the vast majority of real users of JRuby (for example) > could > use some real help now. But I will look at your patches and try to > formulate some strategy myself as well... This stuff would be merely academic if there were no story for how old JVMs play with the new features. So for me a big part of the puzzle of method handles has been to build in a backward- compatibility story that lets them be emulated on old platforms, with efficiency no less than (say) your JRuby invokers. Then it becomes feasible to have the dynlang backend perform a JVM version check at boot-up, and choose better vs. best implementations. And I hope a similar compatibility story can be found for the anonoymous class loader. Best, -- John -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/mlvm-dev/attachments/20080415/cf389e06/attachment.html From john.rose at sun.com Thu Apr 17 00:40:41 2008 From: john.rose at sun.com (john.rose at sun.com) Date: Thu, 17 Apr 2008 07:40:41 +0000 Subject: hg: mlvm/mlvm/hotspot: 2 new changesets Message-ID: <20080417074041.A916C27E59@hg.openjdk.java.net> Changeset: 283f2839e9b2 Author: jrose Date: 2008-04-17 00:38 -0700 URL: http://hg.openjdk.java.net/mlvm/mlvm/hotspot/rev/283f2839e9b2 anonk: builds on Linux, CP patching for doubles Contributed-by: R?mi Forax Reviewed-by: jrose ! anonk.patch Changeset: 971a1e442deb Author: jrose Date: 2008-04-17 00:39 -0700 URL: http://hg.openjdk.java.net/mlvm/mlvm/hotspot/rev/971a1e442deb callcc: builds in product mode Contributed-by: R?mi Forax Reviewed-by: jrose ! callcc.patch From francis.andre at easynet.fr Thu Apr 17 10:12:05 2008 From: francis.andre at easynet.fr (Francis ANDRE) Date: Thu, 17 Apr 2008 19:12:05 +0200 Subject: Da Vinci MLVM: Interest for an very old language running on an extended JVM In-Reply-To: <11D7CEAA-7DB0-4A2F-A5CE-43E3E562CE5D@Sun.COM> References: <47E55A26.2010203@univ-mlv.fr> <47F0C320.3030307@easynet.fr> <11D7CEAA-7DB0-4A2F-A5CE-43E3E562CE5D@Sun.COM> Message-ID: <20080417191205.zfnihqlmty1og4g0@webmail.easynet.fr> John > Interesting; I'm curious what are the "pain points", the features of > Cobol that cause the most difficulty when rendering them as JVM code. > The main "pain points" for cobol are the following: 1/ Record definition, allocation and member reference 2/ Packed decimal format and zoned decimal format 3/ Unsigned binary integers 4/ Exception handling on underflow/oveflow for all computational types 5/ Native support of collating sequence other than ASCII (EBCDIC as for example) 6/ Specific move and compare semantic with an implicit padding with space. I will expose you the point 1 first and the others in following mails. 1/ Problem of accessing unaligned data type member in a cobol record The cobol language provides the capability to define record much alike the struct contruction of the C language and also something called 'redefines' that is similar to the union of the C language. So in Cobol, one could define the user record as 1 user. 2 name pic x(5). // his name on 5 bytes 2 age pic s9(9) binary // his age as a 4 bytes integer Similar a struct in C would be typedef struct { char[5] name; int age; } user; but there is a major difference between the cobol user and he C user is that in cobol the age member is not aligned on its natural boundary. So in cobol, one have CobolOffset(user, name) = 0; CobolOffset(user, age) = 5; CobolSizeof(user) = 9; while most C compiler would produce (unless under user's specific allocation rules as allowed by the MSVC VC compiler) COffset(user, name) = 0; COffset(user, age) = 5; CSizeof(user) = 12; An ideal JVM like program for setting and getting value of the user.age in cobol would be 0: bipush 9 2: newarray byte 4: astore_1 5: aload_1 6: iconst_5 7: bipush 32 9: iastore LocalVariableTable: Start Length Slot Name Signature 0 11 0 args [Ljava/lang/String; 5 6 1 user [B But this code is invalid because the iastore instruction does not comply with the constraint on the target array type that according the spec should be an array of int. So currently, for storing of a simple int value in the cobol user.age, the generated code is equivalent to (considering big endianess): byte[] user; int value = 32; user = new byte[9]; user[5] = (byte)((value & 0xFF000000) >> 24); user[6] = (byte)((value & 0x00FF0000) >> 16); user[7] = (byte)((value & 0x0000FF00) >> 8); user[8] = (byte)((value & 0x000000FF) >> 0); which ends to the bytecodes 0: bipush 32 2: istore_2 3: bipush 9 5: newarray byte 7: astore_1 8: aload_1 9: iconst_5 10: iload_2 11: ldc #16; //int -16777216 13: iand 14: bipush 24 16: ishr 17: i2b 18: bastore 19: aload_1 20: bipush 6 22: iload_2 23: ldc #17; //int 16711680 25: iand 26: bipush 16 28: ishr 29: i2b 30: bastore 31: aload_1 32: bipush 7 34: iload_2 35: ldc #18; //int 65280 37: iand 38: bipush 8 40: ishr 41: i2b 42: bastore 43: aload_1 44: bipush 8 46: iload_2 47: sipush 255 50: iand 51: iconst_0 52: ishr 53: i2b 54: bastore 55: return Start Length Slot Name Signature 0 56 0 args [Ljava/lang/String; 8 48 1 user [B 3 53 2 value I So, as you can see, it is quite not performant, takes a lot of bytecode space for just a simple assignement. As the loading of a int gives the equivalent size of code, the equivalent cobol code to int = int + 1 is highly inefficient. 2/ Proposal My proposal is to relax the type array constraint on all xALOAD and all xASTORE JVM instructions so that the first list of bytecode as 0: bipush 9 2: newarray byte 4: astore_1 5: aload_1 6: iconst_5 7: bipush 32 9: iastore be valid. Upon executing the xALOAD or xASTORE instruction, the JVM should verify that the accessed bytes not be outside the target byte array and throw an OutOfArrayMemory exception otherwise. 3/ CobolVirtualExtension and CobolVirtualMachine One could think this extension in term of extending the current JVM and allocating a specific range of major/minor classes for VM supporting this extension. A Cobol class would be allowed to execute the relaxed xALOAD/xASTORE while a JVM would not. This insure that the Java security currently in place by the JVM would not be threaded while it would be relaxed for Cobol class only. Francis PS: I am preparing a more formal proposal based on the original JVM specs seconf edition, but that's the idea! John Rose a ?crit?: > On Mar 31, 2008, at 3:55 AM, Francis ANDRE wrote: > >> My primary business area is the modernization of legacy application >> running mainly on mainframes and mostly written in Cobol. I >> already developed a working prototype of a native cobol >> compiler&run time that generates standard JVM classes. But due to >> the nature of the Cobol language itself on one side, and due the >> specification of the JVM that is bundle with the Java language on >> the other side, there is a lot of inefficiency/penalties both in >> term of runtime design and performance of execution of the >> resulting compiled Cobol application. > > Interesting; I'm curious what are the "pain points", the features of > Cobol that cause the most difficulty when rendering them as JVM code. > >> That is why I am wondering if the Da Vinci Machine could be the >> place to extend the JVM to something like a CobolVirtualMachine. >> Would you be interested in such extensions? What would be your >> position regarding this project? > > I am most interested in experimenting with JVM extensions that will > help a variety of languages. It seems likely to me that Cobol will not > have unique difficulties with the JVM, but instead will shed light on > how to make the JVM into a multi-language substrate. > >> I presume you know already the figures about Cobol applications but >> just as a remainder: >> >> Arranga (2000) estimates between *18 billion* and *200 billion* >> lines of COBOL code are running production applications worldwide >> >> IMO, those figures could justify a interest by the JVM community >> (or Sun itself?) to get a industrial Cobol environment running as >> the Java one. Moreover, it is a real trend that most of responsible >> of large Cobol applications would like/want go to "Java" and >> providing a unique VM that could run both a Java class with the >> same security as the standard JVM and a Cobol class would IMHO have >> a real appealing in term of business. > > That seems likely, for some Cobol users. Although the same legacy > constraints that keep some people on Cobol may also prevent them from > considering JVM technology. They may have a very low tolerance for > change and risk. The question about the giant Cobol installed base is, > how often do those users change their Cobol implementations (while > keeping their old sources)? If their Cobol is not portable to start > with, it seems a lost cause to convert it onto a new platform. > >> May be you are aware of tools that translate directly Cobol to >> Java: yes they work... technically but in the reality, it does not >> make it because the produced Java code is quite far from the >> original code and thus unmaintainable (we are speaking there of >> applications between 500 Kilos LOC and 10 Millions LOC or more). > > You could compile Cobol directly to bytecodes, and not compromise with > a Java rendering. Would the bytecode architecture force distortions on > the Cobol program structure? Could you compile a Cobol program into a > package full of interlinked classes? > > For examples of distortions caused by misfit between source language > and bytecode architecture, which I call "pain points" for language > implementors, see > http://openjdk.java.net/projects/mlvm/pdf/LangNet20080128.pdf . > > I'm looking forward to hearing more on this subject! > > Best wishes, > -- John From francis.andre at easynet.fr Fri Apr 18 03:34:57 2008 From: francis.andre at easynet.fr (Francis ANDRE) Date: Fri, 18 Apr 2008 12:34:57 +0200 Subject: Da Vinci MLVM: Interest for an very old language running on an extended JVM In-Reply-To: References: <47E55A26.2010203@univ-mlv.fr> <47F0C320.3030307@easynet.fr> <11D7CEAA-7DB0-4A2F-A5CE-43E3E562CE5D@Sun.COM> <20080417191205.zfnihqlmty1og4g0@webmail.easynet.fr> Message-ID: <20080418123457.ymekwt51j7u8sswo@webmail.easynet.fr> >> byte[] user; >> int value = 32; >> >> user = new byte[9]; >> user[5] = (byte)((value & 0xFF000000) >> 24); >> user[6] = (byte)((value & 0x00FF0000) >> 16); >> user[7] = (byte)((value & 0x0000FF00) >> 8); >> user[8] = (byte)((value & 0x000000FF) >> 0); > > This does lead to big bytecodes. You can compress the bytecodes by > using a subroutine: > class PackedArray { ... > int getInt(byte[] array, int offset) { ... } > } > Yes, I am already using a subroutine that LOAD & STORE integer of 1, 2 and 4 bytes long with the same signature as yours(byte[], offset). But I wanted to emphasis on the performance issue without prompting the subroutines. > Corresponding code: >> 0: bipush 9 >> 2: newarray byte >> 4: astore_1 >> 5: aload_1 >> 6: iconst_5 >> 7: bipush 32 >> 9: invokestatic #PackedArray.setInt([BII)V > >> 2/ Proposal >> My proposal is to relax the type array constraint on all xALOAD and >> all xASTORE JVM instructions > The performance story is better too, since whether it's a bytecode or > subroutine, you may have to wait for the compiler team to cook up the > desired optimizations, but it's usually a shorter wait for the > subroutine. And now that we are open source, you can submit your own > patch for the desired compiler intrinsic. That will always be easier > than submitting a patch for a new bytecode. I am not requesting new opcodes but just relaxing the array type contraint of the already existing xALOAD & xASTORE when the VM run a Cobol or C class. Moreover, the optimization you mention works only with the JIT generation while using a relaxed identical opcode will work also in interpreted mode (which is a important point for the debugging tool to align the generated code with the original source while getting an optimal performance). > > Packed data structures are an interesting pain point. Perhaps they > need not be packed? Could a Cobol implementation use native JVM > packing order (mapping Cobol structs to natural JVM instances), but > providing on-demand views which simulate the needed byte order, when > required by aliasing? I'm guessing that most Cobol programs would not > rely on byte ordering, most of the time. > The problem for packed/zoned data is identical as for integer: just inefficiency and performance. As for integer, there is no problem for getting the rigth semantic with a metadata of the packed/zoned data and subroutines with signature like (byte[], offset, precision, scale and sign) but the assignement/retrieval of value is done on a byte to byte basis which is highly inefficient while a interpreted/compiled opcode could do it efficiently. More on packed/zoned cobol data in a following mail... Francis From forax at univ-mlv.fr Fri Apr 18 09:47:56 2008 From: forax at univ-mlv.fr (=?ISO-8859-1?Q?R=E9mi_Forax?=) Date: Fri, 18 Apr 2008 18:47:56 +0200 Subject: Patch review: fix infinite loop in anonymousClassLoader.checkHostClass Message-ID: <4808D0BC.8010702@univ-mlv.fr> Hi all. There is a potential infite loop in the current code of the AnonymousClassLoader. In checkHostClass, the caller and the callee need to be normalized to their top level classes, but outer is not updated in the loop. The attachment is a test case to reproduce the bug. I propose the following patch: diff --git a/src/share/classes/java/dyn/AnonymousClassLoader.java b/src/share/classes/java/dyn/AnonymousClassLoader.java --- a/src/share/classes/java/dyn/AnonymousClassLoader.java +++ b/src/share/classes/java/dyn/AnonymousClassLoader.java @@ -275,6 +275,14 @@ class AnonymousClassLoader { this.classFileCP = classFileCP; } + private static Class getTopLevelClass(Class clazz) { + for(Class outer = clazz.getDeclaringClass(); outer != null; + outer = outer.getDeclaringClass()) { + clazz = outer; + } + return clazz; + } + private static Class checkHostClass(Class hostClass) { // called only from the constructor // does a context-sensitive check on caller class @@ -299,10 +307,8 @@ class AnonymousClassLoader { return hostClass; // normalize caller and callee to their top-level classes: - for (Class outer = caller.getDeclaringClass(); outer != null; - caller = outer) { } - for (Class outer = callee.getDeclaringClass(); outer != null; - callee = outer) { } + caller = getTopLevelClass(caller); + callee = getTopLevelClass(callee); if (caller == callee) return caller; -------------- next part -------------- A non-text attachment was scrubbed... Name: AnonymousCheckHostClassInfiniteLoop.java Type: text/x-java Size: 327 bytes Desc: not available Url : http://mail.openjdk.java.net/pipermail/mlvm-dev/attachments/20080418/07b65863/attachment.bin From John.Rose at Sun.COM Fri Apr 18 14:13:39 2008 From: John.Rose at Sun.COM (John Rose) Date: Fri, 18 Apr 2008 14:13:39 -0700 Subject: Patch review: fix infinite loop in anonymousClassLoader.checkHostClass In-Reply-To: <4808D0BC.8010702@univ-mlv.fr> References: <4808D0BC.8010702@univ-mlv.fr> Message-ID: <4A36F8D5-2466-4BD9-BB11-8C252E077904@sun.com> Good fix, Remi. Thanks! Folks, I'm looking forward to a long collaboration with lots of MLVM patches. Let's use the message format described here: http://openjdk.java.net/contribute/ At this early point, since sub-projects like anonk are tracked under omnibus bugs, you do not need to include a bug number with your patch. Mention the sub-project identifier (anonk) instead of the bug number. Thus, a formal patch submission for Remi's bug would have this subject line: Subject: [PATCH] anonk: fix infinite loop in anonymousClassLoader.checkHostClass For simple fixes, the formal patch submission and the review request be accomplished in the same Email. For complex fixes or works in progress, you might want to ask for review first: Subject: Patch review: anonk: fix infinite loop in anonymousClassLoader.checkHostClass If the patch is large, consider submitting it in the form of a link to a webrev, like this: http://webrev.invokedynamic.info/jrose/6610906/ At present, you are on your own finding web hosting for the webrev. Sorry! I believe this is in the works behind a bunch of other OpenJDK infrastructure. A version of the webrev script is available at: http://blogs.sun.com/jcc/entry/webrev_for_openjdk_a_code There is much more information about being an OpenJDK developer here: http://openjdk.java.net/guide/ Finally, if you haven't signed the Sun Contributor Agreement, you'll need to do that before the project can incorporate your work. Best wishes, -- John On Apr 18, 2008, at 9:47 AM, R?mi Forax wrote: > There is a potential infite loop in the current code of the > AnonymousClassLoader. > In checkHostClass, the caller and the callee need to be normalized > to their top level classes, > but outer is not updated in the loop. > The attachment is a test case to reproduce the bug. > > I propose the following patch: From forax at univ-mlv.fr Sat Apr 19 12:09:20 2008 From: forax at univ-mlv.fr (=?ISO-8859-1?Q?R=E9mi_Forax?=) Date: Sat, 19 Apr 2008 21:09:20 +0200 Subject: AnonymousClassLoader overhaul Message-ID: <480A4360.7090704@univ-mlv.fr> Hi all, there are several changes, i would like to apply on AnonymousClassLoader: - make a clean separation between the classloader, the constant pool parser and the patch array. - avoid to iterate over all constant pool entries when applying a patch on the same base class (remove indexOf() in makePatchArray) - try to make it thread safe - allocate the bytecode array using ByteBuffer instead of byte[] (not sure it's possible). - generify it (Class => Class) i will try to submit a patch soon. R?mi From John.Rose at Sun.COM Tue Apr 22 19:02:59 2008 From: John.Rose at Sun.COM (John Rose) Date: Tue, 22 Apr 2008 19:02:59 -0700 Subject: stack manipulation update In-Reply-To: <480E2CF2.3080307@jku.at> References: <47FCE6FC.6000501@jku.at> <480E2CF2.3080307@jku.at> Message-ID: [CC-ing mlvm-dev with my reply, since your note is of great general interest. -- John] On Apr 22, 2008, at 11:22 AM, Lukas Stadler wrote: > Hi! >> If you have a login for that wiki, send me your login identifier, >> and I'll add you to the author list. ('ChristianWimmer' already >> has author access, and every JVM developer should have it, though >> I have to enter the names one at a time for now.) > I've created a wikis.sun.com user named "LukasStadler". It would be > great if you could add that account to the author list. Done this morning... Please feel free to create or edit pages in HotSpotInternals, especially notes on implementation as you learn your way around. > Building: Maybe it should be mentioned that the mercurial mq > extension needs to be enabled. It took me a while to figure out > where the "qselect" command comes from... Thanks; I'll fix it in the README. >> Are the continuations really bounded? (Looks like the framework's >> main method, as reflected, serves as a boundary token.) What >> happens when you try to return from a bounded continuation? > Good question - right now it is silently assumed that the > continuation is reinstated at a correct position (in this case the > stackframes are removed up to and including the main method, and > then the main method is the first stackframe to be installed again) I suppose when a continuation is resumed all frames up to its bound should be overwritten. >> I thought there was some serialization story there. The source >> code mentions a need to setAccessible(true). (Another potential >> security challenge, out of many in this work.) > Hm... I don't think that there's a way - Method doesn't implement > Serializable and there's no special code in the > ObjectOutputStream... But it's not that big a problem anyway, I > just don't see why Class is serializable and Method is not. Here is a guess: There's a security hole if you allow arbitrary methods to be created from raw bytes with their "accessible" bit set to true. But the fix would be to do the same security checks that setAccessible does in the first place. >>> - Sounds like a reorganization of the code may allow nearly >>> complete reuse of the deopt blob >> I hope so. There are not that many degrees of freedom in >> interpreter frames, so it should not need fundamentally new >> techniques. > I added an entry point into the deopt_blob that can be passed an > UnrollInfo object (and that doesn't call fetch_unroll_info). It's > only ~10 new lines of code. One problem that I see is that it may > be necessary to remove the stackframe of an inlined method from a > compiled frame to reach the boundary method, which means that the > topmost frame may need to be deoptimized. This might prove, uhm, > interesting. (because it shouldn't return to the Interpreter after > it's done) That's one reason I made the boundary method be a fixed native method doCopyStackContext. All those edge cases are easier to control that way. I don't see (yet) that it's a significant loss of generality. > I also added code to vframeArray and Deoptimization that creates a > vframeArray from a StackSerializer. So far it works like a charm, > although I'm ignoring monitors for now. (I don't really know how to > create or acquire the BasicLock objects for monitors, > ObjectSynchronizer::inflate?) I don't remember off hand, but I'd look at how the current deopt. or OSR code does the trick. > I have created Java classes for the stack state (see the attached > files) and I've written native code that performs the stack copy > operation into these objects. I'd like to work with this > representation for the time being. I don't think that this is much > less efficient than the StackSerializer (apart from the obviously > larger memory consumption). Currently I'm writing code to create > the vframeArray from this representation and I'll send you an > overview of all the changes I made as soon as I'm finished with that. You probably need more random access than the serialized form can give you. Hmmm... The array-based representation is fine for prototyping but for production use is too transparent to optimize or secure. It's not a short term problem. What we need is a sort of union-type list abstraction that can equally easily store references and primitives (at least ints; would like IJFD), but without exposing its internals too much. > interface MachineWordList extends List { > int size(); > byte type(int i); > Object get(int i); > int getInt(int i); > long getLong(int i); > // setters? > void set(int i, Object x); > void setInt(int i, int x); > // arrays? > byte[] typeArray(); > Object[] toArray(); > int[] toIntArray(); > } But this is a cleanup; you don't have to turn aside from the main work. > >> Generators: That's an important use case for "small scale" >> continuations. Open question: Does the existing design (a >> serialized blob of data) give enough visibility to the compiler to >> allow it to optimize across continuation invocations (thus >> inlining generator patterns)? Probably not, unless the unsafe >> copyStack/resumeStack structures are made more stateless and >> immutable, so they can be examined by the compiler. > An ImmutableContinuation object, which allows no access to its > contents whatsoever, would fit this purpose, I think. The compiler > could even detect cases where the continuation is used as a > nonlocal return in inlined methods and create no continuation at > all... Yes, that's the end-to-end optimization that we want for things like generators. Great work! -- John > package javax.stack; > > import java.lang.reflect.Method; > > /** > * Contains the defining state of one stack frame activation. Read > access provides access to private data, unverified > * write access may allow code to escape security constraints, > violate JVM invariants or even break the JVM. > * > * Data within this object is somewhat packed: the values and > objects of local variables and expressions are > * sequentially written into valueSlots and objectSlots. After that > the monitors are written into objectSlots. > */ > public class Stackframe { > private Method method; > private int bci; > > private boolean[] localIsObject; > private boolean[] expressionIsObject; > private int[] valueSlots; > private Object[] objectSlots; > > public Method getMethod() { > return method; > } > > public void setMethod(Method method) { > this.method = method; > } > > public int getBci() { > return bci; > } > > public void setBci(int bci) { > this.bci = bci; > } > > public int getLocalCount() { > return localIsObject == null ? 0 : localIsObject.length; > } > > public int getExpressionCount() { > return expressionIsObject == null ? 0 : expressionIsObject.length; > } > > public int getMonitorCount() { > return (valueSlots == null ? 0 : valueSlots.length) + > (objectSlots == null ? 0 : objectSlots.length) > - getLocalCount() - getExpressionCount(); > } > > public int getValueSlot(int index) { > return valueSlots[index]; > } > > public Object getObjectSlot(int index) { > return objectSlots[index]; > } > > public void setValueSlots(int[] valueSlots) { > this.valueSlots = valueSlots; > } > > public void setObjectSlots(Object[] objectSlots) { > this.objectSlots = objectSlots; > } > > public boolean localIsObject(int index) { > return localIsObject[index]; > } > > public boolean expressionIsObject(int index) { > return expressionIsObject[index]; > } > > public void setLocalIsObject(boolean[] localIsObject) { > this.localIsObject = localIsObject; > } > > public void setExpressionIsObject(boolean[] expressionIsObject) { > this.expressionIsObject = expressionIsObject; > } > > } > package javax.stack; > > import java.lang.reflect.Method; > > /** > * Allows to store stack frames so that they can be inspected, > modified and reinstated. Any access beyond querying the > * stack frame count and the methods poses a potential security risk. > */ > public class Continuation { > private Stackframe[] stackframes; > > public int getStackframeCount() { > return stackframes == null ? 0 : stackframes.length; > } > > public Method getMethod(int index) { > return stackframes[index].getMethod(); > } > > public Stackframe getStackframe(int index) { > // TODO: security check ... > return stackframes[index]; > } > > public Stackframe[] getStackframes() { > // TODO: security check ... > return stackframes; > } > > public void setStackframes(Stackframe[] stackframes) { > // TODO: security check > this.stackframes = stackframes; > } > } -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/mlvm-dev/attachments/20080422/caa4b4ff/attachment-0001.html From John.Rose at Sun.COM Wed Apr 23 00:30:37 2008 From: John.Rose at Sun.COM (John Rose) Date: Wed, 23 Apr 2008 00:30:37 -0700 Subject: Da Vinci MLVM: Interest for an very old language running on an extended JVM In-Reply-To: <20080418123457.ymekwt51j7u8sswo@webmail.easynet.fr> References: <47E55A26.2010203@univ-mlv.fr> <47F0C320.3030307@easynet.fr> <11D7CEAA-7DB0-4A2F-A5CE-43E3E562CE5D@Sun.COM> <20080417191205.zfnihqlmty1og4g0@webmail.easynet.fr> <20080418123457.ymekwt51j7u8sswo@webmail.easynet.fr> Message-ID: On Apr 18, 2008, at 3:34 AM, Francis ANDRE wrote: >> The performance story is better too, since whether it's a bytecode or >> subroutine, you may have to wait for the compiler team to cook up the >> desired optimizations, but it's usually a shorter wait for the >> subroutine. And now that we are open source, you can submit your own >> patch for the desired compiler intrinsic. That will always be easier >> than submitting a patch for a new bytecode. > > I am not requesting new opcodes but just relaxing the array type > contraint > of the already existing xALOAD & xASTORE when the VM run a Cobol or > C class. It really amounts to the same complexity as a new opcode, since every tool or JVM phase that looks at opcodes may need to know the new usages. (And in this case, we probably already have intrinsic methods which compile to the code you want. See below.) > Moreover, the optimization you mention works only with the JIT > generation > while using a relaxed identical opcode will work also in > interpreted mode > (which is a important point for the debugging tool to align the > generated > code with the original source while getting an optimal performance). It seems to me just as easy to align source code to a BCI which contains an invokestatic as a BCI which contains an iaload. > The problem for packed/zoned data is identical as for integer: just > inefficiency > and performance. With very few exceptions, the fastest way to get performance on some operation is first to name it (as a method), then have the compiler treat it as an intrinsic. Here is a list of all the intrinsics in HotSpot (start 50% down around line 416): http://hg.openjdk.java.net/jdk7/jdk7/hotspot/file/8b0b3490194f/src/ share/vm/classfile/vmSymbols.hpp Some of these have in the past been requested as special bytecodes, but they all work better as methods. By the way, aligned (or on Intel unaligned little-endian) accesses can be performed in VM-specific code by operations like Unsafe.getInt. For example, those are used for fast type-punning access in NIO buffers. A NIO byte buffer can be punned (in a type- safe way) with an int buffer, to get the kind of accesses you want. It is implemented with intrinsics in Unsafe. There is no need for special bytecodes. > As for integer, there is no problem for getting the rigth semantic > with a metadata of the packed/zoned data and subroutines with > signature like (byte[], offset, precision, scale and sign) but the > assignement/retrieval of value is done on a byte to byte basis > which is highly inefficient while a interpreted/compiled opcode > could do it efficiently. And so also do compiled intrinsics provide a way to do these things efficiently. Think of bytecodes as low-level IR, which is very expensive to extend, and intrinsic method calls as a higher-level extensible part of the IR, which is it easy to extend compatibly. If there really is a need for interpreted performance (which is in practice very rarely demonstrated) the interpreter has ways to treat methods as intrinsics also. This is almost never done. See enum MethodKind in: http://hg.openjdk.java.net/jdk7/jdk7/hotspot/file/8b0b3490194f/src/ share/vm/interpreter/abstractInterpreter.hpp > More on packed/zoned cobol data in a following mail... Thank you! -- John From forax at univ-mlv.fr Wed Apr 23 04:15:54 2008 From: forax at univ-mlv.fr (=?ISO-8859-1?Q?R=E9mi_Forax?=) Date: Wed, 23 Apr 2008 13:15:54 +0200 Subject: AnonymousClassLoader overhaul Message-ID: <480F1A6A.4020302@univ-mlv.fr> John Rose a ?crit : > On Apr 19, 2008, at 12:09 PM, R?mi Forax wrote: > >> Hi all, >> there are several changes, i would like to apply on >> AnonymousClassLoader: >> - make a clean separation between the classloader, the constant pool >> parser and >> the patch array. >> - avoid to iterate over all constant pool entries when applying a patch >> on the same base class (remove indexOf() in makePatchArray) >> - try to make it thread safe >> - allocate the bytecode array using ByteBuffer instead of byte[] >> (not sure it's possible). >> - generify it (Class => Class) > > Those are all good moves on the road from demo to product... Thanks. The source code is available here: http://www-igm.univ-mlv.fr/~forax/tmp/davinci-anonk.zip I've sent a previous message monday to the mlvm list but it seems it was rejected because its size was too large (codes as attachment). So i re-sent it now with a link. The zip contains: - AnonymousClassLoader: load class (or a patch) as inner-class of a host class. - ClassPatch: a class and its patch array, it is created from a ConstantPoolParser - ConstantPoolParser: parse the constant pool, call a visitor for each entry - ConstantPoolVisitor: called by the parser, can modified a patch using of of it's method put*. AnonymousTest is a test Why a visitor ? it avoid to reify the constant pool, i.e to create one object by entry, this approach is not new, ASM use a similar trick. After hesitation, i've decided to surface constant pool index instead of using opaque handle, so at Java level, it's possible to try to change a constant pool entry by an incompatible one. ClassPatch is now the only class that is mutable, the other are non mutable. I'm not a native english speaker, so doc comments are written in common (+french) english, it will cool if someone can fix them :) > > When working with the CP parser, consider that we'll be wanting a > Java-only emulation of the loader. For that we'll need to edit class > files as well as read them. Note that ldc of non-string object will > require bytecode editing, not just CP editing. Forward compatibility is cool but i'm not sure emulating an AnonymousClassLoader is a good idea. I think it's not up to language implementors to simuate their features because they see the big picture. Furthermore, bytecode editing is in my opinion too hard to not use library like ASM or BCEL. BTW, i now understand why you want MethodHandle to be impelemented by plain Java class. > > (Bytecode editing of extended invoke sites might also be a plausible > way to manage extensions like method handles... But it's probably > easier to tell the language backend to sniff the JVM version and alter > its code generation in the first place. Except for a few cases like > static bytecode compilation.) yes > > -- John > R?mi From John.Rose at Sun.COM Wed Apr 23 10:31:40 2008 From: John.Rose at Sun.COM (John Rose) Date: Wed, 23 Apr 2008 10:31:40 -0700 Subject: AnonymousClassLoader overhaul In-Reply-To: <480F1A6A.4020302@univ-mlv.fr> References: <480F1A6A.4020302@univ-mlv.fr> Message-ID: On Apr 23, 2008, at 4:15 AM, R?mi Forax wrote: > The source code is available here: > http://www-igm.univ-mlv.fr/~forax/tmp/davinci-anonk.zip > > I've sent a previous message monday to the mlvm list but it seems it > was rejected because its size was too large (codes as attachment). > So i re-sent it now with a link. Thanks, Remi. I've grabbed the file. I'll read it through ASAP. The mail robot sent me a moderation request, but when I tried to approve it, the robot got buggy and pretended it didn't know anything. It looks like mailing large volumes of source or patches will be unreliable. Other projects must have this sort of problem; how do they work around it? For now, I have raised the limit to 200kb, which should serve us for small patches. I hesitate to remove the limit, since big emails will stress people's inboxes or mail servers, and in extreme cases could get our server blacklisted. (Should I worry about that?) -- John From forax at univ-mlv.fr Wed Apr 23 13:58:13 2008 From: forax at univ-mlv.fr (=?ISO-8859-1?Q?R=E9mi_Forax?=) Date: Wed, 23 Apr 2008 22:58:13 +0200 Subject: AnonymousClassLoader overhaul In-Reply-To: References: <480F1A6A.4020302@univ-mlv.fr> Message-ID: <480FA2E5.2000500@univ-mlv.fr> John Rose a ?crit : > On Apr 23, 2008, at 4:15 AM, R?mi Forax wrote: > > >> The source code is available here: >> http://www-igm.univ-mlv.fr/~forax/tmp/davinci-anonk.zip >> >> I've sent a previous message monday to the mlvm list but it seems it >> was rejected because its size was too large (codes as attachment). >> So i re-sent it now with a link. >> > > Thanks, Remi. I've grabbed the file. I'll read it through ASAP. > Ok, let me know waht you want to change. I will update the wiki when you will be ok with the implementation. > The mail robot sent me a moderation request, but when I tried to > approve it, the robot got buggy and pretended it didn't know anything. > Robots get better every day. They now really act like human :) > It looks like mailing large volumes of source or patches will be > unreliable. Other projects must have this sort of problem; how do > they work around it? > At FOSDEM, Mark Reinhold says that perhaps bug parade (or at least some parts) will be replaced by bugzilla. Bugzilla allows anyone registered to send patches. > For now, I have raised the limit to 200kb, which should serve us for > small patches. I hesitate to remove the limit, since big emails will > stress people's inboxes or mail servers, and in extreme cases could > get our server blacklisted. (Should I worry about that?) > I think 200k is enough. > -- John > R?mi From forax at univ-mlv.fr Thu Apr 24 02:26:34 2008 From: forax at univ-mlv.fr (=?ISO-8859-1?Q?R=E9mi_Forax?=) Date: Thu, 24 Apr 2008 11:26:34 +0200 Subject: AnonymousClassLoader and ClassLoader Message-ID: <4810524A.40504@univ-mlv.fr> Yesterday evening, I've tried to replace the application classloader (the one used to load the application) by one that use an anymous class loader to load the class. And i've tested by launching SwingSet2. And it doesn't work because UI program like SwingSet lookup their resources like image, icons etc. using getResource()/getResourceAsStream(). getResource() makes the assumption that if getClassLoader() returns null the class is loaded by the bootstrap classloader. In my opinion, getResource() is not the only method that makes this assumption so an anonymous class should be able to specify an external classloader when defined i.e. Unsafe.defineAnonymousClass should take a classloader as argument. R?mi From forax at univ-mlv.fr Thu Apr 24 04:43:42 2008 From: forax at univ-mlv.fr (=?ISO-8859-1?Q?R=E9mi_Forax?=) Date: Thu, 24 Apr 2008 13:43:42 +0200 Subject: AnonymousClassLoader and ClassLoader In-Reply-To: <4810524A.40504@univ-mlv.fr> References: <4810524A.40504@univ-mlv.fr> Message-ID: <4810726E.6020003@univ-mlv.fr> R?mi Forax a ?crit : > Yesterday evening, I've tried to replace the application classloader > (the one used to load the application) by one that use an anymous class > loader > to load the class. And i've tested by launching SwingSet2. > > And it doesn't work because UI program like SwingSet lookup their > resources > like image, icons etc. using getResource()/getResourceAsStream(). > getResource() makes the assumption that if getClassLoader() returns null > the class is loaded by the bootstrap classloader. > > In my opinion, getResource() is not the only method that makes this > assumption > so an anonymous class should be able to specify an external classloader > when defined i.e. Unsafe.defineAnonymousClass should take > a classloader as argument. > > R?mi > I answer to myself, it's only in the case the host-class is null, otherwise the classloader used is the one of the host-class. What is exactly the meaning of a null host-class ? R?mi From charles.nutter at sun.com Thu Apr 24 07:00:36 2008 From: charles.nutter at sun.com (Charles Oliver Nutter) Date: Thu, 24 Apr 2008 09:00:36 -0500 Subject: Got a build up and going now... Message-ID: <48109284.3020908@sun.com> Ok, finally got around to getting OpenJDK + MLVM patches up and building. Today I'm planning to look at: - JRuby using anonymous classloading Ideally this will just be a matter of swapping in a different classloader, but then would come the removal of a lot of hassle we currently have to keep names unique and ensure they're GCable. I'll have to come up with some memory metrics to show that it's still working. - Ruby continuation support using copyStack/resumeStack I saw this was in the patch repo also, and can't resist playing with it. In order for us to support Ruby's continuations, I'd also need to wire up preservation of the Ruby stack, but that's just a bit of grunt work. Will update with status as I get anything working. - Charlie From charles.nutter at sun.com Thu Apr 24 07:38:15 2008 From: charles.nutter at sun.com (Charles Oliver Nutter) Date: Thu, 24 Apr 2008 09:38:15 -0500 Subject: welcome to the Da Vinci archipelago In-Reply-To: References: <20080415083330.3A26627D5C@hg.openjdk.java.net> <94006E28-9323-4539-8B46-DEB3429EF3B1@sun.com> <4804C6F1.7040000@sun.com> Message-ID: <48109B57.3040100@sun.com> John Rose wrote: > On Apr 15, 2008, at 8:17 AM, Charles Oliver Nutter wrote: >> >> I'm also interested in what we can do to help languages on pre-JDK7 >> >> JVMs, since the vast majority of real users of JRuby (for example) could >> >> use some real help now. But I will look at your patches and try to >> >> formulate some strategy myself as well... >> > > This stuff would be merely academic if there were no story for how old > JVMs play with the new features. So for me a big part of the puzzle of > method handles has been to build in a backward-compatibility story that > lets them be emulated on old platforms, with efficiency no less than > (say) your JRuby invokers. > > Then it becomes feasible to have the dynlang backend perform a JVM > version check at boot-up, and choose better vs. best implementations. > > And I hope a similar compatibility story can be found for the anonoymous > class loader. The general concept I've had for backward-compatible invokers would be to provide a way for consumers to plug in their own argument and exception-mapping logic. So where JRuby, say, takes the input arguments to a given "call" on DynamicMethod and translates them into pre and post-call logic, exception-handling and finally wrappers, and the eventual construction of an argument list for the target method, the more generic and generally-useful version would allow this behavior to be customized, either by passing in an implementation of "argument mapping", etc, or by providing template code that could be rewritten by the handle generator. This same logic is going to be needed for most languages that have non-standard (i.e. non-Java) mechanisms for collecting methods together into a type-representing bundle, and with so many pre-JDK7 installations out there, having a backward compatible solution is a must. And yay, I think I finally got the hg patchset to apply. Onward to experimentation! - Charlie From charles.nutter at sun.com Thu Apr 24 07:51:54 2008 From: charles.nutter at sun.com (Charles Oliver Nutter) Date: Thu, 24 Apr 2008 09:51:54 -0500 Subject: AnonymousClassLoader and ClassLoader In-Reply-To: <4810524A.40504@univ-mlv.fr> References: <4810524A.40504@univ-mlv.fr> Message-ID: <48109E8A.1090507@sun.com> R?mi Forax wrote: > Yesterday evening, I've tried to replace the application classloader > (the one used to load the application) by one that use an anymous class > loader > to load the class. And i've tested by launching SwingSet2. > > And it doesn't work because UI program like SwingSet lookup their > resources > like image, icons etc. using getResource()/getResourceAsStream(). > getResource() makes the assumption that if getClassLoader() returns null > the class is loaded by the bootstrap classloader. > > In my opinion, getResource() is not the only method that makes this > assumption > so an anonymous class should be able to specify an external classloader > when defined i.e. Unsafe.defineAnonymousClass should take > a classloader as argument. An interesting thought. I can think of a few places I expect to be able to call getClassLoader on a target class I've loaded at runtime, but that's generally because I want to associate other classes dependent on the target class (i.e. method handles for a freshly-loaded compiled Ruby script) with the same throwaway classloader. Given the looser nature of the anonymous classloader, that particular scenario would go away. I'd just always grab my trust anonymous classloader and throw classes at it, knowing they'd GC when they were no longer in use. I don't think the anonymous classloader is going to be something you can just start using lightly, since assumptions existing classes make about classloading are sure to get in the way. - Charlie From francis.andre at easynet.fr Thu Apr 24 08:13:47 2008 From: francis.andre at easynet.fr (Francis ANDRE) Date: Thu, 24 Apr 2008 17:13:47 +0200 Subject: Da Vinci MLVM: Interest for an very old language running on an extended JVM In-Reply-To: References: <47E55A26.2010203@univ-mlv.fr> <47F0C320.3030307@easynet.fr> <11D7CEAA-7DB0-4A2F-A5CE-43E3E562CE5D@Sun.COM> <20080417191205.zfnihqlmty1og4g0@webmail.easynet.fr> <20080418123457.ymekwt51j7u8sswo@webmail.easynet.fr> Message-ID: <4810A3AB.8070506@easynet.fr> >> I am not requesting new opcodes but just relaxing the array type contraint >> of the already existing xALOAD & xASTORE when the VM run a Cobol or C class. > It really amounts to the same complexity as a new opcode, since every > tool or JVM phase that looks at opcodes may need to know the new usages. Yes, I understand the point of complexity, but all along the history of the JVM, everything has been done for improving performance (as JIT and others techniques)even with a high development cost. So, why not going in this direction for the most heavily used data type in Cobol(integer, packed and zoned decimal)! > > > It seems to me just as easy to align source code to a BCI which > contains an invokestatic as a BCI which contains an iaload. Yes, generating an iaload or a invokestatic demands the same effort, but IMHO, the iaload performance will always beat the invokestatic performance (specially in long living loops where integer/indexes are involved) >> The problem for packed/zoned data is identical as for integer: just >> inefficiency >> and performance. > > With very few exceptions, the fastest way to get performance on some > operation is first to name it (as a method), then have the compiler > treat it as an intrinsic. Here is a list of all the intrinsics in > HotSpot (start 50% down around line 416): I did not know about those intrinsics in HotSpot. That is a great news for getting performance on unaligned integer and packed/zoned decimal. But this is specific to HotSpot if I understand well, and thus, this kind of optimization may be not available on other VM like JRockIt or Harmony or more important on the IBM VM running on z/OS. What makes me in favor of the relaxing option is that, once agreed as a specification, the Cobol class can run on all VMs without caring about the presence or absence of a specific optimization. This separation of concern is a real good point from the perspective of the compiler's builder and for getting an identical Cobol class behavior among different VMs. > > And so also do compiled intrinsics provide a way to do these things > efficiently. Think of bytecodes as low-level IR, which is very > expensive to extend, and intrinsic method calls as a higher-level > extensible part of the IR, which is it easy to extend compatibly. Ok, I will go in this direction...seems promising for getting performance on packed/zoned decimal reference and arithmetic computation too! >> More on packed/zoned cobol data in a following mail... Here is an extract from the IBM Principles of operations of any z/OS IBM hardware ? copyright IBM, reprinted with permission from the IBM Principles of Operations Manual Decimal integers consist of one or more decimal digits and a sign. Each digit and the sign are represented by a 4-bit code. The decimal digits are in binary-coded decimal (BCD) form, with the values 0-9 encoded as 0000-1001. The sign is usually represented as 1100 (C hex) for plus and 1101 (D hex) for minus. These are the preferred sign codes, which are generated by the machine for the results of decimal-arithmetic operations. There are also several alternate sign codes (1010, 1110, and 1111 for plus; 1011 for minus). The alternate sign codes are accepted by the machine as valid in source operands but are not generated for results. Decimal integers may have different lengths, from one to 16 bytes. There are two decimal formats: packed and zoned. In the packed format, each byte contains two decimal digits, except for the rightmost byte, which contains the sign code in the right half. For decimal arithmetic, the number of decimal digits in the packed format can vary from one to 31. Because decimal integers must consist of whole bytes and there must be a sign code on the right, the number of decimal digits is always odd. If an even number of significant digits is desired, a leading zero must be inserted on the left. In the zoned format, each byte consists of a decimal digit on the right and the zone code 1111 (F hex) on the left, except for the rightmost byte where the sign code replaces the zone code. Thus, a decimal integer in the zoned format can have from one to 16 digits. The zoned format may be used directly for input and output in the extended binary-coded-decimal interchange code (EBCDIC), except that the sign must be separated from the rightmost digit and handled as a separate character. For positive (unsigned) numbers, however, the sign can simply be represented by the zone code of the rightmost digit because the zone code is one of the acceptable alternate codes for plus. In either format, negative decimal integers are represented in true notation with a separate sign. As for binary integers, the radix point (decimal point) of decimal integers is considered to be fixed at the right, and any scaling is done by the programmer. The following are some examples of decimal integers shown in hexadecimal notation: Decimal Value Packed Format Zoned Format +123 12 3C F1 F2 C3 or or 12 3F F1 F2 F3 -4321 04 32 1D F4 F3 F2 D1 +000050 00 00 05 0C F0 F0 F0 F0 F5 C0 or or 00 00 05 0F F0 F0 F0 F0 F5 F0 -7 7D D7 00000 00 00 0C F0 F0 F0 F0 C0 or or 00 00 0F F0 F0 F0 F0 F0 Here is a good summary for the interest of using zoned decimal to display a decimal value and converting it from/to a packed one. http://www.simotime.com/datapk01.htm More on this and computing overflow/underflow in a next mail Francis From charles.nutter at sun.com Thu Apr 24 08:36:42 2008 From: charles.nutter at sun.com (Charles Oliver Nutter) Date: Thu, 24 Apr 2008 10:36:42 -0500 Subject: DVM patches applied, but build fails Message-ID: <4810A90A.9030303@sun.com> Seems like I'm missing something, or otherwise not patching correctly: g++ -DLINUX -D_GNU_SOURCE -DIA32 -DPRODUCT -I. -I../generated/adfiles -I../generated/jvmtifiles -I/home/headius/jdk7/jdk7/hotspot/src/share/vm/asm -I/home/headius/jdk7/jdk7/hotspot/src/share/vm/c1 -I/home/headius/jdk7/jdk7/hotspot/src/share/vm/ci -I/home/headius/jdk7/jdk7/hotspot/src/share/vm/classfile -I/home/headius/jdk7/jdk7/hotspot/src/share/vm/code -I/home/headius/jdk7/jdk7/hotspot/src/share/vm/compiler -I/home/headius/jdk7/jdk7/hotspot/src/share/vm/gc_implementation -I/home/headius/jdk7/jdk7/hotspot/src/share/vm/gc_implementation/parallelScavenge -I/home/headius/jdk7/jdk7/hotspot/src/share/vm/gc_implementation/shared -I/home/headius/jdk7/jdk7/hotspot/src/share/vm/gc_implementation/parNew -I/home/headius/jdk7/jdk7/hotspot/src/share/vm/gc_implementation/concurrentMarkSweep -I/home/headius/jdk7/jdk7/hotspot/src/share/vm/gc_interface -I/home/headius/jdk7/jdk7/hotspot/src/share/vm/interpreter -I/home/headius/jdk7/jdk7/hotspot/src/share/vm/libadt -I/home/headius/jdk7/jdk7/hotspot/src/share/vm/memory -I/home/headius/jdk7/jdk7/hotspot/src/share/vm/oops -I/home/headius/jdk7/jdk7/hotspot/src/share/vm/opto -I/home/headius/jdk7/jdk7/hotspot/src/share/vm/prims -I/home/headius/jdk7/jdk7/hotspot/src/share/vm/runtime -I/home/headius/jdk7/jdk7/hotspot/src/share/vm/services -I/home/headius/jdk7/jdk7/hotspot/src/share/vm/utilities -I/home/headius/jdk7/jdk7/hotspot/src/cpu/x86/vm -I/home/headius/jdk7/jdk7/hotspot/src/os/linux/vm -I/home/headius/jdk7/jdk7/hotspot/src/os_cpu/linux_x86/vm -I../generated -DHOTSPOT_RELEASE_VERSION="\"12.0-b01\"" -DHOTSPOT_BUILD_TARGET="\"product\"" -DHOTSPOT_BUILD_USER="\"headius\"" -DJRE_RELEASE_VERSION="\"1.7.0-internal-headius_24_apr_2008_09_53-b00\"" -DHOTSPOT_VM_DISTRO="\"OpenJDK\"" -DCOMPILER2 -DCOMPILER1 -fno-rtti -fno-exceptions -D_REENTRANT -fcheck-new -m32 -march=i586 -pipe -O3 -fno-strict-aliasing -DVM_LITTLE_ENDIAN -Werror -Wpointer-arith -Wconversion -Wsign-compare -c -o ciEnv.o /home/headius/jdk7/jdk7/hotspot/src/share/vm/ci/ciEnv.cpp /home/headius/jdk7/jdk7/hotspot/src/share/vm/ci/ciEnv.cpp: In member function ?ciConstant ciEnv::get_constant_by_index_impl(ciInstanceKlass*, int)?: /home/headius/jdk7/jdk7/hotspot/src/share/vm/ci/ciEnv.cpp:487: error: ?class constantPoolOopDesc? has no member named ?pseudo_string_at? make[5]: *** [ciEnv.o] Error 1 make[5]: Leaving directory `/home/headius/jdk7/jdk7/build/linux-i586/hotspot/outputdir/linux_i486_compiler2/product' make[4]: *** [the_vm] Error 2 make[4]: Leaving directory `/home/headius/jdk7/jdk7/build/linux-i586/hotspot/outputdir/linux_i486_compiler2/product' make[3]: *** [product] Error 2 make[3]: Leaving directory `/home/headius/jdk7/jdk7/build/linux-i586/hotspot/outputdir' make[2]: *** [generic_build2] Error 2 make[2]: Leaving directory `/home/headius/jdk7/jdk7/hotspot/make' make[1]: *** [product] Error 2 make[1]: Leaving directory `/home/headius/jdk7/jdk7/hotspot/make' make: *** [hotspot-build] Error 2 - Charlie From charles.nutter at sun.com Thu Apr 24 08:44:25 2008 From: charles.nutter at sun.com (Charles Oliver Nutter) Date: Thu, 24 Apr 2008 10:44:25 -0500 Subject: DVM patches applied, but build fails In-Reply-To: <4810A90A.9030303@sun.com> References: <4810A90A.9030303@sun.com> Message-ID: <4810AAD9.7060301@sun.com> Charles Oliver Nutter wrote: > Seems like I'm missing something, or otherwise not patching correctly: After a clean and rebuild, errors come out slightly differently: /home/headius/jdk7/jdk7/hotspot/src/share/vm/classfile/classFileParser.cpp: In member function ?constantPoolHandle ClassFileParser::parse_constant_pool(Thread*)?: /home/headius/jdk7/jdk7/hotspot/src/share/vm/classfile/classFileParser.cpp:316: error: ?find_well_known_klass? is not a member of ?SystemDictionary? /home/headius/jdk7/jdk7/hotspot/src/share/vm/classfile/classFileParser.cpp: In member function ?void ClassFileParser::patch_constant_pool(constantPoolHandle, int, Handle, Thread*)?: /home/headius/jdk7/jdk7/hotspot/src/share/vm/classfile/classFileParser.cpp:434: error: ?is_instance? is not a member of ?java_lang_Class? make[5]: *** [classFileParser.o] Error 1 - Charlie From charles.nutter at sun.com Thu Apr 24 08:52:42 2008 From: charles.nutter at sun.com (Charles Oliver Nutter) Date: Thu, 24 Apr 2008 10:52:42 -0500 Subject: Status of callcc Message-ID: <4810ACCA.60304@sun.com> While I'm waiting, hopefully, for someone to hold my hg-inexperienced hand, I was wondering about the status of copyStack/restoreStack at present. A few questions off the top of my head: - What's the correct way to use Unsafe to test this out? All I can find online are the nasty reflection hacks that make a field visible. - What's the performance like for copy/restore? I ask because I'm curious if it would be possible to implement a cooperative threading environment with the stack copying capabilities as they exist today. - Is this mostly complete, waiting for discussions about a public API, or is it still under development? - Charlie From mental at rydia.net Thu Apr 24 09:22:59 2008 From: mental at rydia.net (MenTaLguY) Date: Thu, 24 Apr 2008 9:22:59 -0700 Subject: Status of callcc In-Reply-To: <4810ACCA.60304@sun.com> References: <4810ACCA.60304@sun.com> Message-ID: <0951dd0854d96df48b75342cc539514e@localhost> On Thu, 24 Apr 2008 10:52:42 -0500, Charles Oliver Nutter wrote: > - What's the performance like for copy/restore? I ask because I'm > curious if it would be possible to implement a cooperative threading > environment with the stack copying capabilities as they exist today. I would expect it to be comparable to (or more likely a bit worse than) Ruby 1.8's threading, which uses copy/restore of the C stack. -mental From forax at univ-mlv.fr Thu Apr 24 09:45:29 2008 From: forax at univ-mlv.fr (=?ISO-8859-1?Q?R=E9mi_Forax?=) Date: Thu, 24 Apr 2008 18:45:29 +0200 Subject: Status of callcc In-Reply-To: <4810ACCA.60304@sun.com> References: <4810ACCA.60304@sun.com> Message-ID: <4810B929.8050602@univ-mlv.fr> Charles Oliver Nutter a ?crit : > While I'm waiting, hopefully, for someone to hold my hg-inexperienced > hand, I was wondering about the status of copyStack/restoreStack at > present. A few questions off the top of my head: > > - What's the correct way to use Unsafe to test this out? All I can find > online are the nasty reflection hacks that make a field visible. > - What's the performance like for copy/restore? I ask because I'm > curious if it would be possible to implement a cooperative threading > environment with the stack copying capabilities as they exist today. > - Is this mostly complete, waiting for discussions about a public API, > or is it still under development? > Reading from the source: still under dev, you can serialize a stack but not deserailize it: > - Charlie > R?mi From charles.nutter at sun.com Thu Apr 24 09:54:51 2008 From: charles.nutter at sun.com (Charles Oliver Nutter) Date: Thu, 24 Apr 2008 11:54:51 -0500 Subject: Status of callcc In-Reply-To: <0951dd0854d96df48b75342cc539514e@localhost> References: <4810ACCA.60304@sun.com> <0951dd0854d96df48b75342cc539514e@localhost> Message-ID: <4810BB5B.7090903@sun.com> MenTaLguY wrote: > On Thu, 24 Apr 2008 10:52:42 -0500, Charles Oliver Nutter wrote: >> - What's the performance like for copy/restore? I ask because I'm >> curious if it would be possible to implement a cooperative threading >> environment with the stack copying capabilities as they exist today. > > I would expect it to be comparable to (or more likely a bit worse than) Ruby 1.8's > threading, which uses copy/restore of the C stack. Well, ideally I'd like to see some experimental numbers on this...so that may be one of the first things I try. Ruby's model was exactly what I had in mind. Even if it's not the most efficient thing in the world, it would be a usable mechanism. - Charlie From lukas.stadler at jku.at Thu Apr 24 11:23:07 2008 From: lukas.stadler at jku.at (Lukas Stadler) Date: Thu, 24 Apr 2008 20:23:07 +0200 Subject: Status of callcc In-Reply-To: <4810ACCA.60304@sun.com> References: <4810ACCA.60304@sun.com> Message-ID: <4810D00B.5020900@jku.at> Hi! I'm working on this for 1 1/2 months now (under Johns supervision). I've implemented the deserialization part and I'm currently finishing an alternative representation of the serialized stack that is more accessible from the Java side. This should soon be finished and I'll hopefully be able to submit a patch with this code within the next 1-2 weeks. I've written a few small applications, for example one that serializes the stack to a file and resumes it later on, like a webserver that hibernates an inactive session to a database. performance: I haven't done any performance measurements yet. My code hasn't evolved far enough (I'm only working on the openjdk project for 1 1/2 months now, part time...) but there's a paper on a similar approach to continuations in a JVM that also discusses performance: http://lamp.epfl.ch/~dragos/files/ecoop-workshop.pdf completeness: I'd say it's under development, and with the public API comes the task of finding and dealing with all the VM invariants that can be broken by this. And security, of course, is a big question too. There's ongoing communication on these points on this mailing list, under the subject "stack manipulation API" I hope this helps, - Lukas Charles Oliver Nutter wrote: > While I'm waiting, hopefully, for someone to hold my hg-inexperienced > hand, I was wondering about the status of copyStack/restoreStack at > present. A few questions off the top of my head: > > - What's the correct way to use Unsafe to test this out? All I can find > online are the nasty reflection hacks that make a field visible. > - What's the performance like for copy/restore? I ask because I'm > curious if it would be possible to implement a cooperative threading > environment with the stack copying capabilities as they exist today. > - Is this mostly complete, waiting for discussions about a public API, > or is it still under development? > > - Charlie > _______________________________________________ > mlvm-dev mailing list > mlvm-dev at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev > From charles.nutter at sun.com Thu Apr 24 11:24:22 2008 From: charles.nutter at sun.com (Charles Oliver Nutter) Date: Thu, 24 Apr 2008 13:24:22 -0500 Subject: Status of callcc In-Reply-To: <4810D00B.5020900@jku.at> References: <4810ACCA.60304@sun.com> <4810D00B.5020900@jku.at> Message-ID: <4810D056.7010808@sun.com> Lukas Stadler wrote: > Hi! > > I'm working on this for 1 1/2 months now (under Johns supervision). I've > implemented the deserialization part and I'm currently finishing an > alternative representation of the serialized stack that is more > accessible from the Java side. This should soon be finished and I'll > hopefully be able to submit a patch with this code within the next 1-2 > weeks. > I've written a few small applications, for example one that serializes > the stack to a file and resumes it later on, like a webserver that > hibernates an inactive session to a database. > > performance: I haven't done any performance measurements yet. My code > hasn't evolved far enough (I'm only working on the openjdk project for 1 > 1/2 months now, part time...) but there's a paper on a similar approach > to continuations in a JVM that also discusses performance: > http://lamp.epfl.ch/~dragos/files/ecoop-workshop.pdf > > completeness: I'd say it's under development, and with the public API > comes the task of finding and dealing with all the VM invariants that > can be broken by this. And security, of course, is a big question too. > There's ongoing communication on these points on this mailing list, > under the subject "stack manipulation API" Thanks very much for the update. I will track that thread and try to weigh in where I'm able. And I'll certainly be ready to play with it when you feel like it's in at least "usable enough" experimental state to start trying things out.. - Charlie From charles.nutter at sun.com Thu Apr 24 11:29:49 2008 From: charles.nutter at sun.com (Charles Oliver Nutter) Date: Thu, 24 Apr 2008 13:29:49 -0500 Subject: Switch profiling for better switch performance? Message-ID: <4810D19D.3030203@sun.com> I'm a little fuzzy on the status/level of optimization HotSpot performance based on switch profiling. John Rose's wiki page on HotSpot optimizations claims that switches are profiled, but that the information isn't used very well. And on JRuby and JOni projects, we have seen that large switches have progressively worse performance. John mentioned partial work that had been done on optimized jump-tables for switch jitting, but I don't know where that went. I ask because as more languages come online on JVM, we're going to run into more cases where a bytecode VM *on top of* the JVM is the right way to go. For example. JRuby will want to support the two Ruby bytecode implementations in Ruby 1.9 and Rubinius. Jython will want to support Python's .pyc bytecode. JOni is already a very fast and fully-functional bytecode-based regexp engine. Using a bytecode machine on JVM could also provide an easier path to fast startup and cheap code execution for languages that support eval or that are more fluid like Ruby. Currently, in order to save on startup costs, JRuby starts out all code intepreted (AST-walking), eventually JITing to Java bytecode which is then presumably picked up by HotSpot. But we could probably get a performance boost if we could implement a fast, stackless bytecode engine rather than an AST walker. At the moment, however, we worry about switch performance. I know resources are a problem, but is there any thought going towards improving switch performance? I know at least one JRuby team member that could provide some pretty unpleasant benchmark results that show the pain we're facing. - Charlie From John.Rose at Sun.COM Thu Apr 24 11:48:31 2008 From: John.Rose at Sun.COM (John Rose) Date: Thu, 24 Apr 2008 11:48:31 -0700 Subject: Open JDK build with mlvm patches In-Reply-To: <5def42a80804240506n6dc22605o48f4147aada7fd4@mail.gmail.com> References: <5def42a80804240506n6dc22605o48f4147aada7fd4@mail.gmail.com> Message-ID: <980F841B-EE28-40D9-952B-AC787CE2C30E@Sun.COM> [cc-ing mlvm-dev in reply, because of general interest] On Apr 24, 2008, at 5:06 AM, Thamaraiselvan Poomalai wrote: > Hello John, > > I have successfully compiled hotsopt and jdk with mlvm patches from > http://hg.openjdk.java.net/mlvm/mlvm. Thanks for giving it a spin, Selvan. > 1) Though i did full jdk build after applying patches, java/dyn > package is skipped by build process (i couldn't trace compiled java/ > dyn packages under sources/build/openjdk_full_debug/classes/java). > Could we also include necessary make files for java/dyn package in > our patch? Yes, that's coming. > 2) Here are the things i did to make build successful, > > After executing below commands from README.txt, > $ mkdir davinci > ... ... ... > ... ... ... > $ sh patches/make/each-patch-repo.sh hg qselect --reapply $guards > > i executed below command to push patch changes to source repository, > > $ sh patches/make/each-patch-repo.sh hg qpush -a Hmm. I thought --reapply would do this; We'll add that to the instructions. > Output of above command was, > =========================== > [root at iwsg OpenJDK]# sh patches/make/each-patch-repo.sh hg qpush -a > + (cd sources/hotspot; hg qpush -a) > applying anonk.patch > skipping callcc.patch - guarded by '-testable' > Now at: anonk.patch > + (cd sources/jdk; hg qpush -a) > applying anonk.patch > skipping callcc.patch - guarded by '-testable' > Now at: anonk.patch > =========================== > > It skipped callcc.patch for hotspot and jdk. When i try to do build > with callcc.patch skipped, i got following error, Yes, callcc.patch is very rough right now. Lukas Stadler has a much more complete version which I think he'll be releasing soon. > =========================== > /share/software/OpenJDK/sources/hotspot/src/share/vm/classfile/ > classFileParser.cpp: In member function 'constantPoolHandle > ClassFileParser::parse_constant_pool(Thread*)': > /share/software/OpenJDK/sources/hotspot/src/share/vm/classfile/ > classFileParser.cpp:316: error: 'find_well_known_klass' is not a > member of 'SystemDictionary' > /share/software/OpenJDK/sources/hotspot/src/share/vm/classfile/ > classFileParser.cpp: In member function 'void > ClassFileParser::patch_constant_pool(constantPoolHandle, int, > Handle, Thread*)': > /share/software/OpenJDK/sources/hotspot/src/share/vm/classfile/ > classFileParser.cpp:434: error: 'is_instance' is not a member of > 'java_lang_Class' > make[5]: *** [classFileParser.o] Error 1 > make[5]: Leaving directory `/share/software/OpenJDK/sources/build/ > openjdk_full_debug/hotspot/outputdir/linux_i486_compiler2/jvmg' > make[4]: *** [the_vm] Error 2 > =========================== Thanks. It looks like callcc.patch has a tweak that anonk.patch should have instead. The well-known-klass fix is not yet in the baseline (I'm working on it today). > I removed "#-testable" from sources/jdk/.hg/patches/series, sources/ > hotspot/.hg/patches/series and re-applied callcc.patch.After > removing "#-testable", series file looks as below > > anonk.patch #-/anonk #+jdk7-b25 > #meth.patch #-/meth #+jdk7-b25 #-buildable > #indy.patch #-/indy #+jdk7-b25 #-buildable > #inti.patch #-/inti #+jdk7-b25 #-buildable > callcc.patch #-/callcc #+jdk7-b25 To get callcc in, it would be better to qselect buildable (not buildable + testable). The series file is supposed to represent the rough development status of each patch. > After applying anonk.patch and callcc.patch to sources, hotspot and > full jdk build was successful. > > My login name for wiki (http://wikis.sun.com/display/mlvm/Home) is > p.thamarai at gmail.com OK, I added you (user id selvan) as an author to http://wikis.sun.com/ display/mlvm . Please feel free to record useful information on the wiki. -- John -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/mlvm-dev/attachments/20080424/e8a26cca/attachment-0001.html From John.Rose at Sun.COM Thu Apr 24 12:23:27 2008 From: John.Rose at Sun.COM (John Rose) Date: Thu, 24 Apr 2008 12:23:27 -0700 Subject: Switch profiling for better switch performance? In-Reply-To: <4810D19D.3030203@sun.com> References: <4810D19D.3030203@sun.com> Message-ID: On Apr 24, 2008, at 11:29 AM, Charles Oliver Nutter wrote: > I know resources are a problem, but is there any thought going towards > improving switch performance? I know at least one JRuby team member > that > could provide some pretty unpleasant benchmark results that show the > pain we're facing. That would be a great first project for a new hotspot programmer. The present set of Sun engineers is a couple hundred feet underwater, and the upward view shows a number of such optimization projects, large and small, floating in the sunlight, out of our present reach. We are not limited by our vision, but by our numbers. Another difficulty is that it generally takes people a year or two to get comfortable with the details of the compilers (server and client JIT). But the fastest way in is total immersion (to continue the aqueous metaphor). I personally would enjoy helping someone improve the collection and use of profiling information in hotspot, for switches. Especially if it would make bytecode interpreters go faster (most especially, RE engines). The first step is to file a bug report (on bad performance of switch), with test code. The next step is to study the materials on contributing to OpenJDK. http://openjdk.java.net/contribute/ -- John -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/mlvm-dev/attachments/20080424/d7b3a253/attachment.html From John.Rose at Sun.COM Thu Apr 24 12:59:52 2008 From: John.Rose at Sun.COM (John Rose) Date: Thu, 24 Apr 2008 12:59:52 -0700 Subject: [PATCH] callcc, anonk: anonk will not build unless callcc is pushed also In-Reply-To: <980F841B-EE28-40D9-952B-AC787CE2C30E@Sun.COM> References: <5def42a80804240506n6dc22605o48f4147aada7fd4@mail.gmail.com> <980F841B-EE28-40D9-952B-AC787CE2C30E@Sun.COM> Message-ID: <5708D7DB-7D38-4D1F-9A27-E9963A24DCC3@sun.com> This is to fix the build bug observed by Selvan and Charlie. It is a second-order patch (patch on a patch repository). Apply it in davinci/patches/hotspot/. May I have a reviewer, please? -- John -------------- next part -------------- A non-text attachment was scrubbed... Name: wkk-tweak.dpatch Type: application/octet-stream Size: 2945 bytes Desc: not available Url : http://mail.openjdk.java.net/pipermail/mlvm-dev/attachments/20080424/6783e348/attachment.obj From John.Rose at Sun.COM Thu Apr 24 13:08:11 2008 From: John.Rose at Sun.COM (John Rose) Date: Thu, 24 Apr 2008 13:08:11 -0700 Subject: [PATCH] adjust and clarify build instructions Message-ID: For this one, too, may I have a reviewer? Thanks. -- John P.S. Reviews can include suggestions for improvement! -------------- next part -------------- A non-text attachment was scrubbed... Name: readme.patch Type: application/octet-stream Size: 1870 bytes Desc: not available Url : http://mail.openjdk.java.net/pipermail/mlvm-dev/attachments/20080424/e6ca404c/attachment.obj From charles.nutter at sun.com Thu Apr 24 13:23:57 2008 From: charles.nutter at sun.com (Charles Oliver Nutter) Date: Thu, 24 Apr 2008 15:23:57 -0500 Subject: [PATCH] callcc, anonk: anonk will not build unless callcc is pushed also In-Reply-To: <5708D7DB-7D38-4D1F-9A27-E9963A24DCC3@sun.com> References: <5def42a80804240506n6dc22605o48f4147aada7fd4@mail.gmail.com> <980F841B-EE28-40D9-952B-AC787CE2C30E@Sun.COM> <5708D7DB-7D38-4D1F-9A27-E9963A24DCC3@sun.com> Message-ID: <4810EC5D.9090606@sun.com> John Rose wrote: > This is to fix the build bug observed by Selvan and Charlie. > > It is a second-order patch (patch on a patch repository). Apply it in > davinci/patches/hotspot/. > > May I have a reviewer, please? Checking it out now. Should this be built with or without testable (i.e. are you looking to get a testable patchset working as a result of this)? - Charlie From charles.nutter at sun.com Thu Apr 24 13:31:22 2008 From: charles.nutter at sun.com (Charles Oliver Nutter) Date: Thu, 24 Apr 2008 15:31:22 -0500 Subject: [PATCH] adjust and clarify build instructions In-Reply-To: References: Message-ID: <4810EE1A.3070309@sun.com> John Rose wrote: > For this one, too, may I have a reviewer? Thanks. -- John > > P.S. Reviews can include suggestions for improvement! Looks good to me, and now includes the two key commands I had to hunt for: qpop and qpush. No suggestions for improvement at the moment, because I know exactly today's worth of fiddling about hg. - Charlie From charles.nutter at sun.com Thu Apr 24 13:48:46 2008 From: charles.nutter at sun.com (Charles Oliver Nutter) Date: Thu, 24 Apr 2008 15:48:46 -0500 Subject: [PATCH] callcc, anonk: anonk will not build unless callcc is pushed also In-Reply-To: <5708D7DB-7D38-4D1F-9A27-E9963A24DCC3@sun.com> References: <5def42a80804240506n6dc22605o48f4147aada7fd4@mail.gmail.com> <980F841B-EE28-40D9-952B-AC787CE2C30E@Sun.COM> <5708D7DB-7D38-4D1F-9A27-E9963A24DCC3@sun.com> Message-ID: <4810F22E.3090205@sun.com> John Rose wrote: > This is to fix the build bug observed by Selvan and Charlie. > > It is a second-order patch (patch on a patch repository). Apply it in > davinci/patches/hotspot/. > > May I have a reviewer, please? I was able to build, but I can't seem to load java.dyn.AnonymousClassLoader in any code I'm running. That could be unrelated, perhaps some step I missed? - Charlie From charles.nutter at sun.com Thu Apr 24 14:06:07 2008 From: charles.nutter at sun.com (Charles Oliver Nutter) Date: Thu, 24 Apr 2008 16:06:07 -0500 Subject: [PATCH] callcc, anonk: anonk will not build unless callcc is pushed also In-Reply-To: <4810F22E.3090205@sun.com> References: <5def42a80804240506n6dc22605o48f4147aada7fd4@mail.gmail.com> <980F841B-EE28-40D9-952B-AC787CE2C30E@Sun.COM> <5708D7DB-7D38-4D1F-9A27-E9963A24DCC3@sun.com> <4810F22E.3090205@sun.com> Message-ID: <4810F63F.70703@sun.com> Charles Oliver Nutter wrote: > I was able to build, but I can't seem to load > java.dyn.AnonymousClassLoader in any code I'm running. That could be > unrelated, perhaps some step I missed? Doesn't seem like java.dyn is getting stuffed into the build output for me: headius at headius-desktop:~/jdk7/jdk7/tmp$ ls ../build/linux-i586/classes/java/ applet awt beans io lang math net nio rmi security sql text util headius at headius-desktop:~/jdk7/jdk7/tmp$ find ../build/linux-i586/ -name dyn headius at headius-desktop:~/jdk7/jdk7/tmp$ I'm trying a clean build now in case something got screwed up. I've confirmed AnonymousClassLoader.java exists under jdk/src/share/classes/java/dyn. - Charlie From John.Rose at Sun.COM Thu Apr 24 14:22:44 2008 From: John.Rose at Sun.COM (John Rose) Date: Thu, 24 Apr 2008 14:22:44 -0700 Subject: [PATCH] callcc, anonk: anonk will not build unless callcc is pushed also In-Reply-To: <4810F22E.3090205@sun.com> References: <5def42a80804240506n6dc22605o48f4147aada7fd4@mail.gmail.com> <980F841B-EE28-40D9-952B-AC787CE2C30E@Sun.COM> <5708D7DB-7D38-4D1F-9A27-E9963A24DCC3@sun.com> <4810F22E.3090205@sun.com> Message-ID: <7A00B695-6845-46A1-A535-7D2228CAB077@Sun.COM> On Apr 24, 2008, at 1:23 PM, Charles Oliver Nutter wrote: > John Rose wrote: >> This is to fix the build bug observed by Selvan and Charlie. >> It is a second-order patch (patch on a patch repository). Apply >> it in davinci/patches/hotspot/. >> May I have a reviewer, please? > > Checking it out now. Should this be built with or without testable > (i.e. are you looking to get a testable patchset working as a > result of this)? The default, buildable + testable. > I was able to build, but I can't seem to load > java.dyn.AnonymousClassLoader in any code I'm running. That could > be unrelated, perhaps some step I missed? Yes, it's unrelated. Check anonk.txt for an incremental javac command plus bootclasspath stuff. That's what I've been using to start with, just as a matter of taste. So, I'll count your build success as a review. (Unless you have more review comments? :-) We'll tweak the JDK build shortly if necessary. (Remi, did you have to take special action to get java.dyn.* into rt.jar?) -- John From forax at univ-mlv.fr Thu Apr 24 14:21:38 2008 From: forax at univ-mlv.fr (=?ISO-8859-1?Q?R=E9mi_Forax?=) Date: Thu, 24 Apr 2008 23:21:38 +0200 Subject: [PATCH] callcc, anonk: anonk will not build unless callcc is pushed also In-Reply-To: <4810F22E.3090205@sun.com> References: <5def42a80804240506n6dc22605o48f4147aada7fd4@mail.gmail.com> <980F841B-EE28-40D9-952B-AC787CE2C30E@Sun.COM> <5708D7DB-7D38-4D1F-9A27-E9963A24DCC3@sun.com> <4810F22E.3090205@sun.com> Message-ID: <4810F9E2.2050508@univ-mlv.fr> Charles Oliver Nutter a ?crit : > John Rose wrote: > >> This is to fix the build bug observed by Selvan and Charlie. >> >> It is a second-order patch (patch on a patch repository). Apply it in >> davinci/patches/hotspot/. >> >> May I have a reviewer, please? >> > > I was able to build, but I can't seem to load > java.dyn.AnonymousClassLoader in any code I'm running. That could be > unrelated, perhaps some step I missed? > did you try with the code available here ? http://www-igm.univ-mlv.fr/~forax/tmp/davinci-anonk.zip AnonymousTest is the test case. > - Charlie > R?mi From john.rose at sun.com Thu Apr 24 14:35:07 2008 From: john.rose at sun.com (john.rose at sun.com) Date: Thu, 24 Apr 2008 21:35:07 +0000 Subject: hg: mlvm/mlvm: adjust and clarify build instructions Message-ID: <20080424213507.33C5C27580@hg.openjdk.java.net> Changeset: 916d22cb98a6 Author: jrose Date: 2008-04-24 14:33 -0700 URL: http://hg.openjdk.java.net/mlvm/mlvm/rev/916d22cb98a6 adjust and clarify build instructions Reviewed-by: charles.nutter at sun.com ! README.txt From john.rose at sun.com Thu Apr 24 14:35:26 2008 From: john.rose at sun.com (john.rose at sun.com) Date: Thu, 24 Apr 2008 21:35:26 +0000 Subject: hg: mlvm/mlvm/hotspot: callcc, anonk: anonk will not build unless callcc is pushed also Message-ID: <20080424213526.A3E9427585@hg.openjdk.java.net> Changeset: db693384cccb Author: jrose Date: 2008-04-24 14:31 -0700 URL: http://hg.openjdk.java.net/mlvm/mlvm/hotspot/rev/db693384cccb callcc, anonk: anonk will not build unless callcc is pushed also Summary: move wkk tweak from callcc to anonk (pending wkk fix is bug 6652736) Reviewed-by: charles.nutter at sun.com ! anonk.patch ! callcc.patch From forax at univ-mlv.fr Thu Apr 24 14:32:10 2008 From: forax at univ-mlv.fr (=?ISO-8859-1?Q?R=E9mi_Forax?=) Date: Thu, 24 Apr 2008 23:32:10 +0200 Subject: [PATCH] callcc, anonk: anonk will not build unless callcc is pushed also In-Reply-To: <7A00B695-6845-46A1-A535-7D2228CAB077@Sun.COM> References: <5def42a80804240506n6dc22605o48f4147aada7fd4@mail.gmail.com> <980F841B-EE28-40D9-952B-AC787CE2C30E@Sun.COM> <5708D7DB-7D38-4D1F-9A27-E9963A24DCC3@sun.com> <4810F22E.3090205@sun.com> <7A00B695-6845-46A1-A535-7D2228CAB077@Sun.COM> Message-ID: <4810FC5A.3060808@univ-mlv.fr> John Rose a ?crit : > On Apr 24, 2008, at 1:23 PM, Charles Oliver Nutter wrote: >> John Rose wrote: >>> This is to fix the build bug observed by Selvan and Charlie. >>> It is a second-order patch (patch on a patch repository). Apply it >>> in davinci/patches/hotspot/. >>> May I have a reviewer, please? >> >> Checking it out now. Should this be built with or without testable >> (i.e. are you looking to get a testable patchset working as a result >> of this)? > > The default, buildable + testable. > >> I was able to build, but I can't seem to load >> java.dyn.AnonymousClassLoader in any code I'm running. That could be >> unrelated, perhaps some step I missed? > > Yes, it's unrelated. Check anonk.txt for an incremental javac command > plus bootclasspath stuff. That's what I've been using to start with, > just as a matter of taste. > > So, I'll count your build success as a review. (Unless you have more > review comments? :-) > > We'll tweak the JDK build shortly if necessary. (Remi, did you have > to take special action to get java.dyn.* into rt.jar?) no, i have just prepend the classes to the bootstrap classpath using: -Xbootclasspath/p: > > -- John R?mi From kumpera at gmail.com Thu Apr 24 14:50:51 2008 From: kumpera at gmail.com (Rodrigo Kumpera) Date: Thu, 24 Apr 2008 18:50:51 -0300 Subject: continuations design Message-ID: <8cca42d80804241450i58e282caxbaf19e4218bf413@mail.gmail.com> Hey guys, I've been lurking this list for a while and I got a few questions about the callcc design. >From reading the patches when yielding from a contituation the pertinent stack frames are copied. Have you considered allocating the continuation stack separately and switching to it on callcc? Growing it on stack overflow is sticky but doable. This would only work for delimited continuations, would require to switch back to the original stack on managed<->unmanaged transitions. I guess the overall performance hit would be small, but avoiding a lot of copies would surelly be a big win. Other question, how do you plan to handle yielding when runtime frames are in the middle? As, for example, caused by class initialization. Thanks, Rodrigo -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/mlvm-dev/attachments/20080424/55bd4fff/attachment.html From forax at univ-mlv.fr Thu Apr 24 15:44:35 2008 From: forax at univ-mlv.fr (=?ISO-8859-1?Q?R=E9mi_Forax?=) Date: Fri, 25 Apr 2008 00:44:35 +0200 Subject: AnonymousClassLoader and ClassLoader In-Reply-To: <48109E8A.1090507@sun.com> References: <4810524A.40504@univ-mlv.fr> <48109E8A.1090507@sun.com> Message-ID: <48110D53.4030104@univ-mlv.fr> Charles Oliver Nutter a ?crit : > R?mi Forax wrote: >> Yesterday evening, I've tried to replace the application classloader >> (the one used to load the application) by one that use an anymous >> class loader >> to load the class. And i've tested by launching SwingSet2. >> >> And it doesn't work because UI program like SwingSet lookup their >> resources >> like image, icons etc. using getResource()/getResourceAsStream(). >> getResource() makes the assumption that if getClassLoader() returns null >> the class is loaded by the bootstrap classloader. >> >> In my opinion, getResource() is not the only method that makes this >> assumption >> so an anonymous class should be able to specify an external classloader >> when defined i.e. Unsafe.defineAnonymousClass should take >> a classloader as argument. > > An interesting thought. I can think of a few places I expect to be > able to call getClassLoader on a target class I've loaded at runtime, > but that's generally because I want to associate other classes > dependent on the target class (i.e. method handles for a > freshly-loaded compiled Ruby script) with the same throwaway > classloader. Given the looser nature of the anonymous classloader, > that particular scenario would go away. I'd just always grab my trust > anonymous classloader and throw classes at it, knowing they'd GC when > they were no longer in use. > > I don't think the anonymous classloader is going to be something you > can just start using lightly, since assumptions existing classes make > about classloading are sure to get in the way. The problem is: even if you want to use it lightly, i think you can't control the fact that an user can send your closure or any instance of a class loaded by the anonymous loader to a lib that relies on getClassLoader(). > > - Charlie R?mi From John.Rose at Sun.COM Thu Apr 24 16:09:44 2008 From: John.Rose at Sun.COM (John Rose) Date: Thu, 24 Apr 2008 16:09:44 -0700 Subject: AnonymousClassLoader and ClassLoader In-Reply-To: <4810726E.6020003@univ-mlv.fr> References: <4810524A.40504@univ-mlv.fr> <4810726E.6020003@univ-mlv.fr> Message-ID: <762D78F2-E5A3-42C2-A658-E9C79BB13817@sun.com> On Apr 24, 2008, at 4:43 AM, R?mi Forax wrote: > I answer to myself, it's only in the case the host-class is null, > otherwise the classloader used is the one of the host-class. Right. The new class is supposed to inherit the same access rights, protection domain, class loader, from the host class. (In particular, the new class should be able to access the private members of the host class. This gives a structured way to define "new" code in the old class.) > What is exactly the meaning of a null host-class ? Here's what I had in mind, which I think is the most useful: Default (system) protection domain and class loader. As if there were some empty system class that acts as the host class. -- John From John.Rose at Sun.COM Thu Apr 24 16:20:40 2008 From: John.Rose at Sun.COM (John Rose) Date: Thu, 24 Apr 2008 16:20:40 -0700 Subject: AnonymousClassLoader and ClassLoader In-Reply-To: <4810524A.40504@univ-mlv.fr> References: <4810524A.40504@univ-mlv.fr> Message-ID: <9F9ADA1B-DAAE-415E-92BD-C2B7B425B2B4@sun.com> On Apr 24, 2008, at 2:26 AM, R?mi Forax wrote: > In my opinion, getResource() is not the only method that makes this > assumption The getResource method probably needs to be adjusted since anonymous classes have names that are intentionally distorted. The anonymous class name ends with "/${hashcode}" so that attempts to do forName will rapidly fail, and so that it is clear which classes are anonymous. The logic for Class.resolveName needs to take this case into account if resources are to work. (I assume that it is a good idea for them to "work", but maybe it's not.) It's a good question, and a nice experiment. -- John From charles.nutter at sun.com Thu Apr 24 16:49:42 2008 From: charles.nutter at sun.com (Charles Oliver Nutter) Date: Thu, 24 Apr 2008 18:49:42 -0500 Subject: [PATCH] callcc, anonk: anonk will not build unless callcc is pushed also In-Reply-To: <7A00B695-6845-46A1-A535-7D2228CAB077@Sun.COM> References: <5def42a80804240506n6dc22605o48f4147aada7fd4@mail.gmail.com> <980F841B-EE28-40D9-952B-AC787CE2C30E@Sun.COM> <5708D7DB-7D38-4D1F-9A27-E9963A24DCC3@sun.com> <4810F22E.3090205@sun.com> <7A00B695-6845-46A1-A535-7D2228CAB077@Sun.COM> Message-ID: <48111C96.4050207@sun.com> John Rose wrote: > So, I'll count your build success as a review. (Unless you have more > review comments? :-) I realized I had run without testable, so I re-ran to confirm...build looks fine. - Charlie From charles.nutter at sun.com Thu Apr 24 21:44:22 2008 From: charles.nutter at sun.com (Charles Oliver Nutter) Date: Thu, 24 Apr 2008 23:44:22 -0500 Subject: [PATCH] callcc, anonk: anonk will not build unless callcc is pushed also In-Reply-To: <7A00B695-6845-46A1-A535-7D2228CAB077@Sun.COM> References: <5def42a80804240506n6dc22605o48f4147aada7fd4@mail.gmail.com> <980F841B-EE28-40D9-952B-AC787CE2C30E@Sun.COM> <5708D7DB-7D38-4D1F-9A27-E9963A24DCC3@sun.com> <4810F22E.3090205@sun.com> <7A00B695-6845-46A1-A535-7D2228CAB077@Sun.COM> Message-ID: <481161A6.7020207@sun.com> John Rose wrote: > On Apr 24, 2008, at 1:23 PM, Charles Oliver Nutter wrote: >> John Rose wrote: >>> This is to fix the build bug observed by Selvan and Charlie. >>> It is a second-order patch (patch on a patch repository). Apply it >>> in davinci/patches/hotspot/. >>> May I have a reviewer, please? >> >> Checking it out now. Should this be built with or without testable >> (i.e. are you looking to get a testable patchset working as a result >> of this)? > > The default, buildable + testable. > >> I was able to build, but I can't seem to load >> java.dyn.AnonymousClassLoader in any code I'm running. That could be >> unrelated, perhaps some step I missed? > > Yes, it's unrelated. Check anonk.txt for an incremental javac command > plus bootclasspath stuff. That's what I've been using to start with, > just as a matter of taste. Ahh, that's the step I was missing. Thanks. It loads and constructs now, so I can continue :) > So, I'll count your build success as a review. (Unless you have more > review comments? :-) None at all! > We'll tweak the JDK build shortly if necessary. (Remi, did you have to > take special action to get java.dyn.* into rt.jar?) > > -- John > From forax at univ-mlv.fr Fri Apr 25 08:47:30 2008 From: forax at univ-mlv.fr (=?ISO-8859-1?Q?R=E9mi_Forax?=) Date: Fri, 25 Apr 2008 17:47:30 +0200 Subject: AnonymousClassLoader and ClassLoader In-Reply-To: <9F9ADA1B-DAAE-415E-92BD-C2B7B425B2B4@sun.com> References: <4810524A.40504@univ-mlv.fr> <9F9ADA1B-DAAE-415E-92BD-C2B7B425B2B4@sun.com> Message-ID: <4811FD12.7040901@univ-mlv.fr> John Rose a ?crit : > On Apr 24, 2008, at 2:26 AM, R?mi Forax wrote: > >> In my opinion, getResource() is not the only method that makes this >> assumption > > The getResource method probably needs to be adjusted since anonymous > classes have names that are intentionally distorted. The anonymous > class name ends with "/${hashcode}" so that attempts to do forName > will rapidly fail, and so that it is clear which classes are anonymous. Are you sure hashcode are unique and never creates a name collisions ? > The logic for Class.resolveName needs to take this case into account > if resources are to work. (I assume that it is a good idea for them > to "work", but maybe it's not.) It's a good question, and a nice > experiment. > > -- John Perhaps it's a good idea to try to surface isAnonymousClass() a Java level or even better getHostClass() (or getAnonymousHostClass()). I will try to create a patch for that this week-end, i seems to be a good idea to try to understand the VM internals. Another experiment, trying to resolve an anonymous class using ClassLoader.resolveClass() leads to a crash in SystemDictionnary. Step to reproduce, change sun.misc.Launcher by the one in attachement, I have chnage the application classloader to load all non bootstrap class as anonymous one (see findClass) but when it tries to call resolveClass() (in ClassLoader.loadClass()) on an anonymous class, the VM crashs. I don't know if trying to resolve an anonymous class is legal or not. The following lines are a cut&paste of the crash using the fastdebug VM (patched with anonk and callcc). R?mi [forax at localhost eclipse]$ java -davinci -Xbootclasspath/p:classes -cp test/classes:/usr/local/java/demo/jfc/SwingSet2/SwingSet2.jar AllAnonymousLauncher SwingSet2 load anonymous AllAnonymousLauncher resource sun.misc.URLClassPath$FileLoader$1 at f6a746 anonymous loaded class AllAnonymousLauncher/26208195 # To suppress the following error report, specify this argument # after -XX: or in .hotspotrc: SuppressErrorAt=/systemDictionary.cpp:850 # # An unexpected error has been detected by Java Runtime Environment: # # Internal Error (/home/forax/java/workspace/davinci/sources/hotspot/src/share/vm/classfile/systemDictionary.cpp:850), pid=15414, tid=3084950416 # Error: assert(kk == k(),"should be present in dictionary") # # Java VM: OpenJDK Client VM (12.0-b01-internal-fastdebug mixed mode linux-x86) # An error report file with more information is saved as: # /home/forax/java/workspace/davinci/eclipse/hs_err_pid15414.log # # If you would like to submit a bug report, please visit: # http://java.sun.com/webapps/bugreport/crash.jsp # Current thread is 3084950416 Dumping core ... Abandon -------------- next part -------------- A non-text attachment was scrubbed... Name: Launcher.java Type: text/x-java Size: 22642 bytes Desc: not available Url : http://mail.openjdk.java.net/pipermail/mlvm-dev/attachments/20080425/e9883d05/attachment-0003.bin -------------- next part -------------- A non-text attachment was scrubbed... Name: AllAnonymousLauncher.java Type: text/x-java Size: 533 bytes Desc: not available Url : http://mail.openjdk.java.net/pipermail/mlvm-dev/attachments/20080425/e9883d05/attachment-0004.bin -------------- next part -------------- A non-text attachment was scrubbed... Name: hs_err_pid15414.log Type: text/x-log Size: 10082 bytes Desc: not available Url : http://mail.openjdk.java.net/pipermail/mlvm-dev/attachments/20080425/e9883d05/attachment-0005.bin From John.Rose at Sun.COM Fri Apr 25 12:48:05 2008 From: John.Rose at Sun.COM (John Rose) Date: Fri, 25 Apr 2008 12:48:05 -0700 Subject: AnonymousClassLoader and ClassLoader In-Reply-To: <4811FD12.7040901@univ-mlv.fr> References: <4810524A.40504@univ-mlv.fr> <9F9ADA1B-DAAE-415E-92BD-C2B7B425B2B4@sun.com> <4811FD12.7040901@univ-mlv.fr> Message-ID: <749B4C2D-CE3C-4642-9CD8-29BE542EF089@Sun.COM> On Apr 25, 2008, at 8:47 AM, R?mi Forax wrote: > John Rose a ?crit : >> On Apr 24, 2008, at 2:26 AM, R?mi Forax wrote: >> >>> In my opinion, getResource() is not the only method that makes this >>> assumption >> >> The getResource method probably needs to be adjusted since >> anonymous classes have names that are intentionally distorted. >> The anonymous class name ends with "/${hashcode}" so that attempts >> to do forName will rapidly fail, and so that it is clear which >> classes are anonymous. > Are you sure hashcode are unique and never creates a name collisions ? No, there *will* be name collisions. The hashcode is to help with debugging, and should be ignored by any code that is attempting to assign a name to the anonymous class. (Everything after the slash should be stripped.) Using the name in any way at all is a slippery slope, but obtaining the resources from the class's formal name seems like it's not too much of a hack. >> The logic for Class.resolveName needs to take this case into >> account if resources are to work. (I assume that it is a good >> idea for them to "work", but maybe it's not.) It's a good >> question, and a nice experiment. >> >> -- John > Perhaps it's a good idea to try to surface isAnonymousClass() a > Java level > or even better getHostClass() (or getAnonymousHostClass()). > I will try to create a patch for that this week-end, i seems to be > a good idea > to try to understand the VM internals. Yes; you can see from the JVM patch that I'm on the edge of doing this. The useful idea at present is that the class anonymous iff it has a slash in its getName; it would be bad for it to have a legal getName. > Another experiment, > trying to resolve an anonymous class using ClassLoader.resolveClass > () leads to a crash in SystemDictionnary. > Step to reproduce, change sun.misc.Launcher by the one in attachement, > I have chnage the application classloader to load all non bootstrap > class as anonymous one > (see findClass) but when it tries to call resolveClass() (in > ClassLoader.loadClass()) on an anonymous class, > the VM crashs. > > I don't know if trying to resolve an anonymous class is legal or not. No, it's not legal. It should throw a ClassNotFound or similar exception. Should not crash, obviously. > The following lines are a cut&paste of the crash using the > fastdebug VM (patched with anonk and callcc). Right, a failure like that is to be expected if the query gets that far. When you load everything as anonymous, how do you resolve intra- module references? That seems hopeless. You can resolve some references by using the CP patching feature, but that does not cover CP circularities. We need a simultaneous loading model to create complex graphs of anonymous classes. I don't see the need to do this yet. (First figure out a multi-class loading model for normal classes! I have some good ideas for this based on my work with Pack200, but they are back-burnered because of more pressing features.) -- John From charles.nutter at sun.com Fri Apr 25 16:00:30 2008 From: charles.nutter at sun.com (Charles Oliver Nutter) Date: Fri, 25 Apr 2008 18:00:30 -0500 Subject: AnonymousClassLoader and ClassLoader In-Reply-To: <48110D53.4030104@univ-mlv.fr> References: <4810524A.40504@univ-mlv.fr> <48109E8A.1090507@sun.com> <48110D53.4030104@univ-mlv.fr> Message-ID: <4812628E.4090002@sun.com> R?mi Forax wrote: > The problem is: even if you want to use it lightly, i think you can't > control > the fact that an user can send your closure or any instance of a class > loaded > by the anonymous loader to a lib that relies on getClassLoader(). Generally the purposes I have in mind for anonymously-loaded classes are entirely controlled: - As the body of a newly JITted method, only ever accessed from within the JRuby dispatch hierarchy - As a method handle, only ever accessed from within the JRuby dispatch hierarchy - As a compiled regex, only ever accessed from within the JOni regex library - As a "more" optimized call site, only ever accessed from the interpreter or compiled code I had never expected that anonymously loaded classes would be a general purpose way to create new interface impls or subclasses I'd pass out to arbitrary libraries. My interest in them (for now) is purely as lightweight "code boxes" for purposes I keep safely tucked away. - Charlie From forax at univ-mlv.fr Fri Apr 25 16:36:20 2008 From: forax at univ-mlv.fr (=?ISO-8859-1?Q?R=E9mi_Forax?=) Date: Sat, 26 Apr 2008 01:36:20 +0200 Subject: AnonymousClassLoader and ClassLoader In-Reply-To: <749B4C2D-CE3C-4642-9CD8-29BE542EF089@Sun.COM> References: <4810524A.40504@univ-mlv.fr> <9F9ADA1B-DAAE-415E-92BD-C2B7B425B2B4@sun.com> <4811FD12.7040901@univ-mlv.fr> <749B4C2D-CE3C-4642-9CD8-29BE542EF089@Sun.COM> Message-ID: <48126AF4.40809@univ-mlv.fr> John Rose a ?crit : > On Apr 25, 2008, at 8:47 AM, R?mi Forax wrote: > >> John Rose a ?crit : >>> On Apr 24, 2008, at 2:26 AM, R?mi Forax wrote: >>> >>>> In my opinion, getResource() is not the only method that makes this >>>> assumption >>> >>> The getResource method probably needs to be adjusted since anonymous >>> classes have names that are intentionally distorted. The anonymous >>> class name ends with "/${hashcode}" so that attempts to do forName >>> will rapidly fail, and so that it is clear which classes are anonymous. >> Are you sure hashcode are unique and never creates a name collisions ? > > No, there *will* be name collisions. The hashcode is to help with > debugging, and should be ignored by any code that is attempting to > assign a name to the anonymous class. (Everything after the slash > should be stripped.) Using the name in any way at all is a slippery > slope, but obtaining the resources from the class's formal name seems > like it's not too much of a hack. ok > >>> The logic for Class.resolveName needs to take this case into account >>> if resources are to work. (I assume that it is a good idea for them >>> to "work", but maybe it's not.) It's a good question, and a nice >>> experiment. >>> >>> -- John >> Perhaps it's a good idea to try to surface isAnonymousClass() a Java >> level >> or even better getHostClass() (or getAnonymousHostClass()). >> I will try to create a patch for that this week-end, i seems to be a >> good idea >> to try to understand the VM internals. > > Yes; you can see from the JVM patch that I'm on the edge of doing > this. The useful idea at present is that the class anonymous iff it > has a slash in its getName; it would be bad for it to have a legal > getName. hum, i've chosen another path. I've exported to the VM the instanceKlass.host_klass() at Java level. It takes me less than hour to figure how to do that and since that i know try to to find how to create a patch using mercurial queues knowing that i've forget to apply the last patch (callcc without anonk) before hacking the VM. It's half past one am here, i am going to rest and i will submit the patch tomorrow. > >> Another experiment, >> trying to resolve an anonymous class using ClassLoader.resolveClass() >> leads to a crash in SystemDictionnary. >> Step to reproduce, change sun.misc.Launcher by the one in attachement, >> I have chnage the application classloader to load all non bootstrap >> class as anonymous one >> (see findClass) but when it tries to call resolveClass() (in >> ClassLoader.loadClass()) on an anonymous class, >> the VM crashs. >> >> I don't know if trying to resolve an anonymous class is legal or not. > > No, it's not legal. It should throw a ClassNotFound or similar > exception. Should not crash, obviously. ok, i will investigate > >> The following lines are a cut&paste of the crash using the fastdebug >> VM (patched with anonk and callcc). > > Right, a failure like that is to be expected if the query gets that far. > > When you load everything as anonymous, how do you resolve intra-module > references? That seems hopeless. You can resolve some references by > using the CP patching feature, but that does not cover CP > circularities. We need a simultaneous loading model to create complex > graphs of anonymous classes. I don't see the need to do this yet. For me, it was just a test. I've created an empty class and tried to load all other classes as anonymous of this one so there is no reference between the empty class and each anonymous class. And between anonymous class, i've kept the mapping between name and anonymous class in a weak hashmap. > > (First figure out a multi-class loading model for normal classes! I > have some good ideas for this based on my work with Pack200, but they > are back-burnered because of more pressing features.) > > -- John R?mi From charles.nutter at sun.com Sat Apr 26 14:10:28 2008 From: charles.nutter at sun.com (Charles Oliver Nutter) Date: Sat, 26 Apr 2008 16:10:28 -0500 Subject: AnonymousClassLoader in JRuby Message-ID: <48139A44.8090002@sun.com> I've managed to wire the AnonymousClassLoader into JRuby. A patch is attached, that will probably be rather confusing to anyone unfamiliar with JRuby. ClassCache is basically a smart class-loading utility that can be used to share loaded classes across JRuby instances without keeping hard references to them and preventing them from unloading. AnonymousClassLoader is an all-static utility class that wraps the logic of instantiating "one-shot" classloaders for loading bodies of code. In this case, it either uses java.dyn.AnonymousClassLoader or our own "OneShotClassLoader" which is not anonymous but which is used for only a single class. The code generally appears to work about the same with java.dyn.AnonymousClassLoader available, which is good. But I'm having trouble quantifying the benefit to JRuby. I'd like to be able to show how it helps, but memory profiles look practically the same between the version using an OneShotClassLoader per method body and the version using AnonymousClassLoader per method body. I have not done a lot of work to isolate the cost of loading, but it seems to be unnoticeable in my simple benchmark which defines 100_000 Ruby methods and forces them to JIT. So the runtime benefits may not be so great. The practical benefits, such as being able to chuck byte[] into AnonymousClassLoader without decorating class names and ensuring uniqueness, I have not yet utilized in this code. Those benefits may show how writing class caches like the one in JRuby are made a lot easier. For the moment, in order to allow JRuby to support both MLVM work and pre-JDK7 JVMs, this patch still does name-mangling to ensure methods are unique. I think part of my confusion is that originally I desired a class loader for which I could have a single instance I would throw *many* byte[] at, and they'd all be loaded using that classloader but without hard references and without the overhead of a classloader-per-method. But the AnonymousClassLoader interface appears to one exactly one byte[] per instance, though it does have considerably reduced in-memory cost per AnonymousClassLoader instance. Basically, what I've been looking for to make my life easier and memory costs lower is: ClassLoader cl = someClassLoader....; Class first = cl.defineClass(firstClass); Class second = cl.defineClass(secondClass); first = null; // and at some point I would expect the class to GC Is this the purpose of AnonymousClassLoader? Am I doing it wrong? I'll be poking around for example code now. - Charlie -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: anonymous_class_loading_support.patch Url: http://mail.openjdk.java.net/pipermail/mlvm-dev/attachments/20080426/12d83c9d/attachment.ksh From charles.nutter at sun.com Sat Apr 26 14:30:10 2008 From: charles.nutter at sun.com (Charles Oliver Nutter) Date: Sat, 26 Apr 2008 16:30:10 -0500 Subject: AnonymousClassLoader in JRuby In-Reply-To: <48139A44.8090002@sun.com> References: <48139A44.8090002@sun.com> Message-ID: <48139EE2.50305@sun.com> Ok, reading back over some notes and anonk.txt I think I understand a bit better. Tell me if I'm understanding things right. java.dyn.AnonymousClassLoader is not a java.lang.ClassLoader itself. It bypasses the normal classloading chain by calling the new Unsafe method that allocates and loads a class directly. Given a byte[] and a host class, you can use AnonymousClassLoader to simply load a class; you can also use it to produce patched versions of a class, reducing the metadata necessary for each new class loaded by only altering the constant pool. The classes are not associated with a classloader, and so the normal overhead we face of having a classloader-per-class (aggravated by a class-per-method in JRuby) is reduced. From an API perspective, we've largely achieved the "GCable class loading" concept in JRuby out of necessity. But on JDK6 and lower we pay the cost of having all those classes holding on to a clasloader in memory. I have been unable to see an improved memory cost, but it's possible the actual cost of those classloaders in this test is outweighed by other JRuby runtime metadata associated with a given JITed method. Perhaps a simple Java test that loads 10_000 really simple classes via a OneShotClassLoader and via AnonymousClassLoader would be a better way to measure this? It's also possible that the ClassCache concept would be useful/important to include or publish along with AnonymousClassLoader, since I would expect that most people loading classes anonymously will want a way to cache and retrieve already-loaded versions of code they're trying to load. This also sees like it might be necessary since anonymous classes will be otherwise impossible to look up, forcing even the same code in different places to instantiate a new class. Any thoughts on this? ClassCache (unpatched) in its entirety is here: http://svn.codehaus.org/jruby/trunk/jruby/src/org/jruby/util/ClassCache.java FYI, in JRuby, the key we use is generated by rendering the Ruby AST into a string containing a sexp representation of the code. This allows multiple instances of JRuby using the same class cache to share each others anonymously-loaded method bodies. - Charlie Charles Oliver Nutter wrote: > I've managed to wire the AnonymousClassLoader into JRuby. A patch is > attached, that will probably be rather confusing to anyone unfamiliar > with JRuby. > > ClassCache is basically a smart class-loading utility that can be used > to share loaded classes across JRuby instances without keeping hard > references to them and preventing them from unloading. > > AnonymousClassLoader is an all-static utility class that wraps the logic > of instantiating "one-shot" classloaders for loading bodies of code. In > this case, it either uses java.dyn.AnonymousClassLoader or our own > "OneShotClassLoader" which is not anonymous but which is used for only a > single class. > > The code generally appears to work about the same with > java.dyn.AnonymousClassLoader available, which is good. But I'm having > trouble quantifying the benefit to JRuby. I'd like to be able to show > how it helps, but memory profiles look practically the same between the > version using an OneShotClassLoader per method body and the version > using AnonymousClassLoader per method body. I have not done a lot of > work to isolate the cost of loading, but it seems to be unnoticeable in > my simple benchmark which defines 100_000 Ruby methods and forces them > to JIT. > > So the runtime benefits may not be so great. The practical benefits, > such as being able to chuck byte[] into AnonymousClassLoader without > decorating class names and ensuring uniqueness, I have not yet utilized > in this code. Those benefits may show how writing class caches like the > one in JRuby are made a lot easier. For the moment, in order to allow > JRuby to support both MLVM work and pre-JDK7 JVMs, this patch still does > name-mangling to ensure methods are unique. > > I think part of my confusion is that originally I desired a class loader > for which I could have a single instance I would throw *many* byte[] at, > and they'd all be loaded using that classloader but without hard > references and without the overhead of a classloader-per-method. But the > AnonymousClassLoader interface appears to one exactly one byte[] per > instance, though it does have considerably reduced in-memory cost per > AnonymousClassLoader instance. > > Basically, what I've been looking for to make my life easier and memory > costs lower is: > > ClassLoader cl = someClassLoader....; > Class first = cl.defineClass(firstClass); > Class second = cl.defineClass(secondClass); > first = null; // and at some point I would expect the class to GC > > Is this the purpose of AnonymousClassLoader? Am I doing it wrong? I'll > be poking around for example code now. > > - Charlie > > > ------------------------------------------------------------------------ > > _______________________________________________ > mlvm-dev mailing list > mlvm-dev at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev From charles.nutter at sun.com Sat Apr 26 14:38:53 2008 From: charles.nutter at sun.com (Charles Oliver Nutter) Date: Sat, 26 Apr 2008 16:38:53 -0500 Subject: AnonymousClassLoader in JRuby In-Reply-To: <48139EE2.50305@sun.com> References: <48139A44.8090002@sun.com> <48139EE2.50305@sun.com> Message-ID: <4813A0ED.2070800@sun.com> Another thought occurs about how/where this would be useful. Consider it a "duh" moment for me, given that much of this work is being driven by JSR292... ACL will be incredibly useful for generating inexpensive method handles. Duh. So you have a template that represents the general wa