Porting JavaFX

ozemale at ozemail.com.au ozemale at ozemail.com.au
Mon Dec 3 11:49:50 PST 2012

Thanks for the detailed info Richard.

Are you in a position to state where you are up to in the iOS port? 
It sounds like there's a long way to go or have you got many of these
areas working already?



----- Original Message -----
From: "Richard Bair" 
To:"OpenJFX Mailing List" 
Sent:Mon, 3 Dec 2012 11:05:45 -0800
Subject:Porting JavaFX

 The post is intended to lay out the different activities that need to
be undertaken in order to do a port from JavaFX to any platform, but
with special emphasis on iOS. There are many, many details which will
only be discoverable and actionable once the code is all open sourced,
so this post itself is necessarily only high level and intended as
some kind of guide to ensuing discussions.

 There are, essentially, 3 things that must be done to port JavaFX to
any platform:
 * Port a JVM and the Java class libraries
 * Port JavaFX (Glass, and to some minimal extent, Prism)
 * Provide a toolchain for building apps for the target environment

 In the existing desktop ports, #3 is sort of taken for granted
because you have javac and such available to you. However as you
attempt to target app stores, the jfxpackager & app bundling becomes
much more important. On iOS, there is a lot here that has to be done,
providing "entitlements", code signing with an Apple certificate,
proper bundle generation, etc. There will most likely need to be some
toolchain interop with Xcode for example.

 *Port A JVM and the Java Class Libraries*

 This is the first major body of work. Getting a JVM that will run on
the target device is the first order of business. There are several
options available, although no official HotSpot support (yet) for iOS
Some open source VM options might include XMLVM or Avian. Any VM that
wants to have acceptable performance on iOS and WinRT will need to
support ahead-of-time (AOT) compilation since just-in-time (JIT)
compilation is prohibited on these platforms. [Aside: on iOS for
example it simply *is not possible* to mark a memory segment as
executable. That means that there is no way to have runtime code
generation. This is done for security reasons -- JIT security bugs in
fact do exist, the Safari JIT having had such bug publicly exposed a
year or so ago. Note that Apple allows Safari to JIT, but they aren't
going to let 3rd parties do the same). 

 The next step is getting a port of the Java class libraries. Some
things will just work -- anything that the JVM was responsible for
implementing, such as threading, or anything that is just plain Java
code. But many other things such as networking and imageIO, for
example, will need to be implemented. For iOS, starting from the
OpenJDK Mac code base might be a good start. You don't have to worry
about Swing / AWT since the basis we would use to target these devices
is JavaSE Embedded, not JavaSE itself, and I do not believe that
Swing/AWT are required for SE Embedded. But if you are really
adventurous, there is no reason (that I know of) that you couldn't get
all of JavaSE working since the Mac OS port of SE uses a lightweight
AWT (AWT components are emulated using Swing -- let that swirl in your
noodle for a while ;-)).

 For a quick and dirty hack, you can cut all kinds of corners here.
For an actual GA release, porting all the required SE libraries over
is an absolute necessity. This is, in my estimation, the most
difficult part of doing a port to iOS, but maybe that is just because
I'm not as familiar with what needs porting on SE vs. FX.

 *Port JavaFX*

 Porting JavaFX to a platform mostly involves a new port of Glass. On
all ports that we have now (and most of them that we've prototyped),
we reuse Prism and Glass and simply provide a new port of Glass. At
one point in the past, when Prism & Glass were still not GA quality,
we maintained a completely separate Toolkit implementation
(com.sun.javafx.tk.Toolkit) which was based on Swing / AWT This level
of abstraction we want to remove from the platform, such that the
Scene Graph classes (what I often call "the top half" -- meaning, the
top part of our cake diagram) will refer directly to Glass / Prism
classes ("the bottom half"), whereas now there are a series of
interfaces that separate the two.

 At this point, every port we're looking at will be a port of Glass /
Prism. Most of the work is in Glass. Glass is the "windowing layer",
and has all the code for handling the event queue, windows, views
(areas inside the window that get rendered), etc. There is a boundary
between Prism & Glass that is kind of messy which is responsible for
negotiating the creation of the View surface so that it is established
correctly for OpenGL. Otherwise Prism pretty much just works, because
we use OpenGL ES 2 for both desktop and mobile, and we have a pretty
good implementation that we use everywhere else.

 Of course, you could also provide a stylesheet that gave native looks
to iOS controls (not hard to do), and would be quite a nice addition
to the platform (now that Jasper has added public API for setting the
user agent stylesheet).

 Right now, Glass is designed such that there is a non-trivial amount
of native code involved in doing a port. Anybody with experience with
Objective-C and iOS should feel reasonably well at home in it. Ideally
we'd use a heck of a lot less native code to do a port, but for now,
what we have is what we have.


 I've about mentioned as much as I wanted to mention on toolchain. It
is an important part of the story, so as to make developers
productive. Ideally we want to allow people to develop on any platform
for any platform. Some work will need to be done to see if that is
even possible for iOS. Today every toolchain I know of for iOS
requires Xcode. Maybe you can figure out how to do the cryptographic
signing from Linux or Windows instead, and not require Xcode at all.

 Also, of course, creating a co-bundled application using a subset of
the JRE is critical to get sizes down to something respectable. I
believe 50mb is the limit for an application that wants to be
downloadable over-the-air. Obviously we want to keep the runtime to a
reasonable fraction of that to allow room for the application itself.


 On iOS, our existing prototypes don't have AOT. This has proven to be
a big barrier to performance. I actually was blown away at how fast
the hotspot interpreter was on iOS -- but no matter how fast it was,
it is really (*really*) hard to get smooth graphics performance
without AOT. We did do an amazing amount of performance work in this
area a few months ago which has benefited desktop & embedded as well
(often to a surprising degree). However there is only so much you can
do in an interpreter.

 On the other hand, AOT, we believe, when used on everything will
balloon the size of the application. So AOT needs to have some kind of
control of what is AOT'd. Critical sections of Prism and the Scene
Graph would need to be AOT'd to get optimal performance. We had an
example where we ran FX on normal iPhone and on a jail broken iPod
with JIT, and it was very fast on the iPod with JIT. We got to the
point where the interpreted JVM on iPhone was quite good even for the
schedule builder (which is reasonably complicated) -- the performance
(on interpreter!) was indistinguishable from native, but required
various performance tweaks and hacks in the application code.

 So although without a JIT with FX we could get good performance with
effort on the application developer's part (requiring some extensive
knowledge of the platform), with AOT, performance is good by default.

 Android was interesting. We had two ports, one that used the Android
graphics APIs and one that used Prism / Glass. The Prism/ Glass
version was at least as fast as the one using Android APIs. This
validated our notion of using Glass / Prism everywhere and removing
the Toolkit APIs as it was our feeling that we would not any longer
need this level of abstraction.

 *GWT Port*

 For what it is worth, a few years ago we prototyped several ports of
JavaFX using GWT to be able to run in a web browser without a plugin.
This is not likely to be something that *we* will do, because any
non-plugin solution for the web will not be "real" Java -- it cannot
properly support threading and many other things that are integral to
a real WORA solution around Java. Some solutions such as Emscripten
are very interesting to me, which basically is a VM that JIT's into
JavaScript (!!) I've long dreamt about having a JVM implemented in
JavaScript such that we could run real Java in the browser without a
plugin. Of course, performance is generally abysmal because you are
interpreting Java in JavaScript. However with something like
Emscripten, if we were very, very clever, you might get "browser
native" performance by translating Java byte codes into JavaScript
code which would then be handled natively by the browser. Key
problems: threading & memory management (missing PhantomReference,
WeakReference, etc support in JavaScript).

 However, even if *we* are not going to support GWT-like solution,
there is no reason that the open source community could not do so.

 Our solution for this explored a few options. The idea was that much
of the public API would be able to translate directly to JavaScript
via GWT. Also the idea was that the Toolkit interface allowed us to
virtualize the threading rules, such that you could have an
implementation where the UI thread and render thread were the same
thing, or different things. And in fact we will try to preserve this
as we remove the Toolkit abstraction, such that a JavaME port (for
example) or a GWT port could continue to work. Since there is not good
threading support in JavaScript, we need all of the FX code to run in
the same thread.

 There would no doubt need to be patches to enable this kind of
implementation, and I'd be view such attempts very favorably.

 We had an implementation that used SVG for rendering, an
implementation using DOM + Canvas, and DOM + WebGL. There were
problems with all attempts, but I think each worked reasonably well
for many things. Typical problem areas are effects, editable text, and
media. If you could run most of Glass / Prism then using WebGL at the
very bottom might work, but performance isn't going to be all that
grand. Performance of animations is going to be problematic unless you
can leverage CSS animations in some way. There are a lot of issues

 That's it for now.

More information about the openjfx-dev mailing list