Integrating native Open GL code in JavaFX and alternatives
artem.ananiev at oracle.com
Mon Jan 28 07:16:42 PST 2013
On 1/26/2013 12:04 AM, Scott Palmer wrote:
> On 2013-01-25, at 2:16 PM, Artem Ananiev <artem.ananiev at oracle.com>
>> A few comments from my side:
>> 1. If a native Window/Scene handle is provided, it will be enough
>> to embed whatever you need. It will even be possible to embed D3D
>> content into JavaFX running with OpenGL pipeline :)
> There will be issues in JavaFX that we can't work around as easily as
> we code in Swing. For example clipping. How can my video preview
> overlay participate in a Scene such that I can put it inside a
> ScrollPane or SplitPane? With Swing I could manage this by placing a
> heavyweight component in the UI. Rectangular clipping was easy to
> deal with by wrapping in another heavyweight component - the OS
> windowing system took care of it. Later Swing got smarter about
> mixing heavy/light components. JavaFX doesn't have something
To use heavyweight component in a scroll/split pane, we need heavyweight
scroll/split pane :) In other words, such scenarios don't work well in
Do you mean limited heavyweight/lightweight component mixing, introduced
to 6u10 and the forward-ported to 7? Note the word "limited" here: the
mixing is really limited, it doesn't cover non-rectangular shapes,
transparent components, and many more. It actually works fine for
simplest scenarios only. The code we put to AWT to support the limited
mixing is not something I would like to see in JavaFX.
>> The question is what is the native handle? On Windows, HWND is the
>> answer. On Mac: NSView or CALayer?
> CALayer seems to be the way things are going. NSView was the Java 6
> native layer, now in Java 7 it is CALayer. But this is native code…
> let there be a way to get either from a platform specific structure.
Believe it or not, we will support both CALayer-based and NSView-based
embedded windows in JDK8 (and probably next JDK7 updates). Check the
following JDK bug for details:
Classes are not public (java.*, javax.*) APIs, though, but are supposed
to be backwards compatible. If we implement this functionality in
JavaFX, we'll likely support both CALayers and NSViews.
>> In the latter case we'll also need mechanism to flush embedded
>> layer content into the parent (window) layer, and the thing will
>> get complicated.
>> 2. Another challenge with heavyweight embedding is integration into
>> FX scene. Focus traversal, transforms, effects, modality, etc.
> Can be depending on the use case. For a video preview panel, I
> wouldn't' need most of that. #3 would be best.
>> 3. Alternatively, we can try to implement semi-heavyweight
>> embedding. What we need is to make Prism be able to render
>> externally managed textures (GL, D3D). Input events will be handled
>> as for any other FX nodes, transforms/effects will also be of no
>> problem. What's questionable here is a) performance b) sharing GL
>> context or D3D device between FX and application code. I'm not a
>> graphics expert, but suspect there may be problems in this area.
> I agree. But I think it is worth getting a graphics expert (I'm not
> a OpenGL/D3D person either) to think about it over the weekend :-)
>> On 1/25/2013 6:28 PM, Scott Palmer wrote:
>>> I see two basic needs:
>>> 1: A JAWT equivalent so we can at least get a handle to the Stage
>>> 2: Some means of embedding our own rendering surface. Likely via a
>>> "texture" described in the native format for the graphics pipeline
>>> used (OpenGL or Direct3D)
>>> In our Swing product (on Windows) we grab the native window handle of
>>> a heavyweight 'placholder' via JAWT and blit a bitmap directly to it
>>> - hardware assisted with a colorspace conversion, if I recall
>>> correctly. JavaFX offers no equivalent.
>>> I know that JavaFX will likely never allow the concept of embedding a
>>> heavyweight native window into a Scene. I don't think that would be
>>> the right approach for it anyway. But we can't even make a separate
>>> preview window natively and give it a proper parent window with the
>>> current APIs.
>>> On 2013-01-25, at 4:14 AM, Robert Krüger <krueger at lesspain.de>
>>>> just to avoid, this gets buried in the other long thread where it
>>>> originated from (Migrating commercial applications from Swing to JFX),
>>>> I'm posting this again as a separate thread. I hope this is OK.
>>>> We are evaluating migrating our Swing Application which includes a
>>>> video player built using native code for decoding, rendering and
>>>> effects to JavaFX. Performance requirements are very ambitious (like
>>>> being close to the playback performance of the best native player, in
>>>> our case Quicktime for most formats we care about, for high definition
>>>> Currently there is no way to do the same thing in JavaFX, which the
>>>> JOGL folks have already run into themselves (see
>>>> We would either need support for getting a native Open GL context to
>>>> draw to a JFX component (panel, canvas whatever), so we can use native
>>>> code to do all sorts of stuff that happens in our video player or we
>>>> would need support for the following operations in JFX:
>>>> - shaders for converting pixel formats (e.g. YUV422->RGB, YUV420->RGB ....)
>>>> - shaders and frame buffer objects (render to texture) for applying
>>>> layered effects to video frames
>>>> - pixel buffer objects for transferring video images to textures (and back)
>>>> - interoperability with opencl and cuda (processing of video frames
>>>> with opencl/cuda and rendering with opengl)
>>>> - reuse of opengl code across platforms (including android and iOS)
>>>> We achieve that currently by having cross-platform C code that is
>>>> connected to each platform via a rather thin layer for communication
>>>> (e.g. to be able implement callbacks for OpenGL rendering parameters
>>>> controlled by a Java-UI). My guess is that it is more likely that you
>>>> will be able to expose the native context than providing all the
>>>> listed things. It is also a maybe a philosophical question whether you
>>>> want that but IMHO you close the door on an interesting range of
>>>> applications if you don't provide anything like that in some way
>>>> because you cannot implement a serious media application of a certain
>>>> kind without it and why reinvent the wheel? IMHO Java technologies in
>>>> the multi-media area (e.g. JavaSound, JMF) have in many cases suffered
>>>> from being good ideas but not making it to a production-quality state
>>>> because doing all that would have meant reinventing the wheel, which
>>>> in complex areas like the one described here is simply not feasible so
>>>> they IMHO remained being mostly of educational value for academic
>>>> environments. Acknowledging that in some areas it does make sense to
>>>> have some native code doesn't diminish the value of Java as a
>>>> platform. Having a lot of portable code in an application is still a
>>>> lot better than having one application per platform. Quite the
>>>> contrary, to me it would make Java/JavaFX more grown-up in the desktop
>>>> area if that was supported some more.
>>>> Sorry for being a bit philosophical in the end ;-).
>>>> Thanks a lot for your help. I'm really hoping we can come to the
>>>> conclusion that migrating to JavaFX is the right way to go for us.
More information about the openjfx-dev