From christian at schlichtherle.de  Mon Jun  6 02:36:56 2011
From: christian at schlichtherle.de (Christian Schlichtherle)
Date: Mon, 6 Jun 2011 11:36:56 +0200
Subject: URI issue
Message-ID: <001a01cc242d$47209e30$d561da90$@de>

Hi Alan,

I just came across an issue with the URIs created by Paths.get(String first,
String. more).

On Windows if the current directory is C:/foo, then

assert "file:///C:/".equals(Paths.get("/").toUri().toString());
assert "file:///C:/foo/".equals(Paths.get("").toUri().toString());

Now the use of the empty authority is problematic because the URI class has
several issues with it:

assert null == URI.create("file:///C:/foo/").getAuthority(); // expected ""
!
assert
"file:/C:/foo/bar".equals(URI.create("file:///C:/foo/").resolve("bar").toStr
ing()); // expected "file:///C:/foo/bar"

As you can see, the URI class "optimizes" away the superfluous empty
authority returned by Paths.get(*).

However, this violates its identity provision that for any hierarchical URI
u,

assert u.equals(new URI(u.getScheme(), u.getAuthority(), u.getPath(),
u.getQuery(), u.getFragment())); // will fail with URI.create("file:///C:/")

See http://download.oracle.com/javase/7/docs/api/java/net/URI.html

All in all, the superfluous empty authority should be avoided in order not
to get in trouble. I came across this in the unit tests for my
TFileSystemProvider class. My workaround for now is to rewrite the URI
returned by Paths.get(*).


Regards,
Christian





From christian at schlichtherle.de  Mon Jun  6 03:10:21 2011
From: christian at schlichtherle.de (Christian Schlichtherle)
Date: Mon, 6 Jun 2011 12:10:21 +0200
Subject: Dynamic loading of FileSystemProvider instances
Message-ID: <002001cc2431$f23a5b90$d6af12b0$@de>

Hi Alan,

we recently discussed that I have a need to determine the NIO.2
FileSystemProvider instances at run time. Because this is not possible, you
suggested I should introduce one VFS scheme for my TrueZIP
TFileSystemProvider. For example, the NIO.2 Path URI "tzp:/archive.zip"
should get identified by my TrueZIP TFilesystemProvider and internally
rewritten to the TrueZIP FsPath URI "zip:file:/archive.zip!/".

While this is certainly possible, it still limits my implementation: The
issue is that I need to determine a TrueZIP FsMountPoint URI for rewriting
the NIO.2 Path URI. In the example above, this is "file:/". However, on
Windows, it could be "file:/C:/" or "file:/D:/" or whatever. In TrueZIP, it
could even be "http://somehost/" or "http://someotherhost/base/" or
whatever. It could even be an entirely different scheme, depending on the
file system driver.

To overcome this, I need a way to determine the NIO.2 FileSystemProvider
instances at run time. To make minimal impact on the current NIO.2 API, I
suggest the following quick draft:

<code>
package java.nio.file.spi;

public interface FileSystemProviderCatalogue extends
Iterable<FileSystemProvider> {
}
</code>

You could then simply use java.util.ServiceLoader to locate
FileSystemProviderCatalogue instances on the run time class path while I
could happily implement the iterator to figure the TFileSystemProvider
instances I need.

Regards,
Christian



From Alan.Bateman at oracle.com  Mon Jun  6 10:23:33 2011
From: Alan.Bateman at oracle.com (Alan Bateman)
Date: Mon, 06 Jun 2011 18:23:33 +0100
Subject: URI issue
In-Reply-To: <001a01cc242d$47209e30$d561da90$@de>
References: <001a01cc242d$47209e30$d561da90$@de>
Message-ID: <4DED0D15.3000009@oracle.com>

Christian Schlichtherle wrote:
> Hi Alan,
>
> I just came across an issue with the URIs created by Paths.get(String first,
> String. more).
>
> On Windows if the current directory is C:/foo, then
>
> assert "file:///C:/".equals(Paths.get("/").toUri().toString());
> assert "file:///C:/foo/".equals(Paths.get("").toUri().toString());
>
> Now the use of the empty authority is problematic because the URI class has
> several issues with it:
>
> assert null == URI.create("file:///C:/foo/").getAuthority(); // expected ""
> !
> assert
> "file:/C:/foo/bar".equals(URI.create("file:///C:/foo/").resolve("bar").toStr
> ing()); // expected "file:///C:/foo/bar"
>
> As you can see, the URI class "optimizes" away the superfluous empty
> authority returned by Paths.get(*).
>   
If URIs u1 and u2 are equal then it doesn't mean that their toString 
methods will yield equal Strings. I think this is the issue that you are 
running into it. In the above then file:/C:/foo/bar is equal to 
file:///C:/foo/bar but because of the way the URIs are created, their 
toString methods yield non equal Strings.

In any case, the reason that the default provider creates the URI from 
the string "file:///C:/foo/" is so that it is interoperable with native 
applications and follows the syntax for URIs on Windows.

-Alan.

From Alan.Bateman at oracle.com  Mon Jun  6 11:40:04 2011
From: Alan.Bateman at oracle.com (Alan Bateman)
Date: Mon, 06 Jun 2011 19:40:04 +0100
Subject: Dynamic loading of FileSystemProvider instances
In-Reply-To: <002001cc2431$f23a5b90$d6af12b0$@de>
References: <002001cc2431$f23a5b90$d6af12b0$@de>
Message-ID: <4DED1F04.10507@oracle.com>

Christian Schlichtherle wrote:
> Hi Alan,
>
> we recently discussed that I have a need to determine the NIO.2
> FileSystemProvider instances at run time. 
I think that discussion was more that there isn't a method to add to the 
list of installed providers at runtime (the list returned by 
FileSystemProvider.installedProviders is created at startup and there 
isn't a method to register/add to it). You can of course use 
ServiceLoader.load(FileSystemProvider.class, loader) to locate 
additional providers yourself.

> Because this is not possible, you
> suggested I should introduce one VFS scheme for my TrueZIP
> TFileSystemProvider. 
That suggestion wasn't exactly related but rather that if you are 
developing a virtual file system then you need to choose a (singular) 
URI scheme in order to deploy as a provider. That provider can of course 
delegate to other providers, which I think is what you are doing.

> For example, the NIO.2 Path URI "tzp:/archive.zip"
> should get identified by my TrueZIP TFilesystemProvider and internally
> rewritten to the TrueZIP FsPath URI "zip:file:/archive.zip!/".
>
> While this is certainly possible, it still limits my implementation: The
> issue is that I need to determine a TrueZIP FsMountPoint URI for rewriting
> the NIO.2 Path URI. In the example above, this is "file:/". However, on
> Windows, it could be "file:/C:/" or "file:/D:/" or whatever. In TrueZIP, it
> could even be "http://somehost/" or "http://someotherhost/base/" or
> whatever. It could even be an entirely different scheme, depending on the
> file system driver.
>
> To overcome this, I need a way to determine the NIO.2 FileSystemProvider
> instances at run time. To make minimal impact on the current NIO.2 API, I
> suggest the following quick draft:
>
> <code>
> package java.nio.file.spi;
>
> public interface FileSystemProviderCatalogue extends
> Iterable<FileSystemProvider> {
> }
> </code>
>
> You could then simply use java.util.ServiceLoader to locate
> FileSystemProviderCatalogue instances on the run time class path while I
> could happily implement the iterator to figure the TFileSystemProvider
> instances I need.
>   
I don't think I fully understand it. If there are providers on the class 
path then they should be returned by the 
FileSystemProvider.installedProviders method.

-Alan

From ballav.bihani at oracle.com  Mon Jun  6 12:00:39 2011
From: ballav.bihani at oracle.com (ballav.bihani at oracle.com)
Date: Mon, 6 Jun 2011 12:00:39 -0700 (PDT)
Subject: Auto Reply: nio-discuss Digest, Vol 33, Issue 1
Message-ID: <9d2d53ab-8ac0-46e6-b710-d754131463ac@default>

I am on leave till 06/16.  For all technical matters, please contact my manager Dhiraj Mutreja (dhiraj.mutreja at oracle.com)


From christian at schlichtherle.de  Mon Jun  6 13:17:59 2011
From: christian at schlichtherle.de (Christian Schlichtherle)
Date: Mon, 6 Jun 2011 22:17:59 +0200
Subject: AW: Dynamic loading of FileSystemProvider instances
In-Reply-To: <4DED1F04.10507@oracle.com>
References: <002001cc2431$f23a5b90$d6af12b0$@de> <4DED1F04.10507@oracle.com>
Message-ID: <002c01cc2486$d527e180$7f77a480$@de>

> I don't think I fully understand it. If there are providers on the
> class
> path then they should be returned by the
> FileSystemProvider.installedProviders method.

This method uses a ServiceLoader which loads the FileSystemProvider class
names from a resource on the class path and instantiates one instance for
each line entry, right?

Well, that's the problem: The resource is created at compile time, so
whatever I put in there I have no chance to determine the number of
FileSystemProvider instances at run time.

However, I really need a way to do this because the FileSystemProvider class
is limited to exactly one scheme. However, TrueZIP supports an arbitrary
number of schemes and I cannot predetermine them in a reliable way. For
example, at run time I might figure that I need to have a FileSystemProvider
for the following schemes:

"tzp.file" for mapping Path URI "file:/foo.zip" to FsPath URI
"zip:file:/foo.zip!/".

"tzp.file.c" for mapping "file:///C:/foo.tar" to "tar:file:/C:/foo.tar!/"

"tzp.file.d" for mapping "file:///D:/foo.tar.bz2" to
"tar.bz2:file:/D:/foo.tar.bz2!/"

"tzp.http" for mapping "http://download.oracle.com/everyting.zip" to
"zip:http://download.oracle.com/everything.zip!/"

"tzp.https" for mapping "https://download.oracle.com/everyting.zip" to
"zip:https://download.oracle.com/everything.zip!/"

etc etc...

I hope this makes it clear now. This is really an important issue. I need to
have a chance to figure the number of FileSystemProvider instances at run
time, otherwise TrueZIP's FileSystemProvider implementation would be
seriously crippled because I can use only one scheme per FileSystemProvider
instance.

Regards,
Christian




From Alan.Bateman at oracle.com  Tue Jun  7 01:27:58 2011
From: Alan.Bateman at oracle.com (Alan Bateman)
Date: Tue, 07 Jun 2011 09:27:58 +0100
Subject: AW: Dynamic loading of FileSystemProvider instances
In-Reply-To: <002c01cc2486$d527e180$7f77a480$@de>
References: <002001cc2431$f23a5b90$d6af12b0$@de> <4DED1F04.10507@oracle.com>
	<002c01cc2486$d527e180$7f77a480$@de>
Message-ID: <4DEDE10E.6040807@oracle.com>

Christian Schlichtherle wrote:
> :
> This method uses a ServiceLoader which loads the FileSystemProvider class
> names from a resource on the class path and instantiates one instance for
> each line entry, right?
>   
Yes, that's right.

> Well, that's the problem: The resource is created at compile time, so
> whatever I put in there I have no chance to determine the number of
> FileSystemProvider instances at run time.
>
> However, I really need a way to do this because the FileSystemProvider class
> is limited to exactly one scheme. However, TrueZIP supports an arbitrary
> number of schemes and I cannot predetermine them in a reliable way. For
> example, at run time I might figure that I need to have a FileSystemProvider
> for the following schemes:
>
> "tzp.file" for mapping Path URI "file:/foo.zip" to FsPath URI
> "zip:file:/foo.zip!/".
>
> "tzp.file.c" for mapping "file:///C:/foo.tar" to "tar:file:/C:/foo.tar!/"
>
> "tzp.file.d" for mapping "file:///D:/foo.tar.bz2" to
> "tar.bz2:file:/D:/foo.tar.bz2!/"
>
> "tzp.http" for mapping "http://download.oracle.com/everyting.zip" to
> "zip:http://download.oracle.com/everything.zip!/"
>
> "tzp.https" for mapping "https://download.oracle.com/everyting.zip" to
> "zip:https://download.oracle.com/everything.zip!/"
>
> etc etc...
>
> I hope this makes it clear now. This is really an important issue. I need to
> have a chance to figure the number of FileSystemProvider instances at run
> time, otherwise TrueZIP's FileSystemProvider implementation would be
> seriously crippled because I can use only one scheme per FileSystemProvider
> instance.
>
>   
I don't think I understand this. It looks like you are dynamically 
generate a URI scheme per mount point and it's not clear (to me anyway) 
if this means you have a file system provider per mount point (which 
doesn't make sense). Are these URIs internal to your provider as they 
doesn't seem to be something that should be exposed to anyone using this 
API?

-Alan.

From christian at schlichtherle.de  Tue Jun  7 02:39:55 2011
From: christian at schlichtherle.de (Christian Schlichtherle)
Date: Tue, 7 Jun 2011 11:39:55 +0200
Subject: AW: AW: Dynamic loading of FileSystemProvider instances
In-Reply-To: <4DEDE10E.6040807@oracle.com>
References: <002001cc2431$f23a5b90$d6af12b0$@de> <4DED1F04.10507@oracle.com>
	<002c01cc2486$d527e180$7f77a480$@de> <4DEDE10E.6040807@oracle.com>
Message-ID: <003801cc24f6$dca44690$95ecd3b0$@de>

> I don't think I understand this. It looks like you are dynamically
> generate a URI scheme per mount point and it's not clear (to me anyway)
> if this means you have a file system provider per mount point (which
> doesn't make sense). Are these URIs internal to your provider as they
> doesn't seem to be something that should be exposed to anyone using
> this
> API?

First, I need to differentiate between Path URIs (those returned by
Path.toUri()) and FsPath URIs (a class in the TrueZIP Kernel). 

This seems to be required because Path URIs need to be hierarchical and each
FileSystemProvider can use only one scheme which is determined at compile
time. An example URI could be "tzp:/archive.zip".

In contrast, FsPath URIs are opaque and the available schemes are figured by
locating file system driver service instances on the class path (by means of
ServiceLoader as you do). The corresponding FsPath URI for the Path URI
above could be "zip:file:/archive.zip!/". However, another meaningful
mapping could be "zip:http://somehost/archive.zip!/", too.

This mapping needs to be done in FileSystemProvider.newFileSystem(URI, Map)
and FileSystemProvider.newFileSystem(Path, Map). Now how do I tell the
FileSystemProvider to do the mapping? It seems reasonable to create
different FileSystemProvider instances for different "realm"s, i.e. root
mount points.

For example, if I had a FileSystemProvider which maps the Path URI scheme
"tzp" to the FsPath URI "file:///C:/", then the FileSystemProvider would
create a FileSystem for the Path URI "tzp:/archive.zip" (FsPath URI
"zip:file:///C:/archive.zip!/"), another one for "tzp:/archive.tar.gz"
(FsPath URI "tar.gz:file:///C:/archive.tar.gz!/") and another one for
"tzp:/dist.tar.gz/app.jar" (FsPath URI
"zip:tar.gz:file:///C:/dist.tar.gz!/app.jar!/").

Another FileSystemProvider could map everything to the root mount point
"http://somehost/" for the same Path URIs.

I don't see how else it could work.

Then I need a way to figure which FileSystemProvider instances I need at run
time. I would enumerate all file system drivers and filter for the
non-federated file system drivers. If I find a non-federated file system
driver for the FsPath URI scheme "file", I might install a
FileSystemProvider for mapping from Path URI schemes "tzp" (default) and
"tzp.file". If I find a non-federated file system driver for the FsPath URI
scheme "http", then I might install a FileSystemProvider "tzp.http". If I
find a non-federated file system driver for FsPath URI scheme "https", the I
might install a FileSystemProvider "tzp.https", and so on.

I'm open for better ideas, but this is the only option I see to support the
dynamic discovery of file system drivers with FileSystemProvider instances.

Best regards,
Christian


From Alan.Bateman at oracle.com  Tue Jun  7 04:19:22 2011
From: Alan.Bateman at oracle.com (Alan Bateman)
Date: Tue, 07 Jun 2011 12:19:22 +0100
Subject: AW: AW: Dynamic loading of FileSystemProvider instances
In-Reply-To: <003801cc24f6$dca44690$95ecd3b0$@de>
References: <002001cc2431$f23a5b90$d6af12b0$@de> <4DED1F04.10507@oracle.com>
	<002c01cc2486$d527e180$7f77a480$@de> <4DEDE10E.6040807@oracle.com>
	<003801cc24f6$dca44690$95ecd3b0$@de>
Message-ID: <4DEE093A.3020607@oracle.com>

Christian Schlichtherle wrote:
> :
> First, I need to differentiate between Path URIs (those returned by
> Path.toUri()) and FsPath URIs (a class in the TrueZIP Kernel). 
>
> This seems to be required because Path URIs need to be hierarchical and each
> FileSystemProvider can use only one scheme which is determined at compile
> time. An example URI could be "tzp:/archive.zip".
>   
Just on the hierarchal vs. opaque issue - this was the spec bug that 
came up in the other thread. We've just fixed it for b145. The fix just 
changes the wording to make it clear that the returned URI is absolute 
(meaning it has a scheme). It's only the default provider where the URIs 
are required to be hierarchal. Hopefully this makes things easier for 
you to plug in some of your existing URIs (under covers anyway, I don't 
think these should be exposed to users of the API).

-Alan.

From christian at schlichtherle.de  Tue Jun  7 09:44:46 2011
From: christian at schlichtherle.de (Christian Schlichtherle)
Date: Tue, 7 Jun 2011 18:44:46 +0200
Subject: Source and Target 1.5
Message-ID: <005601cc2532$36527d70$a2f77850$@de>

Is it me doing something wrong or is it not possible to compile code against
JDK 1.7.0 with source and target set to 1.5? I would like to be able to
figure the environment at runtime. So far I have been happily using source
and target 1.6, but when trying 1.5, my unit tests fail with a verify error.

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/nio-discuss/attachments/20110607/cfe310c8/attachment.html 

From christian at schlichtherle.de  Thu Jun  9 10:58:52 2011
From: christian at schlichtherle.de (Christian Schlichtherle)
Date: Thu, 9 Jun 2011 19:58:52 +0200
Subject: FSP TCK?
Message-ID: <003101cc26ce$e4be58a0$ae3b09e0$@de>

Hi,

 

I wonder if there is a TCK for my FileSystemProvider implementation?

 

Regards,

Christian

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/nio-discuss/attachments/20110609/00bada18/attachment.html 

From Alan.Bateman at oracle.com  Thu Jun  9 11:21:24 2011
From: Alan.Bateman at oracle.com (Alan Bateman)
Date: Thu, 09 Jun 2011 19:21:24 +0100
Subject: FSP TCK?
In-Reply-To: <003101cc26ce$e4be58a0$ae3b09e0$@de>
References: <003101cc26ce$e4be58a0$ae3b09e0$@de>
Message-ID: <4DF10F24.3080804@oracle.com>

Christian Schlichtherle wrote:
>
> Hi,
>
>  
>
> I wonder if there is a TCK for my FileSystemProvider implementation?
>
No, there isn't unfortunately. It would be nice to have some test suite 
to check provider implementations but it does not exist. The unit tests 
for the default provider are in the jdk repository so you may be able to 
modify them (although they test behavior that is specific to the default 
provider).

-Alan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/nio-discuss/attachments/20110609/60b7a805/attachment.html 

From christian at schlichtherle.de  Thu Jun  9 20:06:41 2011
From: christian at schlichtherle.de (Christian Schlichtherle)
Date: Fri, 10 Jun 2011 05:06:41 +0200
Subject: TrueZIP 7.2-beta-1 with NIO.2 FileSystemProvider
Message-ID: <005e01cc271b$6cae6330$460b2990$@de>

Hi everyone,

 

this is a quick note to let you know that I have released a first beta
version of TrueZIP 7.2 to Maven Central.

 

Hot topic: TrueZIP now ships with the new module TrueZIP Path which contains
a FileSystemProvider implementation for JSE 7! This means you can use the
new features of NIO.2 with TPath, TFileSystem and TFileSystemProvider
implementations on top of the TrueZIP Kernel. The TrueZIP kernel has been
extended to support the new features in NIO.2, e.g. SeekableByteChannel and
the different access time types in BasicFileAttributes.

 

Also new: There is an alternate implementation of the TrueZIP Driver FILE
module, named TrueZIP Driver FILE JSE7. These modules provide a file system
driver service for the "file" scheme. The new file system driver in TrueZIP
Driver FILE JSE7 uses the default FileSystemProvider of the NIO.2 API. If
both file system drivers are available on the run time class path, then the
kernel will use the file system driver's priority property to figure which
driver is the best to use.

 

This architecture enables applications to use the JSE7 default
FileSystemProvider and the TrueZIP TFileSystemProvider concurrently. When
the TrueZIP TFileSystemProvider is used, but no archive file is accessed,
then the TrueZIP Kernel module will route all I/O calls to the TrueZIP
Driver FILE JSE7 module which will route them to the JSE7 default
FileSystemProvider.

 

This means you can even use SeekableByteChannels for randomly reading and
writing archive file entries. The TrueZIP Kernel will buffer archive
entries using a write-back caching strategy whenever necessary.

 

BTW: TrueZIP still works with JSE6. Only the NIO.2 features will not be
available. 

 

To get you started, I have set up a new Maven archetype:

 

Group Id: de.schlichtherle.truezip

Artifact Id: truezip-archetype-path

Version: 7.2-beta-1

 

The archetype for TrueZIP File* module applications has been updated, too.

 

Group Id: de.schlichtherle.truezip

Artifact Id: truezip-archetype-file

Version: 7.2-beta-1

 

For instructions how to use these archetypes, please refer to the kick-start
articles on the TrueZIP User website at:
<http://truezip.java.net/kick-start/index.html>
http://truezip.java.net/kick-start/index.html

 

Enjoy TrueZIP!

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/nio-discuss/attachments/20110610/85309211/attachment.html 

From christian at schlichtherle.de  Fri Jun 10 02:16:38 2011
From: christian at schlichtherle.de (Christian Schlichtherle)
Date: Fri, 10 Jun 2011 11:16:38 +0200
Subject: AW: TrueZIP 7.2-beta-1 with NIO.2 FileSystemProvider
Message-ID: <007901cc274f$1b271eb0$51755c10$@de>

Hi everyone,

 

there was an issue with the Maven archetype's POM which has been fixed. The
updated Maven archetype coordinates are:

 

Group Id: de.schlichtherle.truezip

Artifact Id: truezip-archetype-path

Version: 7.2-beta-3

 

. for the Path API (TPath, TFileSystem and TFileSystemProvider), and.

 

Group Id: de.schlichtherle.truezip

Artifact Id: truezip-archetype-file

Version: 7.2-beta-3

 

. for the File* API (TFile, TFileInputStream, TFileOutputStream etc.)

 

Note there is no 7.2-beta-2 release - sorry for the noise.

 

Regards,

Christian

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/nio-discuss/attachments/20110610/0b79dcf8/attachment.html 

From Alan.Bateman at oracle.com  Fri Jun 10 07:04:10 2011
From: Alan.Bateman at oracle.com (Alan Bateman)
Date: Fri, 10 Jun 2011 15:04:10 +0100
Subject: TrueZIP 7.2-beta-1 with NIO.2 FileSystemProvider
In-Reply-To: <005e01cc271b$6cae6330$460b2990$@de>
References: <005e01cc271b$6cae6330$460b2990$@de>
Message-ID: <4DF2245A.1090107@oracle.com>

Christian Schlichtherle wrote:
>
> Hi everyone,
>
>  
>
> this is a quick note to let you know that I have released a first beta 
> version of TrueZIP 7.2 to Maven Central.
>
Good to hear that you've make progress that you've got a first beta 
available.

I tried it. Some things work, some things are clearly not implemented 
but it's a good start. The main thing that I ran into is that the Path 
object returned by FileSystem's getPath doesn't work as expected and you 
need construct a de.schlichtherle.truezip.nio.fsp.TPath yourself via the 
public constructor. I assume you are working on this as it doesn't be 
necessary to import anything from this package to use the API.

-Alan.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/nio-discuss/attachments/20110610/57d2a63d/attachment.html 

From christian at schlichtherle.de  Fri Jun 10 13:37:21 2011
From: christian at schlichtherle.de (Christian Schlichtherle)
Date: Fri, 10 Jun 2011 22:37:21 +0200
Subject: AW: TrueZIP 7.2-beta-1 with NIO.2 FileSystemProvider
In-Reply-To: <4DF2245A.1090107@oracle.com>
References: <005e01cc271b$6cae6330$460b2990$@de> <4DF2245A.1090107@oracle.com>
Message-ID: <000f01cc27ae$36375350$a2a5f9f0$@de>

Hi Alan,

 

thanks. I'm working on the issues you found in your first email. These are
supposedly bugs. So yes, a user can choose to stick with the pure NIO.2 API
and does NOT NEED to have a compile time dependency on the TrueZIP Path
module. This is my objective.

 

However, I do not agree that a client application SHOULD NOT have a compile
time dependency on the TrueZIP Path module. There are some features which I
plan to support which simply can't be done with the NIO.2 API. My primary
concern is that with Paths.get(URI) you can only use the URI scheme
associated with the FileSystemProvider at compile time - we discussed this
at lengths before.

 

Regards,

Christian

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/nio-discuss/attachments/20110610/0853638e/attachment.html 

From Alan.Bateman at oracle.com  Fri Jun 10 13:51:31 2011
From: Alan.Bateman at oracle.com (Alan Bateman)
Date: Fri, 10 Jun 2011 21:51:31 +0100
Subject: AW: TrueZIP 7.2-beta-1 with NIO.2 FileSystemProvider
In-Reply-To: <000f01cc27ae$36375350$a2a5f9f0$@de>
References: <005e01cc271b$6cae6330$460b2990$@de> <4DF2245A.1090107@oracle.com>
	<000f01cc27ae$36375350$a2a5f9f0$@de>
Message-ID: <4DF283D3.8030005@oracle.com>

Christian Schlichtherle wrote:
>
> Hi Alan,
>
>  
>
> thanks. I'm working on the issues you found in your first email. These 
> are supposedly bugs. So yes, a user can choose to stick with the pure 
> NIO.2 API and does NOT NEED to have a compile time dependency on the 
> TrueZIP Path module. This is my objective.
>
>  
>
> However, I do not agree that a client application SHOULD NOT have a 
> compile time dependency on the TrueZIP Path module. There are some 
> features which I plan to support which simply can't be done with the 
> NIO.2 API. My primary concern is that with Paths.get(URI) you can only 
> use the URI scheme associated with the FileSystemProvider at compile 
> time - we discussed this at lengths before.
>
Okay, just mostly pointing that in the current beta that the Path that 
is returned from the FileSystem's getPath method has a few issues but 
they work if TPath objects are created directly. If applications need to 
get to provider specific features then they will of course have a 
dependency on that provider.

-Alan.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/nio-discuss/attachments/20110610/e4bf7702/attachment.html 

From christian at schlichtherle.de  Fri Jun 10 14:10:37 2011
From: christian at schlichtherle.de (Christian Schlichtherle)
Date: Fri, 10 Jun 2011 23:10:37 +0200
Subject: AW: TrueZIP 7.2-beta-1 with NIO.2 FileSystemProvider
In-Reply-To: <4DF2245A.1090107@oracle.com>
References: <005e01cc271b$6cae6330$460b2990$@de> <4DF2245A.1090107@oracle.com>
Message-ID: <002301cc27b2$e543a7a0$afcaf6e0$@de>

Hi Alan,

 

I found some minor quirks in the NIO.2 API. I wonder if it isn't too late to
discuss API changes anyway but AFAIK the JSR 203 is not yet final?!

 

1.       FileSystem.getSeparator() should be Path.getSeparator(): After all
the, the separator is part of the addressing scheme, not the file system.
For example, when wirting a "copy constructor" which accepts another Path
instance as a parameter I had to write
path.toString().replace(path.getFileSystem().getSeparator(), SEPARATOR))
rather than the more natural path.toString().replace(path.getSeparator(),
SEPARATOR)).

2.       Files.createFile(Path, FileAttribute<?>.) uses
FileSystemProvider.newByteChannel() rather than
FileSystemProvider.newOutputStream(). If Path is a TPath and the addressed
entry is an archive entry, this would require the TrueZIP Kernel to create a
cache entry for the archive entry so that you could do random I/O. However,
you're just closing the channel immediately. This results in the most
expensive way to create a new archive entry in TrueZIP.

3.       Path.getFileName() returns a Path rather than just a String. Well,
that's strange - just consider Path a = Paths.get("foo", "bar"); Path b =
a.getFileName(); Now what is supposed to happen if I provide these two
objects to the Files class. Will I get the same results? I could argue that
this would be desirable because after all the two objects are addressing the
same file system entity, just with different labels. However, the
WindowsPath implementation does not act like this. The Javadoc for
Path.getFileName() does not specify this. I think this could get avoided if
Path.getFileName() were just returning a String.

4.       Why are all these I/O methods like newByteChannel(),
newInputStream() and newOutputStream() in FileSystemProvider? I think
FileSystem would be a better match.

 

Anyway, I guess it's way too late to change anything in the public API. You
would just p*** off too many people. But you could still change the
implementation for point 2 and amend the Javadoc for point 3.

 

Regards,

Christian

 

Von: Alan Bateman [mailto:Alan.Bateman at oracle.com] 
Gesendet: Freitag, 10. Juni 2011 16:04
An: christian at schlichtherle.de
Cc: nio-discuss at openjdk.java.net
Betreff: Re: TrueZIP 7.2-beta-1 with NIO.2 FileSystemProvider

 

Christian Schlichtherle wrote: 

Hi everyone,

 

this is a quick note to let you know that I have released a first beta
version of TrueZIP 7.2 to Maven Central.

Good to hear that you've make progress that you've got a first beta
available.

I tried it. Some things work, some things are clearly not implemented but
it's a good start. The main thing that I ran into is that the Path object
returned by FileSystem's getPath doesn't work as expected and you need
construct a de.schlichtherle.truezip.nio.fsp.TPath yourself via the public
constructor. I assume you are working on this as it doesn't be necessary to
import anything from this package to use the API. 

-Alan.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/nio-discuss/attachments/20110610/19a4bb7f/attachment.html 

From Alan.Bateman at oracle.com  Fri Jun 10 14:49:20 2011
From: Alan.Bateman at oracle.com (Alan Bateman)
Date: Fri, 10 Jun 2011 22:49:20 +0100
Subject: AW: TrueZIP 7.2-beta-1 with NIO.2 FileSystemProvider
In-Reply-To: <002301cc27b2$e543a7a0$afcaf6e0$@de>
References: <005e01cc271b$6cae6330$460b2990$@de> <4DF2245A.1090107@oracle.com>
	<002301cc27b2$e543a7a0$afcaf6e0$@de>
Message-ID: <4DF29160.3090603@oracle.com>

Christian Schlichtherle wrote:
>
> Hi Alan,
>
>  
>
> I found some minor quirks in the NIO.2 API. I wonder if it isn't too 
> late to discuss API changes anyway but AFAIK the JSR 203 is not yet 
> final?!
>
>  
>
> 1.       FileSystem.getSeparator() should be Path.getSeparator(): 
> After all the, the separator is part of the addressing scheme, not the 
> file system. For example, when wirting a "copy constructor" which 
> accepts another Path instance as a parameter I had to write 
> path.toString().replace(path.getFileSystem().getSeparator(), 
> SEPARATOR)) rather than the more natural 
> path.toString().replace(path.getSeparator(), SEPARATOR)).
>
The FileSystem getSeparator method gives you the separator for all Paths 
that are associated with the FileSystem. It might be strange to have it 
as a property of the Path as that leads to the possibility of Paths 
associated with the same FileSystem having different separators. The 
need to use the separator is significantly reduced in this API related 
to java.io.File (getPath uses varargs for example). If I understand 
correctly then you are actually converting a Path associated with one 
provider to a different Path, in which case an alternative is to 
iterator over the name elements and use resolve to combine.

> 2.       Files.createFile(Path, FileAttribute<?>...) uses 
> FileSystemProvider.newByteChannel() rather than 
> FileSystemProvider.newOutputStream(). If Path is a TPath and the 
> addressed entry is an archive entry, this would require the TrueZIP 
> Kernel to create a cache entry for the archive entry so that you could 
> do random I/O. However, you're just closing the channel immediately. 
> This results in the most expensive way to create a new archive entry 
> in TrueZIP.
>
This is just an implementation choice as newOutputStream(CREATE) and 
newByteChannel(CREATE,WRITE) both create the file. However, point taken, 
if a provider implementation treats these differently and has to prepare 
for random access if newByteChannel is called then it may be less 
efficient. This is easily changed (but not for 7 as it is way too late 
to change anything).


> 3.       Path.getFileName() returns a Path rather than just a String. 
> Well, that's strange -- just consider Path a = Paths.get("foo", 
> "bar"); Path b = a.getFileName(); Now what is supposed to happen if I 
> provide these two objects to the Files class. Will I get the same 
> results? I could argue that this would be desirable because after all 
> the two objects are addressing the same file system entity, just with 
> different labels. However, the WindowsPath implementation does not act 
> like this. The Javadoc for Path.getFileName() does not specify this. I 
> think this could get avoided if Path.getFileName() were just returning 
> a String.
>
It returns a Path so that it can be used with other APIs and also has a 
useful side effect that it preserves the platform representation. As 
specified, operations on relative paths are resolved against the current 
directory so if you access "bar" then it will access bar in the current 
directory. For this case this would be no different to creating a Path 
with getPath("bar") and uses it to access a file.


> 4.       Why are all these I/O methods like newByteChannel(), 
> newInputStream() and newOutputStream() in FileSystemProvider? I think 
> FileSystem would be a better match.
>
Users of the API are supposed to use the java.nio.file package with the 
I/O operations delegating to the appropriate provider (implementations 
of FileSystemProvider). We deliberately did not delegate to FileSystem 
as that would clutter the API (so yes, a bit more work for provider 
implementation but they are expected to be few).

-Alan.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/nio-discuss/attachments/20110610/3f35ba58/attachment-0001.html 

From christian at schlichtherle.de  Sat Jun 11 09:19:28 2011
From: christian at schlichtherle.de (Christian Schlichtherle)
Date: Sat, 11 Jun 2011 18:19:28 +0200
Subject: AW: AW: TrueZIP 7.2-beta-1 with NIO.2 FileSystemProvider
In-Reply-To: <4DF29160.3090603@oracle.com>
References: <005e01cc271b$6cae6330$460b2990$@de> <4DF2245A.1090107@oracle.com>
	<002301cc27b2$e543a7a0$afcaf6e0$@de> <4DF29160.3090603@oracle.com>
Message-ID: <003d01cc2853$5b18f2a0$114ad7e0$@de>

Hi Alan,

 

well, if you say that  the NIO.2 specification and implementation is done,
then this discussion is quite academic.

 

Otherwise, I'ld appreciate if at least the Javadoc for Path.getFileName()
could get updated according to what you have explained, because the
semantics of what users can do with the returned Path object are currently
undefined.

 

Regards,

Christian

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/nio-discuss/attachments/20110611/6f763550/attachment.html 

From Alan.Bateman at oracle.com  Sat Jun 11 12:11:05 2011
From: Alan.Bateman at oracle.com (Alan Bateman)
Date: Sat, 11 Jun 2011 20:11:05 +0100
Subject: AW: AW: TrueZIP 7.2-beta-1 with NIO.2 FileSystemProvider
In-Reply-To: <003d01cc2853$5b18f2a0$114ad7e0$@de>
References: <005e01cc271b$6cae6330$460b2990$@de> <4DF2245A.1090107@oracle.com>
	<002301cc27b2$e543a7a0$afcaf6e0$@de> <4DF29160.3090603@oracle.com>
	<003d01cc2853$5b18f2a0$114ad7e0$@de>
Message-ID: <4DF3BDC9.1040601@oracle.com>

Christian Schlichtherle wrote:
>
> Hi Alan,
>
>  
>
> well, if you say that  the NIO.2 specification and implementation is 
> done, then this discussion is quite academic.
>
>  
>
> Otherwise, I'ld appreciate if at least the Javadoc for 
> Path.getFileName() could get updated according to what you have 
> explained, because the semantics of what users can do with the 
> returned Path object are currently undefined.
>
>  
>
> Regards,
>
> Christian
>
Yes, jdk7 is done and it's impossible to get changes in now. There's 
nothing magically about the getFileName method. It simply returns a path 
representing the file name and so you get a relative path with a single 
name element. I wouldn't say this is undefined but if we get a chance in 
the future then we can add wording to make this clearer.

-Alan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/nio-discuss/attachments/20110611/5ff2aeaa/attachment.html 

From Thomas.Salter at unisys.com  Tue Jun 21 07:00:36 2011
From: Thomas.Salter at unisys.com (Salter, Thomas A)
Date: Tue, 21 Jun 2011 09:00:36 -0500
Subject: how often must file store space attributes be read?
Message-ID: <63D5DCACD1E9E34C89C8203C64F521C3FE4CAA197F@USEA-EXCH7.na.uis.unisys.com>

FileStore.getUsableSpace says the following (with similar wording in FileStore.getUnallocatedSpace):

The returned number of available bytes is a hint, but not a guarantee, that it is possible to use most or any of these bytes. The number of usable bytes is most likely to be accurate immediately after the space attributes are obtained. It is likely to be made inaccurate by any external I/O operations including those made on the system outside of this Java virtual machine.

The phrase "after the space attributes are obtained" seems leftover from the earlier implementations that contained a readFileStoreSpaceAttributes method.

Is the intent of this phrase that an implementation could choose to read the attributes only when the FileStore object is instantiated?  Both the UnixFileStore and the WIndowsFileStore objects currently reread the attributes on every access to the three space methods.

We're porting to a system we're it's somewhat expensive to get the space attributes and we get all three at once.  We'd prefer to read them only once.   Yet experience has shown that portable applications assume that the behavior on Windows or Linux is the behavior they'll find everywhere, so unless the docs are very clear, we're force to match the Windows and Linux implementations.




Tom Salter  |  Software Engineer  |  Java & Middleware Development

Unisys  |  476 Swedesford Road  |  Malvern, PA  19355   |  610-648-2568 |  N385-2568


[cid:image001.gif at 01CC2FF8.0D319590]


THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.



-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/nio-discuss/attachments/20110621/5f571f74/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.gif
Type: image/gif
Size: 2511 bytes
Desc: image001.gif
Url : http://mail.openjdk.java.net/pipermail/nio-discuss/attachments/20110621/5f571f74/attachment.gif 

From Alan.Bateman at oracle.com  Tue Jun 21 07:16:55 2011
From: Alan.Bateman at oracle.com (Alan Bateman)
Date: Tue, 21 Jun 2011 15:16:55 +0100
Subject: how often must file store space attributes be read?
In-Reply-To: <63D5DCACD1E9E34C89C8203C64F521C3FE4CAA197F@USEA-EXCH7.na.uis.unisys.com>
References: <63D5DCACD1E9E34C89C8203C64F521C3FE4CAA197F@USEA-EXCH7.na.uis.unisys.com>
Message-ID: <4E00A7D7.6090702@oracle.com>

Salter, Thomas A wrote:
>
> FileStore.getUsableSpace says the following (with similar wording in 
> FileStore.getUnallocatedSpace):
>
>  
>
> The returned number of available bytes is a hint, but not a guarantee, 
> that it is possible to use most or any of these bytes. The number of 
> usable bytes is most likely to be accurate immediately after the space 
> attributes are obtained. It is likely to be made inaccurate by any 
> external I/O operations including those made on the system outside of 
> this Java virtual machine.
>
>  
>
> The phrase "after the space attributes are obtained" seems leftover 
> from the earlier implementations that contained a 
> readFileStoreSpaceAttributes method.
>
>  
>
> Is the intent of this phrase that an implementation could choose to 
> read the attributes only when the FileStore object is instantiated?  
> Both the UnixFileStore and the WIndowsFileStore objects currently 
> reread the attributes on every access to the three space methods.
>
>  
>
> We're porting to a system we're it's somewhat expensive to get the 
> space attributes and we get all three at once.  We'd prefer to read 
> them only once.   Yet experience has shown that portable applications 
> assume that the behavior on Windows or Linux is the behavior they'll 
> find everywhere, so unless the docs are very clear, we're force to 
> match the Windows and Linux implementations.
>
>  
>
The intent is that these methods return an up-to-date result so the 
returned values could be different from the values returned if these 
methods were involved immediately after obtaining the reference to the 
FileStore.  Looking at it now, then this could be clearer. I'll create a 
bug to make this clearer but it will have to be jdk8 as it's impossible 
to change anything in jdk7 now.

-Alan.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/nio-discuss/attachments/20110621/d042491c/attachment.html 

From christian at schlichtherle.de  Thu Jun 23 09:34:36 2011
From: christian at schlichtherle.de (Christian Schlichtherle)
Date: Thu, 23 Jun 2011 18:34:36 +0200
Subject: TrueZIP 7.2 Release Candidate 1
Message-ID: <002a01cc31c3$7155e4a0$5401ade0$@de>

http://truezip.schlichtherle.de/2011/06/23/truezip-7-2-rc-1/

 

In contrast to the buggy Beta 3 preview, this release candidate has been
thoroughly tested with the integration tests ported from the TrueZIP File*
module. So please give this release candidate a  try if you have been
annoyed by the bugs in the Beta 3 preview.

 

Best regards,

Christian Schlichtherle

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/nio-discuss/attachments/20110623/efa09121/attachment.html 

From semen.vadishev at gmail.com  Sun Jun 26 12:19:27 2011
From: semen.vadishev at gmail.com (Semen Vadishev)
Date: Sun, 26 Jun 2011 21:19:27 +0200
Subject: java.nio.file API with JDK 1.6 and older
Message-ID: <4E07863F.9090804@tmatesoft.com>

Hello there,

Our team develops Java libraries to work with Subversion and Mercurial 
version control systems. These libraries heavily depend on file system 
access. Currently we have to execute external programs or use JNI/JNA 
bindings to call necessary functions, since corresponding functionality 
is not present in JDK 1.6 and older:

- No symlink processing;
- No support for some file attributes;
- Poor ACL support;
- No bulk fetch of attributes.

As supposed, java.nio.file API solves these problems. Eventually we will 
rewrite certain pieces of code to use this API. On the other hand our 
libraries have to work with older versions of JDK. So, the goal is to 
support two implementations of file system access ? older one 
(java.io.File with exec calls or JNI/JNA bindings) and newer one 
(java.nio.file interface).

Here are two ways how we can achieve that:

I. Use wrapper interface.

We build a wrapper interface which delegates its functionality to 
java.nio.file API, if corresponding classes are available at runtime, 
otherwise it delegates calls to existing code which works fine under JDK 
1.6 and older.

II. Make java.nio.file API work with older JDKs.

Currently we're investigating an opportunity to write a standalone 
library which has java.nio.file API and at the same time it could be run 
under JRE 1.6, 1.5 or even 1.4. Basically, this library should include 
all the classes from java.nio.file package and a 1.5/1.6-compatible 
implementation(s) of file store, file system provider, etc. Introduced 
SPI helps a lot here.

The trick is obvious ? one writes code using java.nio.file no matter for 
which version of JDK, then one puts the described library along with 
created code. At runtime class loader decides which classes to load and 
use ? under JRE 1.7 default implementation is available so JVM uses it, 
under older JREs classes from the library are loaded.

Generally we prefer the second approach though it certainly needs much 
more effort. We also believe that a lot of Java developers will benefit 
from such a library. Hopefully, this community will help us to 
understand if we're on right track.

Currently we have the following ideas and questions:

1. Implementation issues.

I can suggest the following approaches to implement the library:

a) Reuse JDK 1.7 code as much as possible. That means that standalone 
library is just extracted code from JDK 1.7 with certain modifications 
which make it compatible with older JDKs. This code includes 
java.nio.file package and sun.nio.fs sources available in JDK 
repository, plus all the JNI bindings.

I'm no expert, so this suggestion may seem insane for someone. Here are 
some thoughts on that:

+ Reuse of well-tested and established code base.
+ Minimal effort to support such library for third-party company.
+ If I understand things correctly, such code extraction is in course of 
global JDK modularization scheduled for 1.8 release.
- Library has to include the compiled code for a number of supported 
platforms.
? Is that technically possible?

b) Implement from scratch all the native stuff using JNI/JNA to access 
necessary libraries at runtime. That means that library includes 
java.nio.file API which is backed by java.io.File methods and some 
native functions called via JNI/JNA.

+ In case of JNA library will need just a tiny JNI wrapper, so there 
won't be a lot of native code in distribution.
- Much more effort in developing and supporting this code for 
third-party company.

c) Execute necessary programs for functionality missing in JDK 1.6 and 
older.

+ Some developers avoid to call native functions which can crash the 
whole JVM, instead they prefer executing separate processes.
- Performance penalty for executing a bunch of processes.

2. Test coverage.

No matter what implementation we will choose, reuse of existing tests is 
always an advantage. Especially when we're talking about different 
implementations of the same interface.

Another question: may we reuse java.nio.file QA infrastructure to test 
the library on all supported platforms?

3. License issues.

Licensing could be another great problem for us. Generally it would be 
nice to have Apache-like license for the library, but it seems 
impossible (I'm no expert in this area though). A lot depends on what 
implementation will we choose after all.


Thanks a lot in advance.

-- 
Semen Vadishev,
TMate Software,
http://svnkit.com/ - Java [Sub]Versioning Library!
http://hg4j.com/ - Java Mercurial Library!
http://sqljet.com/ - Java SQLite Library!


From Alan.Bateman at oracle.com  Sun Jun 26 13:07:45 2011
From: Alan.Bateman at oracle.com (Alan Bateman)
Date: Sun, 26 Jun 2011 21:07:45 +0100
Subject: java.nio.file API with JDK 1.6 and older
In-Reply-To: <4E07863F.9090804@tmatesoft.com>
References: <4E07863F.9090804@tmatesoft.com>
Message-ID: <4E079191.6060208@oracle.com>

Semen,

There have been a couple of projects started to get this API working 
with jdk6 but I don't know if anyone actually released anything. A 
non-technical snag is that the API would need to be in its own name 
spaces (can't be java.**).  Another thing to keep in mind is that the 
API isn't standalone as it includes changes in other areas 
(java.nio.channels in particular). Otherwise I think the only technical 
issue is that there are a dependencies on some JDK internal classes 
(such as sun.misc.Unsafe) that will require attention.

You asked about tests and I would expect the tests in 
jdk/test/java/nio/file/** will be useful. The test harness we use is 
jtreg [1] so you would need to change them if you wan to use JUnit or 
another test framework.

I can't help on the license questions.

-Alan

[1] http://openjdk.java.net/jtreg/index.html