From Alan.Bateman at Sun.COM Sat Nov 1 03:38:13 2008 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Sat, 01 Nov 2008 10:38:13 +0000 Subject: Java API for SCTP In-Reply-To: <514a207b0810311357u7f81ff9es24b1c842be5025e3@mail.gmail.com> References: <514a207b0810311357u7f81ff9es24b1c842be5025e3@mail.gmail.com> Message-ID: <490C3195.1020909@sun.com> Dennis Hjort wrote: > Hi, > > I am writing to find out the current status on the support for SCTP in > java. When I googled around, I found the posts made by Christopher > Hegarty [1] which is a pretty recent post, and wanted to express my > interest in what I hope is a open support for SCTP, as the project I'm > currently working in today has to rely on the Java SCTP API offered by > Ivan Skytte J?rgensen [2], which is a JNI package which does not have > all functionality that the SCTP socket draft specifies. > > Our project (which is a telecom project) needs the SCTP support to be > able to communicat with the system under test, and a native Java > support would be great. We would like to have non-blocking SCTP > instead of our current implementation which has to support the > blocking SCTP read/write the JNI library implements, which creates the > need for a pretty advanced handling. > > I see that the draft for SCTP currently lies in com.sun.nio.sctp [3], > so it has not reached the general audience yet, and I hope I can show > that there exists interest for it. > > Also, please send contact me if I can help in any way for it to get > into the JDK release 7. > There was a recent vote to sponsor this project [1] and it should be setup soon. I'm sure Chris will send a note to this mailing list when it is created. As regards non-blocking - yes, it is integrated with the multiplexing/Selector API. For now the implementation is making use of an internal API for this. In the medium term we hope to define a provider-specific API to allow other selectable channel types to register with the Selector. -Alan. [1] http://mail.openjdk.java.net/pipermail/discuss/2008-October/001348.html From vinod.puchi at gmail.com Wed Nov 12 13:38:53 2008 From: vinod.puchi at gmail.com (Vinod Singh) Date: Wed, 12 Nov 2008 23:38:53 +0200 Subject: Fwd: JAVA SCTP 0-5.7 is not binding to the correct port In-Reply-To: <27200cf20811121333h39d225a8yc216482fae4a8e68@mail.gmail.com> References: <27200cf20811121333h39d225a8yc216482fae4a8e68@mail.gmail.com> Message-ID: <27200cf20811121338n3d24c730hcf9a7a4e27b03421@mail.gmail.com> It's great if anybody could respond me... ---------- Forwarded message ---------- From: Vinod Singh Date: Nov 12, 2008 11:33 PM Subject: JAVA SCTP 0-5.7 is not binding to the correct port To: isj at image.dk Hi, First of all, thanks a lot for Java support of SCTP. I am using Fedora core linux kernel 2.6.26.6-79 Your library - JAVA SCTP 0-5.7 If I try to run your *SCTPTest.java ,* it is failing with error "*Exception in thread "main" java.net.SocketException: connect() failed , errno =111"* ** I have trid to debug this issue and looks like the server is not binding correctly to the given port. The input port is different than the actual port shows in netstat command. I have to give demo tomorrow.. It will be great help if you provide some insight on this.. Thanks a lot.... Regards, Vinod Kumar Singh -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-discuss/attachments/20081112/eef637eb/attachment.html From Alan.Bateman at Sun.COM Wed Nov 12 14:19:01 2008 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Wed, 12 Nov 2008 22:19:01 +0000 Subject: Fwd: JAVA SCTP 0-5.7 is not binding to the correct port In-Reply-To: <27200cf20811121338n3d24c730hcf9a7a4e27b03421@mail.gmail.com> References: <27200cf20811121333h39d225a8yc216482fae4a8e68@mail.gmail.com> <27200cf20811121338n3d24c730hcf9a7a4e27b03421@mail.gmail.com> Message-ID: <491B5655.5020103@sun.com> Vinod Singh wrote: > It's great if anybody could respond me... Looks like you have the wrong mailing list (the SCTP project on OpenJDK isn't quite setup yet). If it helps, then errno 111 is "Connection refused" on Linux. -Alan. > > > ---------- Forwarded message ---------- > From: *Vinod Singh* > > Date: Nov 12, 2008 11:33 PM > Subject: JAVA SCTP 0-5.7 is not binding to the correct port > To: isj at image.dk > > > Hi, > > First of all, thanks a lot for Java support of SCTP. > > I am using Fedora core linux kernel 2.6.26.6-79 > > Your library - JAVA SCTP 0-5.7 > > If I try to run your /SCTPTest.java ,/ it is failing with error > > "/Exception in thread "main" java.net.SocketException: connect() > failed , errno =111"/ > // > I have trid to debug this issue and looks like the server is not > binding correctly to the given port. The input port is different than > the actual port shows in netstat command. > > I have to give demo tomorrow.. > > It will be great help if you provide some insight on this.. > > Thanks a lot.... > > Regards, > Vinod Kumar Singh From Alan.Bateman at Sun.COM Thu Nov 13 13:33:53 2008 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Thu, 13 Nov 2008 21:33:53 +0000 Subject: Thread pool configuration Message-ID: <491C9D41.7080400@sun.com> A number of people have mailed me in the last few weeks with various questions on how thread pools are used by the asynchronous I/O implementation. I've started a page [1] with a few notes that may be useful. When I get time I will update it with details as to how to interpret thread dumps on each operating system as that is important to diagnose problems (the thread dumps do vary by operating system due to the differences in the I/O facilities used). -Alan. [1] http://openjdk.java.net/projects/nio/resources/AsynchronousIo.html From avinash.lakshman at gmail.com Thu Nov 13 13:38:09 2008 From: avinash.lakshman at gmail.com (Avinash Lakshman) Date: Thu, 13 Nov 2008 13:38:09 -0800 Subject: Thread pool configuration In-Reply-To: <491C9D41.7080400@sun.com> References: <491C9D41.7080400@sun.com> Message-ID: Here at Facebook we are experimenting with new AIO framework by trying to integrate it with the Javaflow continuations. Has anyone thought about this approach? Cheers Avinash On Thu, Nov 13, 2008 at 1:33 PM, Alan Bateman wrote: > > A number of people have mailed me in the last few weeks with various > questions on how thread pools are used by the asynchronous I/O > implementation. I've started a page [1] with a few notes that may be useful. > When I get time I will update it with details as to how to interpret thread > dumps on each operating system as that is important to diagnose problems > (the thread dumps do vary by operating system due to the differences in the > I/O facilities used). > > -Alan. > > [1] http://openjdk.java.net/projects/nio/resources/AsynchronousIo.html > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-discuss/attachments/20081113/f422d41a/attachment.html From Alan.Bateman at Sun.COM Fri Nov 14 11:27:01 2008 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Fri, 14 Nov 2008 19:27:01 +0000 Subject: Thread pool configuration In-Reply-To: References: <491C9D41.7080400@sun.com> Message-ID: <491DD105.8080409@sun.com> Avinash Lakshman wrote: > Here at Facebook we are experimenting with new AIO framework by trying > to integrate it with the Javaflow continuations. Has anyone thought > about this approach? There are a number of people on this mailing working on NIO frameworks or other libraries that use NIO that may want to jump in here. I'm pretty sure a number of them have experimented with continuation support (perhaps on solutions that pre-date the request suspend support in the Servlet spec???). -Alan. From Jeanfrancois.Arcand at Sun.COM Mon Nov 17 13:54:06 2008 From: Jeanfrancois.Arcand at Sun.COM (Jeanfrancois Arcand) Date: Mon, 17 Nov 2008 16:54:06 -0500 Subject: Thread pool configuration In-Reply-To: References: <491C9D41.7080400@sun.com> Message-ID: <4921E7FE.6090505@sun.com> Avinash Lakshman wrote: > Here at Facebook we are experimenting with new AIO framework by trying > to integrate it with the Javaflow continuations. Has anyone thought > about this approach? I'm currently implementing AIO support in Grizzly[1] and one of our module support continuation (not exactly like Javaflow). The term we are using is called Comet/Ajax Push (http) or Continuation (tcp). Mainly, an application can decide to suspend the execution of a request, and resume it from the same thread or from another thread, after X times. We support such mechanism at the tcp/tls transport and also at the http protocol (far most popular). A+ -- Jeanfrancois [1] grizzly.dev.java.net > > Cheers > Avinash > > On Thu, Nov 13, 2008 at 1:33 PM, Alan Bateman > wrote: > > > A number of people have mailed me in the last few weeks with various > questions on how thread pools are used by the asynchronous I/O > implementation. I've started a page [1] with a few notes that may be > useful. When I get time I will update it with details as to how to > interpret thread dumps on each operating system as that is important > to diagnose problems (the thread dumps do vary by operating system > due to the differences in the I/O facilities used). > > -Alan. > > [1] http://openjdk.java.net/projects/nio/resources/AsynchronousIo.html > > From avinash.lakshman at gmail.com Thu Nov 20 21:23:32 2008 From: avinash.lakshman at gmail.com (Avinash Lakshman) Date: Thu, 20 Nov 2008 21:23:32 -0800 Subject: Thread pool configuration In-Reply-To: <491C9D41.7080400@sun.com> References: <491C9D41.7080400@sun.com> Message-ID: Hi All Does anyone any pointers to any literature which talks about what should be the ideal amount of data to read from disk for maximizing I/O throughput? Another naive way of posing this question is what should be the ideal buffer size for reads for maximizing I/O throughput? A On Thu, Nov 13, 2008 at 1:33 PM, Alan Bateman wrote: > > A number of people have mailed me in the last few weeks with various > questions on how thread pools are used by the asynchronous I/O > implementation. I've started a page [1] with a few notes that may be useful. > When I get time I will update it with details as to how to interpret thread > dumps on each operating system as that is important to diagnose problems > (the thread dumps do vary by operating system due to the differences in the > I/O facilities used). > > -Alan. > > [1] http://openjdk.java.net/projects/nio/resources/AsynchronousIo.html > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-discuss/attachments/20081120/49312e77/attachment.html From dave at cheney.net Fri Nov 21 04:10:49 2008 From: dave at cheney.net (Dave Cheney) Date: Fri, 21 Nov 2008 23:10:49 +1100 Subject: Thread pool configuration In-Reply-To: References: <491C9D41.7080400@sun.com> Message-ID: Hi Avinash, Without AIO support from the kernel it'll always be a crapshot. Some rough rules of thumb (pulling them from somewhere other than my thumb) would be no less than a page up to slightly less than your OS's read ahead buffering will grab in a single request. But this is highly dependent on load, the number of readers, the availability of the data in the OS's cache, the IO load on the lun and the number of outstanding IO requests. Again without any benchmarks to back this up I would suggest a small buffer, 1 page, would give the best throughput. That is, least chance of a worker blocking on disk IO. What you trade off with this approach is system time and context switches. You probably start approaching some bottlenecks in your filesystem or OS at the extreme of this approach. You could go for an IO worker pool (as InnoDB does) where worker threads sleep waiting on a fixed number of IO threads that actually service requests. Some process, either manual or auto magic could adjust the size of the IO worker pool and/or the IO buffer size based on the average service times of the lun you are pulling from. Just some ideas. Cheers Dave On 21/11/2008, at 4:23 PM, Avinash Lakshman wrote: > Hi All > > Does anyone any pointers to any literature which talks about what > should be the ideal amount of data to read from disk for maximizing > I/O throughput? Another naive way of posing this question is what > should be the ideal buffer size for reads for maximizing I/O > throughput? > > A From Alan.Bateman at Sun.COM Fri Nov 21 04:45:45 2008 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Fri, 21 Nov 2008 12:45:45 +0000 Subject: Thread pool configuration In-Reply-To: References: <491C9D41.7080400@sun.com> Message-ID: <4926AD79.2070401@sun.com> Avinash Lakshman wrote: > Hi All > > Does anyone any pointers to any literature which talks about what > should be the ideal amount of data to read from disk for maximizing > I/O throughput? Another naive way of posing this question is what > should be the ideal buffer size for reads for maximizing I/O throughput? > > A It might be worth elaborating more on what you are doing. Usually, reading in multiples of the block size is efficient. Alternatively, mapping the file into memory may be a good solution. If you are just transferring the bits to a socket then FileChannel#transferTo should use the most efficient facility available (sendfile or equivalent). In any case, you will likely need to do your own performance tests. -Alan. From avinash.lakshman at gmail.com Fri Nov 21 07:55:17 2008 From: avinash.lakshman at gmail.com (Avinash Lakshman) Date: Fri, 21 Nov 2008 07:55:17 -0800 Subject: Thread pool configuration In-Reply-To: <4926AD79.2070401@sun.com> References: <491C9D41.7080400@sun.com> <4926AD79.2070401@sun.com> Message-ID: Do the FileChannel's transferTo() and transferFrom() methods use the AIO framework or are they synchronous in nature? I remember reading somewhere that the Linux sendfile() implementation relies on O_DIRECT to not disturb the page cache. Any truth to this? Thanks Avinash On Fri, Nov 21, 2008 at 4:45 AM, Alan Bateman wrote: > Avinash Lakshman wrote: > >> Hi All >> >> Does anyone any pointers to any literature which talks about what should >> be the ideal amount of data to read from disk for maximizing I/O throughput? >> Another naive way of posing this question is what should be the ideal buffer >> size for reads for maximizing I/O throughput? >> >> A >> > It might be worth elaborating more on what you are doing. Usually, reading > in multiples of the block size is efficient. Alternatively, mapping the file > into memory may be a good solution. If you are just transferring the bits > to a socket then FileChannel#transferTo should use the most efficient > facility available (sendfile or equivalent). In any case, you will likely > need to do your own performance tests. > > -Alan. > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-discuss/attachments/20081121/01081754/attachment.html From Alan.Bateman at Sun.COM Fri Nov 21 08:37:11 2008 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Fri, 21 Nov 2008 16:37:11 +0000 Subject: Thread pool configuration In-Reply-To: References: <491C9D41.7080400@sun.com> <4926AD79.2070401@sun.com> Message-ID: <4926E3B7.20602@sun.com> Avinash Lakshman wrote: > Do the FileChannel's transferTo() and transferFrom() methods use the > AIO framework or are they synchronous in nature? I remember reading > somewhere that the Linux sendfile() implementation relies on O_DIRECT > to not disturb the page cache. Any truth to this? FileChannel defines synchronous operations. The facility that the transferTo/transferFrom methods use depend on the target (or source) channel. For example, if you transferTo a socket channel then it will use sendfile; if you transferTo another file channel then it uses memory mapped I/O. We don't have FileChannel integrated with AsynchronousSocketChannel yet but this is something I hope to address. Direct I/O bypasses the file system cache so great for databases and other very advanced cases that do their own caching. This is not something that the average application will use (requires careful alignment of buffers and file offsets for example). -Alan. From avinash.lakshman at gmail.com Fri Nov 21 10:27:52 2008 From: avinash.lakshman at gmail.com (Avinash Lakshman) Date: Fri, 21 Nov 2008 10:27:52 -0800 Subject: close() not working on the nio patch for JDK7 Message-ID: close() on RandomAccessFile doesn't seem to work. We see a lot of open file handles we though we are invoking close() on the RandomAccessFile in a finally block. Is this a bug in this patch? Thanks Avinash -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-discuss/attachments/20081121/4857be78/attachment.html From Alan.Bateman at Sun.COM Fri Nov 21 10:38:46 2008 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Fri, 21 Nov 2008 18:38:46 +0000 Subject: close() not working on the nio patch for JDK7 In-Reply-To: References: Message-ID: <49270036.1050001@sun.com> Avinash Lakshman wrote: > close() on RandomAccessFile doesn't seem to work. We see a lot of open > file handles we though we are invoking close() on the RandomAccessFile > in a finally block. Is this a bug in this patch? > > Thanks > Avinash What patch is this? Are you building your own binaries from a clone of the nio/nio mercurial repository? In any case, I'm not aware of any problems with RandomAccessFile#close. If you have a test case that you can share that would be great. -Alan. From avinash.lakshman at gmail.com Fri Nov 21 13:21:23 2008 From: avinash.lakshman at gmail.com (Avinash Lakshman) Date: Fri, 21 Nov 2008 13:21:23 -0800 Subject: close() not working on the nio patch for JDK7 In-Reply-To: <49270036.1050001@sun.com> References: <49270036.1050001@sun.com> Message-ID: Download we have is from here - http://download.java.net/jdk7/jsr203/binaries/. Is there a more recent version? Thanks Avinash On Fri, Nov 21, 2008 at 10:38 AM, Alan Bateman wrote: > Avinash Lakshman wrote: > >> close() on RandomAccessFile doesn't seem to work. We see a lot of open >> file handles we though we are invoking close() on the RandomAccessFile in a >> finally block. Is this a bug in this patch? >> >> Thanks >> Avinash >> > What patch is this? Are you building your own binaries from a clone of the > nio/nio mercurial repository? In any case, I'm not aware of any problems > with RandomAccessFile#close. If you have a test case that you can share that > would be great. > > -Alan. > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-discuss/attachments/20081121/2a189e91/attachment.html From Alan.Bateman at Sun.COM Fri Nov 21 13:54:36 2008 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Fri, 21 Nov 2008 21:54:36 +0000 Subject: close() not working on the nio patch for JDK7 In-Reply-To: References: <49270036.1050001@sun.com> Message-ID: <49272E1C.3070301@sun.com> Avinash Lakshman wrote: > Download we have is from here - > http://download.java.net/jdk7/jsr203/binaries/. Is there a more recent > version? That is the latest (your mail hinted that you were patching something which is why I asked). Are you able to come up with a test case for this? If not, can you describe what you are doing and which operating system. Does this happen with the latest jdk7 snapshot? -Alan From Christopher.Hegarty at Sun.COM Mon Nov 24 10:16:29 2008 From: Christopher.Hegarty at Sun.COM (Christopher Hegarty - Sun Microsystems Ireland) Date: Mon, 24 Nov 2008 18:16:29 +0000 Subject: Java API for SCTP Message-ID: <492AEF7D.6070802@sun.com> Hi All, I know that there have been a few mails on this alias relating to SCTP support in Java. An OpenJDK project has been sponsored and setup to develop an API for SCTP [1] and a corresponding OpenJDK prototype. I would encourage anyone interested to take a look and help with the API design and implementation. Currently we have a draft API out for review, and an SCTP specific alias for discussing this and any other issues. In the near future I hope to host prototype binaries and have the source in a mercurial repository. -Chris [1] http://openjdk.java.net/projects/sctp/ From Alan.Bateman at Sun.COM Wed Nov 26 06:43:35 2008 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Wed, 26 Nov 2008 14:43:35 +0000 Subject: nio2 build 98 Message-ID: <492D6097.9080801@sun.com> This build is mostly bug fixes and small improvements. It is synchronized with jdk7 b40. For those using the asynchronous I/O API then it would be good to grab the latest bits as there are number of fixes in this build (and the previous build) to bugs that arose under load conditions. Thanks to Jean-Francois Arcand for several test cases that helped identify some tricky problems. Also thanks to R?mi Forax for corrections to the DirectoryStreamFilters class. Binaries, javadoc, summary of changes are in the usual place [1] -Alan. [1] http://openjdk.java.net/projects/nio/ From avinash.lakshman at gmail.com Wed Nov 26 07:53:58 2008 From: avinash.lakshman at gmail.com (Avinash Lakshman) Date: Wed, 26 Nov 2008 07:53:58 -0800 Subject: AIO and some advice needed. Message-ID: Hi All I have been evaluating AIO and along the way I learnt that it goes a long to help out the NCQ features that are available in today's SATA drives - information can be found over here * http://www.seagate.com/content/pdf/whitepaper/D2c_tech_paper_intc-stx_sata_ncq.pdf *. But there are some questions that I have and I am not quite sure what approach is needed here. The whole idea behind NCQ is that one submits multiple I/O requests to the I/O subsystem and NCQ does a very good job of reordering the requests to minimize rotational and seek latency. Obviously more requests can be submitted only if the application can keep submitting asynchronously without blocking. Now AIO can be used in two modes: (1) Submit the read/write request and then wait on the returned future pointer - which looks like a synchronous read/write to me. It appears one cannot take advantage of NCQ over here. (2) Submit the read/write request and then handle the result in the completion handler as opposed to blocking on a future pointer. This really takes advantage of the NCQ features since the thread is not blocked and more requests can be submitted to take advantage of the NCQ features. If I were to use the latter how would one handle reads like the ones that occur in our application: (1) seek(position); (2) readUTF() - read a key value. (3) readInt() - size of the data associated with the data. (4) ..... We need to perform little reads before we know what the result is. So far we were doing buffered reads and then working with the data in to memory. To convert this into true AIO w/o blocking the application thread, would involve some convuluted programming. Is this the norm? Is my understanding correct as far as the programming paradigm is concerned? Are there any samples anyone could share with me that helps me understand this better? Thanks Avinash -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-discuss/attachments/20081126/e7425c3f/attachment.html From Jeanfrancois.Arcand at Sun.COM Wed Nov 26 08:52:37 2008 From: Jeanfrancois.Arcand at Sun.COM (Jeanfrancois Arcand) Date: Wed, 26 Nov 2008 11:52:37 -0500 Subject: [fyi] Tricks and Tips with AIO... Message-ID: <492D7ED5.4030606@sun.com> Salut, I'm starting a new series of blogs about my experience with AIO[0][1]. Grizzly [2] now have supports for AIO (official 1.9.0 release will be next week) and it will be possible to build application on top of it. We also have out http module re-implemented using AIO, and so far the results/performance looks pretty good. Any feedback is welcome! -- Jeanfrancois [0]http://weblogs.java.net/blog/jfarcand/archive/2008/10/tricks_and_tips_8.html [1]http://weblogs.java.net/blog/jfarcand/archive/2008/11/tricks_and_tips_10.html [2]http://grizzly.dev.java.net From Alan.Bateman at Sun.COM Wed Nov 26 09:16:29 2008 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Wed, 26 Nov 2008 17:16:29 +0000 Subject: AIO and some advice needed. In-Reply-To: References: Message-ID: <492D846D.50308@sun.com> Avinash Lakshman wrote: > Hi All > > I have been evaluating AIO and along the way I learnt that it goes a > long to help out the NCQ features that are available in today's SATA > drives The main thing is to have drivers that support it. I don't know what operating systems you use but Solaris 10 and Linux have come with AHCI drivers for some time. I think you need may special drivers for older versions of Windows. > > > (1) Submit the read/write request and then wait on the returned future > pointer - which looks like a synchronous read/write to me. It appears > one cannot take advantage of NCQ over here. You could initiate several I/O requests and then iterate over the Futures to wait on all the results. The only complication is that you would need to deal with short reads (meaning the read completed with less bytes that you expected). > (2) Submit the read/write request and then handle the result in the > completion handler as opposed to blocking on a future pointer. This > really takes advantage of the NCQ features since the thread is not > blocked and more requests can be submitted to take advantage of the > NCQ features. > > If I were to use the latter how would one handle reads like the ones > that occur in our application: > > (1) seek(position); > (2) readUTF() - read a key value. > (3) readInt() - size of the data associated with the data. > (4) ..... > > We need to perform little reads before we know what the result is. This may not be a good candidate. Is this a sequential file or do you know the file position of data you will need in the future? (in order words, does the file position for a read depend on the previous read?) > So far we were doing buffered reads and then working with the data in > to memory. To convert this into true AIO w/o blocking the application > thread, would involve some convuluted programming. Is this the norm? > Is my understanding correct as far as the programming paradigm is > concerned? Are there any samples anyone could share with me that helps > me understand this better? > Yes, this is the programming model. You either wait for the I/O to complete or else you are notified via a completion handler. The sample directory doesn't have examples using AsynchronousFileChannel yet (if anyone would like to contribute same that would be great). -Alan. PS: Did you track down that RandomAccessFile#close problem that you think might be a regression in the nio2 builds? From avinash.lakshman at gmail.com Wed Nov 26 11:00:44 2008 From: avinash.lakshman at gmail.com (Avinash Lakshman) Date: Wed, 26 Nov 2008 11:00:44 -0800 Subject: AIO and some advice needed. In-Reply-To: <492D846D.50308@sun.com> References: <492D846D.50308@sun.com> Message-ID: The close() thing may have been a false alarm. But we are keeping a close eye on it. Will keep you posted. As far as your earlier comment: * If I were to use the latter how would one handle reads like the ones that occur in our application: (1) seek(position); (2) readUTF() - read a key value. (3) readInt() - size of the data associated with the data. (4) ..... We need to perform little reads before we know what the result is. This may not be a good candidate. Is this a sequential file or do you know the file position of data you will need in the future? (in order words, does the file position for a read depend on the previous read?)* Yes I know the position. So in order to avoid convoluted programming what we are trying to do is submit the read request and suspend the continuation. When the completion handler is done we release the continuation to continue with where we left off. Does this sound reasonable? It almost as if sounds like unless the entire system is asynchronous there is not much to gain from AIO. Is my conclusion correct? Avinash On Wed, Nov 26, 2008 at 9:16 AM, Alan Bateman wrote: > Avinash Lakshman wrote: > >> Hi All >> >> I have been evaluating AIO and along the way I learnt that it goes a long >> to help out the NCQ features that are available in today's SATA drives >> > The main thing is to have drivers that support it. I don't know what > operating systems you use but Solaris 10 and Linux have come with AHCI > drivers for some time. I think you need may special drivers for older > versions of Windows. > > >> >> (1) Submit the read/write request and then wait on the returned future >> pointer - which looks like a synchronous read/write to me. It appears one >> cannot take advantage of NCQ over here. >> > You could initiate several I/O requests and then iterate over the Futures > to wait on all the results. The only complication is that you would need to > deal with short reads (meaning the read completed with less bytes that you > expected). > > (2) Submit the read/write request and then handle the result in the >> completion handler as opposed to blocking on a future pointer. This really >> takes advantage of the NCQ features since the thread is not blocked and more >> requests can be submitted to take advantage of the NCQ features. >> >> If I were to use the latter how would one handle reads like the ones that >> occur in our application: >> >> (1) seek(position); >> (2) readUTF() - read a key value. >> (3) readInt() - size of the data associated with the data. >> (4) ..... >> >> We need to perform little reads before we know what the result is. >> > This may not be a good candidate. Is this a sequential file or do you know > the file position of data you will need in the future? (in order words, does > the file position for a read depend on the previous read?) > > So far we were doing buffered reads and then working with the data in to >> memory. To convert this into true AIO w/o blocking the application thread, >> would involve some convuluted programming. Is this the norm? Is my >> understanding correct as far as the programming paradigm is concerned? Are >> there any samples anyone could share with me that helps me understand this >> better? >> >> Yes, this is the programming model. You either wait for the I/O to > complete or else you are notified via a completion handler. The sample > directory doesn't have examples using AsynchronousFileChannel yet (if anyone > would like to contribute same that would be great). > > -Alan. > > PS: Did you track down that RandomAccessFile#close problem that you think > might be a regression in the nio2 builds? > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-discuss/attachments/20081126/a106173c/attachment.html From Alan.Bateman at Sun.COM Wed Nov 26 12:30:43 2008 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Wed, 26 Nov 2008 20:30:43 +0000 Subject: AIO and some advice needed. In-Reply-To: References: <492D846D.50308@sun.com> Message-ID: <492DB1F3.9070309@sun.com> Avinash Lakshman wrote: > : > > Yes I know the position. So in order to avoid convoluted programming > what we are trying to do is submit the read request and suspend the > continuation. When the completion handler is done we release the > continuation to continue with where we left off. Does this sound > reasonable? It almost as if sounds like unless the entire system is > asynchronous there is not much to gain from AIO. Is my conclusion correct? In principal this should be fine. Please do feedback your experiences with this approach. Asynchronous I/O does require an event driven application so it can be awkward to shoehorn into existing code. -Alan. From avinash.lakshman at gmail.com Wed Nov 26 23:26:53 2008 From: avinash.lakshman at gmail.com (Avinash Lakshman) Date: Wed, 26 Nov 2008 23:26:53 -0800 Subject: AIO and some advice needed. In-Reply-To: <492DB1F3.9070309@sun.com> References: <492D846D.50308@sun.com> <492DB1F3.9070309@sun.com> Message-ID: iostat on Linux doesn't seem to show me correct statistics when I use AIO. I know I am performing reads correctly but the r/s is always at zero. What am I doing wrong? I run iostat -x 1. Thanks A On Wed, Nov 26, 2008 at 12:30 PM, Alan Bateman wrote: > Avinash Lakshman wrote: > >> : >> >> Yes I know the position. So in order to avoid convoluted programming what >> we are trying to do is submit the read request and suspend the continuation. >> When the completion handler is done we release the continuation to continue >> with where we left off. Does this sound reasonable? It almost as if sounds >> like unless the entire system is asynchronous there is not much to gain from >> AIO. Is my conclusion correct? >> > In principal this should be fine. Please do feedback your experiences with > this approach. Asynchronous I/O does require an event driven application so > it can be awkward to shoehorn into existing code. > > -Alan. > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-discuss/attachments/20081126/ff05cae8/attachment.html From Alan.Bateman at Sun.COM Thu Nov 27 02:32:00 2008 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Thu, 27 Nov 2008 10:32:00 +0000 Subject: AIO and some advice needed. In-Reply-To: References: <492D846D.50308@sun.com> <492DB1F3.9070309@sun.com> Message-ID: <492E7720.6040001@sun.com> Avinash Lakshman wrote: > iostat on Linux doesn't seem to show me correct statistics when I use > AIO. I know I am performing reads correctly but the r/s is always at > zero. What am I doing wrong? I run iostat -x 1. Hard to say. Are you saying that iostat doesn't work or just that the read stats don't change? If the latter then maybe the file is already cached. I see in proc(5) that you can drop the cache after sync'ing (I've never used it but maybe useful for this test). -Alan. From avinash.lakshman at gmail.com Thu Nov 27 07:45:08 2008 From: avinash.lakshman at gmail.com (Avinash Lakshman) Date: Thu, 27 Nov 2008 07:45:08 -0800 Subject: AIO and some advice needed. In-Reply-To: <492E7720.6040001@sun.com> References: <492D846D.50308@sun.com> <492DB1F3.9070309@sun.com> <492E7720.6040001@sun.com> Message-ID: Here is the test I have been trying to run in order to understand why AIO is claimed to be better. I have a file about 44 GB in size on disk. My Linux machine has 8 cores and 16 GB RAM. I first ran my program where I seek to random offsets within the file and perform a synchronous buffered read of 64K chunks. I collected some statistics from iostat which is shown below: avg-cpu: %user %nice %sys %iowait %idle 0.13 0.00 0.38 12.37 87.12 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util sda 0.00 9.18 0.00 6.12 0.00 122.45 0.00 61.22 20.00 0.00 0.17 0.17 0.10 sdb 11.22 0.00 110.20 0.00 12824.49 0.00 6412.24 0.00 116.37 1.02 9.27 9.18 101.12 sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 avg-cpu: %user %nice %sys %iowait %idle 0.00 0.00 0.00 12.50 87.50 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util sda 0.00 14.14 0.00 10.10 0.00 193.94 0.00 96.97 19.20 0.00 0.00 0.00 0.00 sdb 0.00 1.01 102.02 2.02 11563.64 24.24 5781.82 12.12 111.38 1.01 9.78 9.61 100.00 sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 avg-cpu: %user %nice %sys %iowait %idle 0.00 0.00 0.25 12.41 87.34 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util sda 0.00 9.09 0.00 5.05 0.00 113.13 0.00 56.57 22.40 0.01 1.40 0.80 0.40 sdb 26.26 0.00 109.09 0.00 12759.60 0.00 6379.80 0.00 116.96 1.03 9.41 9.25 100.91 sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 avg-cpu: %user %nice %sys %iowait %idle 0.13 0.00 0.25 12.47 87.15 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util sda 0.00 8.08 0.00 3.03 0.00 88.89 0.00 44.44 29.33 0.00 0.33 0.33 0.10 sdb 0.00 0.00 100.00 0.00 12387.88 0.00 6193.94 0.00 123.88 1.01 10.04 10.05 100.51 sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 avg-cpu: %user %nice %sys %iowait %idle 0.00 0.00 0.25 12.39 87.36 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util sda 0.00 9.18 0.00 6.12 0.00 122.45 0.00 61.22 20.00 0.00 0.17 0.17 0.10 sdb 11.22 0.00 103.06 0.00 12383.67 0.00 6191.84 0.00 120.16 1.02 9.83 9.79 100.92 sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Then I do the same thing using AIO where I just keep submitting requests and then monitor the activity using iostat and the numbers are as follows: avg-cpu: %user %nice %sys %iowait %idle 7.21 0.00 0.76 39.06 52.97 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util sda 0.00 2.04 0.00 6.12 0.00 65.31 0.00 32.65 10.67 0.00 0.17 0.17 0.10 sdb 32.65 0.00 125.51 0.00 14955.10 0.00 7477.55 0.00 119.15 6.68 53.69 7.99 100.31 sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 avg-cpu: %user %nice %sys %iowait %idle 7.20 0.00 0.51 39.90 52.40 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util sda 0.00 14.14 0.00 10.10 0.00 193.94 0.00 96.97 19.20 0.00 0.20 0.20 0.20 sdb 41.41 0.00 140.40 0.00 16339.39 0.00 8169.70 0.00 116.37 6.64 46.76 6.96 97.68 sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 avg-cpu: %user %nice %sys %iowait %idle 7.22 0.00 0.51 38.35 53.92 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util sda 0.00 7.07 0.00 5.05 0.00 96.97 0.00 48.48 19.20 0.00 0.80 0.80 0.40 sdb 19.19 0.00 140.40 0.00 16161.62 0.00 8080.81 0.00 115.11 6.58 46.24 7.01 98.48 sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 avg-cpu: %user %nice %sys %iowait %idle 7.19 0.00 0.50 36.82 55.49 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util sda 0.00 10.20 0.00 3.06 0.00 106.12 0.00 53.06 34.67 0.00 0.00 0.00 0.00 sdb 0.00 0.00 128.57 0.00 15518.37 0.00 7759.18 0.00 120.70 6.68 52.11 7.87 101.22 sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 avg-cpu: %user %nice %sys %iowait %idle 7.09 0.00 0.51 35.06 57.34 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util sda 0.00 9.18 0.00 6.12 0.00 122.45 0.00 61.22 20.00 0.00 0.33 0.33 0.20 sdb 30.61 0.00 143.88 0.00 17795.92 0.00 8897.96 0.00 123.69 6.44 44.59 6.63 95.41 sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 avg-cpu: %user %nice %sys %iowait %idle 7.20 0.00 0.63 41.16 51.01 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdb 24.24 1.01 126.26 2.02 15418.18 24.24 7709.09 12.12 120.38 6.63 51.24 7.69 98.59 sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Now is the million dollar question - which is better and why? Could someone please help me understand these numbers? I am willing to get more information out if need be. Thanks Avinash On Thu, Nov 27, 2008 at 2:32 AM, Alan Bateman wrote: > Avinash Lakshman wrote: > >> iostat on Linux doesn't seem to show me correct statistics when I use AIO. >> I know I am performing reads correctly but the r/s is always at zero. What >> am I doing wrong? I run iostat -x 1. >> > Hard to say. Are you saying that iostat doesn't work or just that the read > stats don't change? If the latter then maybe the file is already cached. I > see in proc(5) that you can drop the cache after sync'ing (I've never used > it but maybe useful for this test). > > -Alan. > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-discuss/attachments/20081127/57197267/attachment.html From Alan.Bateman at Sun.COM Thu Nov 27 10:32:50 2008 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Thu, 27 Nov 2008 18:32:50 +0000 Subject: AIO and some advice needed. In-Reply-To: References: <492D846D.50308@sun.com> <492DB1F3.9070309@sun.com> <492E7720.6040001@sun.com> Message-ID: <492EE7D2.4070508@sun.com> Avinash Lakshman wrote: > Here is the test I have been trying to run in order to understand why > AIO is claimed to be better. > > I have a file about 44 GB in size on disk. My Linux machine has 8 > cores and 16 GB RAM. I first ran my program where I seek to random > offsets within the file and perform a synchronous buffered read of 64K > chunks. I collected some statistics from iostat which is shown below: > > avg-cpu: %user %nice %sys %iowait %idle > 0.13 0.00 0.38 12.37 87.12 > > Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s > avgrq-sz avgqu-sz await svctm %util > sda 0.00 9.18 0.00 6.12 0.00 122.45 0.00 > 61.22 20.00 0.00 0.17 0.17 0.10 > sdb 11.22 0.00 110.20 0.00 12824.49 0.00 6412.24 > 0.00 116.37 1.02 9.27 9.18 101.12 > sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 0.00 0.00 0.00 > sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 0.00 0.00 0.00 > > avg-cpu: %user %nice %sys %iowait %idle > 0.00 0.00 0.00 12.50 87.50 > > Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s > avgrq-sz avgqu-sz await svctm %util > sda 0.00 14.14 0.00 10.10 0.00 193.94 0.00 > 96.97 19.20 0.00 0.00 0.00 0.00 > sdb 0.00 1.01 102.02 2.02 11563.64 24.24 5781.82 > 12.12 111.38 1.01 9.78 9.61 100.00 > sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 0.00 0.00 0.00 > sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 0.00 0.00 0.00 > > avg-cpu: %user %nice %sys %iowait %idle > 0.00 0.00 0.25 12.41 87.34 > > Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s > avgrq-sz avgqu-sz await svctm %util > sda 0.00 9.09 0.00 5.05 0.00 113.13 0.00 > 56.57 22.40 0.01 1.40 0.80 0.40 > sdb 26.26 0.00 109.09 0.00 12759.60 0.00 6379.80 > 0.00 116.96 1.03 9.41 9.25 100.91 > sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 0.00 0.00 0.00 > sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 0.00 0.00 0.00 > > avg-cpu: %user %nice %sys %iowait %idle > 0.13 0.00 0.25 12.47 87.15 > > Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s > avgrq-sz avgqu-sz await svctm %util > sda 0.00 8.08 0.00 3.03 0.00 88.89 0.00 > 44.44 29.33 0.00 0.33 0.33 0.10 > sdb 0.00 0.00 100.00 0.00 12387.88 0.00 6193.94 > 0.00 123.88 1.01 10.04 10.05 100.51 > sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 0.00 0.00 0.00 > sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 0.00 0.00 0.00 > > avg-cpu: %user %nice %sys %iowait %idle > 0.00 0.00 0.25 12.39 87.36 > > Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s > avgrq-sz avgqu-sz await svctm %util > sda 0.00 9.18 0.00 6.12 0.00 122.45 0.00 > 61.22 20.00 0.00 0.17 0.17 0.10 > sdb 11.22 0.00 103.06 0.00 12383.67 0.00 6191.84 > 0.00 120.16 1.02 9.83 9.79 100.92 > sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 0.00 0.00 0.00 > sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 0.00 0.00 0.00 > > Then I do the same thing using AIO where I just keep submitting > requests and then monitor the activity using iostat and the numbers > are as follows: > > avg-cpu: %user %nice %sys %iowait %idle > 7.21 0.00 0.76 39.06 52.97 > > Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s > avgrq-sz avgqu-sz await svctm %util > sda 0.00 2.04 0.00 6.12 0.00 65.31 0.00 > 32.65 10.67 0.00 0.17 0.17 0.10 > sdb 32.65 0.00 125.51 0.00 14955.10 0.00 7477.55 > 0.00 119.15 6.68 53.69 7.99 100.31 > sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 0.00 0.00 0.00 > sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 0.00 0.00 0.00 > > avg-cpu: %user %nice %sys %iowait %idle > 7.20 0.00 0.51 39.90 52.40 > > Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s > avgrq-sz avgqu-sz await svctm %util > sda 0.00 14.14 0.00 10.10 0.00 193.94 0.00 > 96.97 19.20 0.00 0.20 0.20 0.20 > sdb 41.41 0.00 140.40 0.00 16339.39 0.00 8169.70 > 0.00 116.37 6.64 46.76 6.96 97.68 > sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 0.00 0.00 0.00 > sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 0.00 0.00 0.00 > > avg-cpu: %user %nice %sys %iowait %idle > 7.22 0.00 0.51 38.35 53.92 > > Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s > avgrq-sz avgqu-sz await svctm %util > sda 0.00 7.07 0.00 5.05 0.00 96.97 0.00 > 48.48 19.20 0.00 0.80 0.80 0.40 > sdb 19.19 0.00 140.40 0.00 16161.62 0.00 8080.81 > 0.00 115.11 6.58 46.24 7.01 98.48 > sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 0.00 0.00 0.00 > sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 0.00 0.00 0.00 > > avg-cpu: %user %nice %sys %iowait %idle > 7.19 0.00 0.50 36.82 55.49 > > Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s > avgrq-sz avgqu-sz await svctm %util > sda 0.00 10.20 0.00 3.06 0.00 106.12 0.00 > 53.06 34.67 0.00 0.00 0.00 0.00 > sdb 0.00 0.00 128.57 0.00 15518.37 0.00 7759.18 > 0.00 120.70 6.68 52.11 7.87 101.22 > sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 0.00 0.00 0.00 > sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 0.00 0.00 0.00 > > avg-cpu: %user %nice %sys %iowait %idle > 7.09 0.00 0.51 35.06 57.34 > > Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s > avgrq-sz avgqu-sz await svctm %util > sda 0.00 9.18 0.00 6.12 0.00 122.45 0.00 > 61.22 20.00 0.00 0.33 0.33 0.20 > sdb 30.61 0.00 143.88 0.00 17795.92 0.00 8897.96 > 0.00 123.69 6.44 44.59 6.63 95.41 > sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 0.00 0.00 0.00 > sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 0.00 0.00 0.00 > > avg-cpu: %user %nice %sys %iowait %idle > 7.20 0.00 0.63 41.16 51.01 > > Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s > avgrq-sz avgqu-sz await svctm %util > sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 0.00 0.00 0.00 > sdb 24.24 1.01 126.26 2.02 15418.18 24.24 7709.09 > 12.12 120.38 6.63 51.24 7.69 98.59 > sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 0.00 0.00 0.00 > sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 0.00 0.00 0.00 > > Now is the million dollar question - which is better and why? Could > someone please help me understand these numbers? I am willing to get > more information out if need be. The iostat(1) page has descriptions for each of the columns. From a brief scan it looks like the synchronous case is doing 100-110 read requests/sec and reading 5.7-6.4mB/sec (do I have this right?). In the asynchronous case it looks like you are sending 125-140 read requests/sec and reading 7.4-8.8mB/sec. CPU usage can be important and the second test is clearly putting more load on the system. -Alan. From avinash.lakshman at gmail.com Thu Nov 27 20:15:34 2008 From: avinash.lakshman at gmail.com (Avinash Lakshman) Date: Thu, 27 Nov 2008 20:15:34 -0800 Subject: AIO and some advice needed. In-Reply-To: <492EE7D2.4070508@sun.com> References: <492D846D.50308@sun.com> <492DB1F3.9070309@sun.com> <492E7720.6040001@sun.com> <492EE7D2.4070508@sun.com> Message-ID: Thanks Alan. Do you have any performance benchmark numbers that you can share comparing the performance of synchronous and asynchronous approaches on Linux? Thanks Avinash On Thu, Nov 27, 2008 at 10:32 AM, Alan Bateman wrote: > Avinash Lakshman wrote: > >> Here is the test I have been trying to run in order to understand why AIO >> is claimed to be better. >> >> I have a file about 44 GB in size on disk. My Linux machine has 8 cores >> and 16 GB RAM. I first ran my program where I seek to random offsets within >> the file and perform a synchronous buffered read of 64K chunks. I collected >> some statistics from iostat which is shown below: >> >> avg-cpu: %user %nice %sys %iowait %idle >> 0.13 0.00 0.38 12.37 87.12 >> >> Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s >> avgrq-sz avgqu-sz await svctm %util >> sda 0.00 9.18 0.00 6.12 0.00 122.45 0.00 61.22 >> 20.00 0.00 0.17 0.17 0.10 >> sdb 11.22 0.00 110.20 0.00 12824.49 0.00 6412.24 0.00 >> 116.37 1.02 9.27 9.18 101.12 >> sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 >> 0.00 0.00 0.00 0.00 0.00 >> sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 >> 0.00 0.00 0.00 0.00 0.00 >> >> avg-cpu: %user %nice %sys %iowait %idle >> 0.00 0.00 0.00 12.50 87.50 >> >> Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s >> avgrq-sz avgqu-sz await svctm %util >> sda 0.00 14.14 0.00 10.10 0.00 193.94 0.00 96.97 >> 19.20 0.00 0.00 0.00 0.00 >> sdb 0.00 1.01 102.02 2.02 11563.64 24.24 5781.82 12.12 >> 111.38 1.01 9.78 9.61 100.00 >> sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 >> 0.00 0.00 0.00 0.00 0.00 >> sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 >> 0.00 0.00 0.00 0.00 0.00 >> >> avg-cpu: %user %nice %sys %iowait %idle >> 0.00 0.00 0.25 12.41 87.34 >> >> Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s >> avgrq-sz avgqu-sz await svctm %util >> sda 0.00 9.09 0.00 5.05 0.00 113.13 0.00 56.57 >> 22.40 0.01 1.40 0.80 0.40 >> sdb 26.26 0.00 109.09 0.00 12759.60 0.00 6379.80 0.00 >> 116.96 1.03 9.41 9.25 100.91 >> sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 >> 0.00 0.00 0.00 0.00 0.00 >> sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 >> 0.00 0.00 0.00 0.00 0.00 >> >> avg-cpu: %user %nice %sys %iowait %idle >> 0.13 0.00 0.25 12.47 87.15 >> >> Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s >> avgrq-sz avgqu-sz await svctm %util >> sda 0.00 8.08 0.00 3.03 0.00 88.89 0.00 44.44 >> 29.33 0.00 0.33 0.33 0.10 >> sdb 0.00 0.00 100.00 0.00 12387.88 0.00 6193.94 0.00 >> 123.88 1.01 10.04 10.05 100.51 >> sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 >> 0.00 0.00 0.00 0.00 0.00 >> sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 >> 0.00 0.00 0.00 0.00 0.00 >> >> avg-cpu: %user %nice %sys %iowait %idle >> 0.00 0.00 0.25 12.39 87.36 >> >> Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s >> avgrq-sz avgqu-sz await svctm %util >> sda 0.00 9.18 0.00 6.12 0.00 122.45 0.00 61.22 >> 20.00 0.00 0.17 0.17 0.10 >> sdb 11.22 0.00 103.06 0.00 12383.67 0.00 6191.84 0.00 >> 120.16 1.02 9.83 9.79 100.92 >> sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 >> 0.00 0.00 0.00 0.00 0.00 >> sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 >> 0.00 0.00 0.00 0.00 0.00 >> >> Then I do the same thing using AIO where I just keep submitting requests >> and then monitor the activity using iostat and the numbers are as follows: >> >> avg-cpu: %user %nice %sys %iowait %idle >> 7.21 0.00 0.76 39.06 52.97 >> >> Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s >> avgrq-sz avgqu-sz await svctm %util >> sda 0.00 2.04 0.00 6.12 0.00 65.31 0.00 32.65 >> 10.67 0.00 0.17 0.17 0.10 >> sdb 32.65 0.00 125.51 0.00 14955.10 0.00 7477.55 0.00 >> 119.15 6.68 53.69 7.99 100.31 >> sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 >> 0.00 0.00 0.00 0.00 0.00 >> sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 >> 0.00 0.00 0.00 0.00 0.00 >> >> avg-cpu: %user %nice %sys %iowait %idle >> 7.20 0.00 0.51 39.90 52.40 >> >> Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s >> avgrq-sz avgqu-sz await svctm %util >> sda 0.00 14.14 0.00 10.10 0.00 193.94 0.00 96.97 >> 19.20 0.00 0.20 0.20 0.20 >> sdb 41.41 0.00 140.40 0.00 16339.39 0.00 8169.70 0.00 >> 116.37 6.64 46.76 6.96 97.68 >> sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 >> 0.00 0.00 0.00 0.00 0.00 >> sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 >> 0.00 0.00 0.00 0.00 0.00 >> >> avg-cpu: %user %nice %sys %iowait %idle >> 7.22 0.00 0.51 38.35 53.92 >> >> Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s >> avgrq-sz avgqu-sz await svctm %util >> sda 0.00 7.07 0.00 5.05 0.00 96.97 0.00 48.48 >> 19.20 0.00 0.80 0.80 0.40 >> sdb 19.19 0.00 140.40 0.00 16161.62 0.00 8080.81 0.00 >> 115.11 6.58 46.24 7.01 98.48 >> sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 >> 0.00 0.00 0.00 0.00 0.00 >> sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 >> 0.00 0.00 0.00 0.00 0.00 >> >> avg-cpu: %user %nice %sys %iowait %idle >> 7.19 0.00 0.50 36.82 55.49 >> >> Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s >> avgrq-sz avgqu-sz await svctm %util >> sda 0.00 10.20 0.00 3.06 0.00 106.12 0.00 53.06 >> 34.67 0.00 0.00 0.00 0.00 >> sdb 0.00 0.00 128.57 0.00 15518.37 0.00 7759.18 0.00 >> 120.70 6.68 52.11 7.87 101.22 >> sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 >> 0.00 0.00 0.00 0.00 0.00 >> sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 >> 0.00 0.00 0.00 0.00 0.00 >> >> avg-cpu: %user %nice %sys %iowait %idle >> 7.09 0.00 0.51 35.06 57.34 >> >> Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s >> avgrq-sz avgqu-sz await svctm %util >> sda 0.00 9.18 0.00 6.12 0.00 122.45 0.00 61.22 >> 20.00 0.00 0.33 0.33 0.20 >> sdb 30.61 0.00 143.88 0.00 17795.92 0.00 8897.96 0.00 >> 123.69 6.44 44.59 6.63 95.41 >> sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 >> 0.00 0.00 0.00 0.00 0.00 >> sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 >> 0.00 0.00 0.00 0.00 0.00 >> >> avg-cpu: %user %nice %sys %iowait %idle >> 7.20 0.00 0.63 41.16 51.01 >> >> Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s >> avgrq-sz avgqu-sz await svctm %util >> sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 >> 0.00 0.00 0.00 0.00 0.00 >> sdb 24.24 1.01 126.26 2.02 15418.18 24.24 7709.09 12.12 >> 120.38 6.63 51.24 7.69 98.59 >> sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 >> 0.00 0.00 0.00 0.00 0.00 >> sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 >> 0.00 0.00 0.00 0.00 0.00 >> >> Now is the million dollar question - which is better and why? Could >> someone please help me understand these numbers? I am willing to get more >> information out if need be. >> > The iostat(1) page has descriptions for each of the columns. From a brief > scan it looks like the synchronous case is doing 100-110 read requests/sec > and reading 5.7-6.4mB/sec (do I have this right?). In the asynchronous case > it looks like you are sending 125-140 read requests/sec and reading > 7.4-8.8mB/sec. CPU usage can be important and the second test is clearly > putting more load on the system. > > -Alan. > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-discuss/attachments/20081127/8709f500/attachment.html From Alan.Bateman at Sun.COM Fri Nov 28 11:01:11 2008 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Fri, 28 Nov 2008 19:01:11 +0000 Subject: AIO and some advice needed. In-Reply-To: References: <492D846D.50308@sun.com> <492DB1F3.9070309@sun.com> <492E7720.6040001@sun.com> <492EE7D2.4070508@sun.com> Message-ID: <49303FF7.5000304@sun.com> Avinash Lakshman wrote: > Thanks Alan. Do you have any performance benchmark numbers that you > can share comparing the performance of synchronous and asynchronous > approaches on Linux? I don't have numbers to share but one thing to say is that kernel AIO on Linux isn't supported (to my knowledge) for files that aren't opened for direct I/O. Direct I/O is very much for databases and a few cases that do their own caching. This means that when you are using an asynchronous file channel it is actually doing synchronous I/O with a thread pool. This works surprisingly well but you need to play around with sizing of the thread pool to get the right concurrency. In the future we can make use of the kernel support if it changes. I don't think I understand your environment sufficiently to make any reasonable recommendations. I understand there is a big 45GB file but it's not clear to me how it is used. If indeed you can read and process areas of the file in parallel then try AsynchronousFileChannel and send feedback. FileChannel also defines a positional read method and so can support concurrent access without concern for the global file position. -Alan. From avinash.lakshman at gmail.com Fri Nov 28 13:39:01 2008 From: avinash.lakshman at gmail.com (Avinash Lakshman) Date: Fri, 28 Nov 2008 13:39:01 -0800 Subject: AIO and some advice needed. In-Reply-To: <49303FF7.5000304@sun.com> References: <492D846D.50308@sun.com> <492DB1F3.9070309@sun.com> <492E7720.6040001@sun.com> <492EE7D2.4070508@sun.com> <49303FF7.5000304@sun.com> Message-ID: Thanks for the reponse inspite of it being a holiday. I appreciate it. What I am doing is on the main thread submitting read requests to the AsynchronousFileChannel and having it read from positions that are randomly generated. The FileChannel is equipped with a thread pool which has 8 threads. If it is doing synchronous reads with a thread pool then the performance is understandable I guess. Thanks Avinash On Fri, Nov 28, 2008 at 11:01 AM, Alan Bateman wrote: > Avinash Lakshman wrote: > >> Thanks Alan. Do you have any performance benchmark numbers that you can >> share comparing the performance of synchronous and asynchronous approaches >> on Linux? >> > I don't have numbers to share but one thing to say is that kernel AIO on > Linux isn't supported (to my knowledge) for files that aren't opened for > direct I/O. Direct I/O is very much for databases and a few cases that do > their own caching. This means that when you are using an asynchronous file > channel it is actually doing synchronous I/O with a thread pool. This works > surprisingly well but you need to play around with sizing of the thread pool > to get the right concurrency. In the future we can make use of the kernel > support if it changes. I don't think I understand your environment > sufficiently to make any reasonable recommendations. I understand there is a > big 45GB file but it's not clear to me how it is used. If indeed you can > read and process areas of the file in parallel then try > AsynchronousFileChannel and send feedback. FileChannel also defines a > positional read method and so can support concurrent access without concern > for the global file position. > > -Alan. > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-discuss/attachments/20081128/d95f459d/attachment.html From avinash.lakshman at gmail.com Fri Nov 28 18:25:28 2008 From: avinash.lakshman at gmail.com (Avinash Lakshman) Date: Fri, 28 Nov 2008 18:25:28 -0800 Subject: AIO and some advice needed. In-Reply-To: <49303FF7.5000304@sun.com> References: <492D846D.50308@sun.com> <492DB1F3.9070309@sun.com> <492E7720.6040001@sun.com> <492EE7D2.4070508@sun.com> <49303FF7.5000304@sun.com> Message-ID: Looks like Linux has AIO support only if the file has been open with O_DIRECT flag. I guess AIO is moot at this point. However what does it take for us to request O_DIRECT support in this NIO release? We are building a large scale storage system called Cassandra, here at Facebook, details of which can be found here (http://code.google.com/p/the-cassandra-project/) and would really love to have O_DIRECT support via the JDK for certain class of operations. Thanks Avinash On Fri, Nov 28, 2008 at 11:01 AM, Alan Bateman wrote: > Avinash Lakshman wrote: > >> Thanks Alan. Do you have any performance benchmark numbers that you can >> share comparing the performance of synchronous and asynchronous approaches >> on Linux? >> > I don't have numbers to share but one thing to say is that kernel AIO on > Linux isn't supported (to my knowledge) for files that aren't opened for > direct I/O. Direct I/O is very much for databases and a few cases that do > their own caching. This means that when you are using an asynchronous file > channel it is actually doing synchronous I/O with a thread pool. This works > surprisingly well but you need to play around with sizing of the thread pool > to get the right concurrency. In the future we can make use of the kernel > support if it changes. I don't think I understand your environment > sufficiently to make any reasonable recommendations. I understand there is a > big 45GB file but it's not clear to me how it is used. If indeed you can > read and process areas of the file in parallel then try > AsynchronousFileChannel and send feedback. FileChannel also defines a > positional read method and so can support concurrent access without concern > for the global file position. > > -Alan. > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-discuss/attachments/20081128/f03c98c3/attachment.html From david.lloyd at redhat.com Fri Nov 28 19:28:40 2008 From: david.lloyd at redhat.com (David M. Lloyd) Date: Fri, 28 Nov 2008 21:28:40 -0600 Subject: AIO and some advice needed. In-Reply-To: References: <492D846D.50308@sun.com> <492DB1F3.9070309@sun.com> <492E7720.6040001@sun.com> <492EE7D2.4070508@sun.com> <49303FF7.5000304@sun.com> Message-ID: <4930B6E8.1020203@redhat.com> On 11/28/2008 08:25 PM, Avinash Lakshman wrote: > Looks like Linux has AIO support only if the file has been open with > O_DIRECT flag. I guess AIO is moot at this point. However what does it > take for us to request O_DIRECT support in this NIO release? We are > building a large scale storage system called Cassandra, here at > Facebook, details of which can be found here Here's some comments by Linus about O_DIRECT as it pertains to regular files: "There really is no valid reason for EVER using O_DIRECT. You need a buffer whatever IO you do, and it might as well be the page cache. There are better ways to control the page cache than play games and think that a page cache isn't necessary." http://lkml.org/lkml/2007/1/10/233 Read the whole thread - the following comments, and the questions that lead up to it - it's full of interesting tidbits, like this one about the realities of what it means to bypass OS caching: http://lkml.org/lkml/2007/1/11/129 O_DIRECT only makes sense on raw devices: http://lkml.org/lkml/2007/1/11/157 The main problem with O_DIRECT is that userspace pages cannot be "stolen" to act as buffer pages for the filesystem write, thus the write() has to block until the data can be written out all the way: http://lkml.org/lkml/2007/1/11/157 That said, later on in the thread there are some good counterpoints as well. One thing to consider is instead using FileChannel.map() and FileChannel.force() or MappedByteBuffer.force() instead, possibly in combination with a thread pool... this should have similar characteristics to O_DIRECT and has the futher advantage of being available since 1.4 as well. - DML From david.lloyd at redhat.com Sun Nov 30 10:22:51 2008 From: david.lloyd at redhat.com (David M. Lloyd) Date: Sun, 30 Nov 2008 12:22:51 -0600 Subject: Scatter/Gather support on DatagramChannel Message-ID: <4932D9FB.1020808@redhat.com> DatagramChannel has scatter/gather support for connected sockets but not unconnected sockets. How about adding a couple methods: SocketAddress receive(ByteBuffer[] dsts) throws IOException; SocketAddress receive(ByteBuffer[] dsts, int offset, int length) throws IOException; long send(ByteBuffer[] src, SocketAddress target) throws IOException; long send(ByteBuffer[] src, int offset, int length, SocketAddress target) throws IOException; I for one use scatter/gather frequently, and would really welcome such methods on DatagramChannel. Thanks, - DML From Alan.Bateman at Sun.COM Sun Nov 30 12:02:05 2008 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Sun, 30 Nov 2008 20:02:05 +0000 Subject: Scatter/Gather support on DatagramChannel In-Reply-To: <4932D9FB.1020808@redhat.com> References: <4932D9FB.1020808@redhat.com> Message-ID: <4932F13D.7050007@sun.com> David M. Lloyd wrote: > DatagramChannel has scatter/gather support for connected sockets but > not unconnected sockets. > > How about adding a couple methods: > > SocketAddress receive(ByteBuffer[] dsts) throws IOException; > > SocketAddress receive(ByteBuffer[] dsts, int offset, int length) > throws IOException; > > long send(ByteBuffer[] src, SocketAddress target) throws IOException; > > long send(ByteBuffer[] src, int offset, int length, SocketAddress > target) throws IOException; > > I for one use scatter/gather frequently, and would really welcome such > methods on DatagramChannel. > > Thanks, > - DML 6696470 is tracking this but it has been low priority (and is a bit more work that might be apparent because it requires replacing part of the existing implementation, Windows mostly, but also work on other platforms). -Alan.