TCP Sockets - Limiting one write at a time is harming performance (at least on windows)
leonfin at optonline.net
Fri Jun 5 07:52:51 PDT 2009
>You may be right that Windows doesn't do this with
>asynchronous I/O, at least for when buffering is
>disabled (which is what I think you are doing).
>Is this specified anywhere?
I did disable send buffering. I did not find this
publicly documented by Microsoft.
I only know that windows API and .NET does not enforce
this. Also, it says the following in MSDN for WSASend:
"If you are using I/O completion ports, be aware that
the order of calls made to WSASend is also the order
in which the buffers are populated. WSASend should
not be called on the same socket simultaneously from
different threads, because it can result in an
unpredictable buffer order." They don't say that
one has to wait for completion of WSASends before
scheduling next ones (as long as it's not done
concurrently). Although, short-writes will make this
pointless if they are to happen regularly on stream
connections. If they only happen in IOCP under low
resources conditions, then it's probably advantageous
to cleanup some of the connections. This is what's not
documented, but only observed in practice.
>What size are the messages and have you
>disabled Nagle by any chance? Also, what is the
>size of the send buffer that you are comparing with?
I was testing with message sizes of 100, 2048, 4096 bytes.
I tried disabling Nagle, and it only affected testing of small
message sizes (100 bytes). Made throughput much slower.
I tried send buffer size of 0, 8192, 512K.
From: Alan.Bateman at Sun.COM [mailto:Alan.Bateman at Sun.COM]
Sent: Thursday, June 04, 2009 8:25 PM
To: Leon Finker
Cc: nio-dev at openjdk.java.net
Subject: Re: TCP Sockets - Limiting one write at a time is harming
performance (at least on windows)
Leon Finker wrote:
> Yes, short-writes will cause a problem, but on windows with
> model short-writes are not observed in practice (unless resources are
> depleted; in that case connection probably should be closed/cleaned up
> anyway). Not sure about other platforms.
> Have you ever seen (with latest jdk build) that ByteBuffer#hasRemaining is
> true in write's completed() on windows under stress (more likely to happen
> on pre-Vista due to easier non-paged pool resource depletion)? Even though
> Platform SDK IOCP sample does handle this case, I have not seen it being
> handled in other production frameworks. Anyhow, something to ponder on.
> data throughput is definitely much better.
J1 is this week so a belated reply.
Short-writes are "normal" when there is congestion and insufficient
space in the socket's send buffer. Even Windows does this with blocking
and non-blocking I/O. You may be right that Windows doesn't do this with
asynchronous I/O, at least for when buffering is disabled (which is what
I think you are doing). Is this specified anywhere? In any case, the
2x-3x results are kinda surprising. I recall the previous version of
your tests completely saturated the network so I'm not sure how the
throughput is increased. What size are the messages and have you
disabled Nagle by any chance? Also, what is the size of the send buffer
that you are comparing with?
More information about the nio-dev