<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto"><div></div><div>Hi Gustabvo,</div><div><br></div><div>I am not in the PPC64 ML, so replying late. </div><div><br></div><div><blockquote type="cite"><br></blockquote><blockquote type="cite"><blockquote type="cite"><font color="#000000"><span style="background-color: rgba(255, 255, 255, 0);">After a due community review, could you sponsor that change?</span></font></blockquote></blockquote>Sure, I can sponsor this patch after the review. Please initiate the review on jdk 10 base. </div><div><br></div><div>Thanks,</div><div>Sangheon</div><div><br></div><div><br>On Feb 27, 2017, at 8:10 AM, David Holmes <<a href="mailto:david.holmes@oracle.com">david.holmes@oracle.com</a>> wrote:<br><br></div><blockquote type="cite"><div><span>Hi Gustavo,</span><br><span></span><br><span>I am not a NUMA expert but it seems to me that our NUMA support is both incomplete and bit-rotting. It seems evident that UseNUMA is only working in limited contexts that match our testing environment. There were a couple of JEPS proposed to enhance NUMA support back in 2012:</span><br><span></span><br><span>JDK-8046147 JEP 157: G1 GC: NUMA-Aware Allocation</span><br><span>JDK-8046153 JEP 163: Enable NUMA Mode by Default When Appropriate</span><br><span></span><br><span>but they have not progressed. If they were to progress then it seems our overall approach to NUMA would need serious review and update - as per your patch.</span><br><span></span><br><span>I'm also unclear about the distinctions between memory and non-memory nodes wrt. the existing os::Linux NUMA API. It isn't at all clear to me what functions should only be dealing with memory-nodes and which should deal with any kind eg. I expect cpu to node map to map cpu to nodes not cpu to nearest node with memory configured. If that is what is needed then the API's need to be changed and the usage checked - aas that distinction does not presently exist in the code AFAICS.</span><br><span></span><br><span>It is too late to take this patch into 9 IMHO as we don't have the ability to test it effectively, nor is there time for NUMA users to put it through its paces. I think this would have to be part of a bigger NUMA project for 10 that addresses the NUMA API and how it is used.</span><br><span></span><br><span>Thanks,</span><br><span>David</span><br><span></span><br><span>On 24/02/2017 10:02 PM, Gustavo Romero wrote:</span><br><blockquote type="cite"><span>Hi Sangheon,</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>Please find my comments inline.</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>On 06-02-2017 20:23, sangheon wrote:</span><br></blockquote><blockquote type="cite"><blockquote type="cite"><span>Hi Gustavo,</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>On 02/06/2017 01:50 PM, Gustavo Romero wrote:</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>Hi,</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>On Linux/PPC64 I'm getting a series of "mbind: Invalid argument" that seems</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span> exactly the same as reported for x64 [1]:</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>[root@spocfire3 ~]# java -XX:+UseNUMA -version</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>mbind: Invalid argument</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>mbind: Invalid argument</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>mbind: Invalid argument</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>mbind: Invalid argument</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>mbind: Invalid argument</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>mbind: Invalid argument</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>mbind: Invalid argument</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>openjdk version "1.8.0_121"</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>OpenJDK Runtime Environment (build 1.8.0_121-b13)</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>OpenJDK 64-Bit Server VM (build 25.121-b13, mixed mode)</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>[root@spocfire3 ~]# uname -a</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>Linux <a href="http://spocfire3.aus.stglabs.ibm.com">spocfire3.aus.stglabs.ibm.com</a> 3.10.0-327.el7.ppc64le #1 SMP Thu Oct 29 17:31:13 EDT 2015 ppc64le ppc64le ppc64le GNU/Linux</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>[root@spocfire3 ~]# lscpu</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>Architecture: ppc64le</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>Byte Order: Little Endian</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>CPU(s): 160</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>On-line CPU(s) list: 0-159</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>Thread(s) per core: 8</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>Core(s) per socket: 10</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>Socket(s): 2</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>NUMA node(s): 2</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>Model: 2.0 (pvr 004d 0200)</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>Model name: POWER8 (raw), altivec supported</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>L1d cache: 64K</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>L1i cache: 32K</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>L2 cache: 512K</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>L3 cache: 8192K</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>NUMA node0 CPU(s): 0-79</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>NUMA node8 CPU(s): 80-159</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>On chasing down it, looks like it comes from PSYoungGen::initialize() in</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>src/share/vm/gc_implementation/parallelScavenge/psYoungGen.cpp that calls</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>initialize_work(), that calls the MutableNUMASpace() constructor if</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>UseNUMA is set:</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span><a href="http://hg.openjdk.java.net/jdk8u/jdk8u/hotspot/file/567e410935e5/src/share/vm/gc_implementation/parallelScavenge/psYoungGen.cpp#l77">http://hg.openjdk.java.net/jdk8u/jdk8u/hotspot/file/567e410935e5/src/share/vm/gc_implementation/parallelScavenge/psYoungGen.cpp#l77</a></span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>MutableNUMASpace() then calls os::numa_make_local(), that in the end calls</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>numa_set_bind_policy() in libnuma.so.1 [2].</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>I've traced some values for which mbind() syscall fails:</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span><a href="http://termbin.com/ztfs">http://termbin.com/ztfs</a> (search for "Invalid argument" in the log).</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>Assuming it's the same bug as reported in [1] and so it's not fixed on 9 and 10:</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>- Is there any WIP or known workaround?</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>There's no progress on JDK-8163796 and no workaround found yet.</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>And unfortunately, I'm not planning to fix it soon.</span><br></blockquote></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>Hive, a critical component of Hadoop ecosystem, comes with a shell and uses java</span><br></blockquote><blockquote type="cite"><span>(with UseNUMA flag) in the background to run mysql-kind of queries. On PPC64 the</span><br></blockquote><blockquote type="cite"><span>mbind() messages in question make the shell pretty cumbersome. For instance:</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>hive> show databases;</span><br></blockquote><blockquote type="cite"><span>mbind: Invalid argument</span><br></blockquote><blockquote type="cite"><span>mbind: Invalid argument</span><br></blockquote><blockquote type="cite"><span>mbind: Invalid argument (repeat message more 28 times...)</span><br></blockquote><blockquote type="cite"><span>...</span><br></blockquote><blockquote type="cite"><span>OK</span><br></blockquote><blockquote type="cite"><span>mbind: Invalid argument</span><br></blockquote><blockquote type="cite"><span>mbind: Invalid argument</span><br></blockquote><blockquote type="cite"><span>mbind: Invalid argument</span><br></blockquote><blockquote type="cite"><span>default</span><br></blockquote><blockquote type="cite"><span>tpcds_bin_partitioned_orc_10</span><br></blockquote><blockquote type="cite"><span>tpcds_text_10</span><br></blockquote><blockquote type="cite"><span>Time taken: 1.036 seconds, Fetched: 3 row(s)</span><br></blockquote><blockquote type="cite"><span>hive> mbind: Invalid argument</span><br></blockquote><blockquote type="cite"><span>mbind: Invalid argument</span><br></blockquote><blockquote type="cite"><span>mbind: Invalid argument</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>Also, on PPC64 a simple "java -XX:+UseParallelGC -XX:+UseNUMA -version" will</span><br></blockquote><blockquote type="cite"><span>trigger the problem, without any additional flags. So I'd like to correct that</span><br></blockquote><blockquote type="cite"><span>behavior (please see my next comment on that).</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>- Should I append this output in [1] description or open a new one and make it</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span> related to" [1]?</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>I think your problem seems same as JDK-8163796, so adding your output on the CR seems good.</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>And please add logs as well. I recommend to enabling something like "-Xlog:gc*,gc+heap*=trace".</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>IIRC, the problem was only occurred when the -Xmx was small in my case.</span><br></blockquote></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>JVM code used to discover which numa nodes it can bind assumes that nodes are</span><br></blockquote><blockquote type="cite"><span>consecutive and tries to bind from 0 to numa_max_node() [1, 2, 3, 4], i.e. from</span><br></blockquote><blockquote type="cite"><span>0 to the highest node number available on the system. However, at least on PPC64</span><br></blockquote><blockquote type="cite"><span>that assumption is not always true. For instance, consider the following numa</span><br></blockquote><blockquote type="cite"><span>topology:</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>available: 4 nodes (0-1,16-17)</span><br></blockquote><blockquote type="cite"><span>node 0 cpus: 0 8 16 24 32</span><br></blockquote><blockquote type="cite"><span>node 0 size: 130706 MB</span><br></blockquote><blockquote type="cite"><span>node 0 free: 145 MB</span><br></blockquote><blockquote type="cite"><span>node 1 cpus: 40 48 56 64 72</span><br></blockquote><blockquote type="cite"><span>node 1 size: 0 MB</span><br></blockquote><blockquote type="cite"><span>node 1 free: 0 MB</span><br></blockquote><blockquote type="cite"><span>node 16 cpus: 80 88 96 104 112</span><br></blockquote><blockquote type="cite"><span>node 16 size: 130630 MB</span><br></blockquote><blockquote type="cite"><span>node 16 free: 529 MB</span><br></blockquote><blockquote type="cite"><span>node 17 cpus: 120 128 136 144 152</span><br></blockquote><blockquote type="cite"><span>node 17 size: 0 MB</span><br></blockquote><blockquote type="cite"><span>node 17 free: 0 MB</span><br></blockquote><blockquote type="cite"><span>node distances:</span><br></blockquote><blockquote type="cite"><span>node 0 1 16 17</span><br></blockquote><blockquote type="cite"><span> 0: 10 20 40 40</span><br></blockquote><blockquote type="cite"><span> 1: 20 10 40 40</span><br></blockquote><blockquote type="cite"><span> 16: 40 40 10 20</span><br></blockquote><blockquote type="cite"><span> 17: 40 40 20 10</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>In that case we have four nodes, 2 without memory (1 and 17), where the</span><br></blockquote><blockquote type="cite"><span>highest node id is 17. Hence if the JVM tries to bind from 0 to 17, mbind() will</span><br></blockquote><blockquote type="cite"><span>fail except for nodes 0 and 16, which are configured and have memory. mbind()</span><br></blockquote><blockquote type="cite"><span>failures will generate the "mbind: Invalid argument" messages.</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>A solution would be to use in os::numa_get_group_num() not numa_max_node() but</span><br></blockquote><blockquote type="cite"><span>instead numa_num_configured_nodes() which returns the total number of nodes with</span><br></blockquote><blockquote type="cite"><span>memory in the system (so in our example above it will return exactly 2 nodes)</span><br></blockquote><blockquote type="cite"><span>and then inspect numa_all_node_ptr in os::numa_get_leaf_groups() to find the</span><br></blockquote><blockquote type="cite"><span>correct node ids to append (in our case, map ids[0] = 0 [node 0] and ids[1] = 16</span><br></blockquote><blockquote type="cite"><span>[node 16]).</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>One thing is that os::numa_get_leaf_groups() argument "size" will not be</span><br></blockquote><blockquote type="cite"><span>required anymore and will be loose, so the interface will have to be adapted on</span><br></blockquote><blockquote type="cite"><span>other OSs besides Linux I guess [5].</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>It would be necessary to adapt os::Linux::rebuild_cpu_to_node_map()</span><br></blockquote><blockquote type="cite"><span>since not all numa nodes are suitable to be returned by a call to</span><br></blockquote><blockquote type="cite"><span>os::numa_get_group_id() as some cpus would be in a node without memory.</span><br></blockquote><blockquote type="cite"><span>In that case we can return the closest numa node instead. A new way to translate</span><br></blockquote><blockquote type="cite"><span>indices to nodes is also useful since nodes are not always consecutive.</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>Finally, although "numa_nodes_ptr" is not present in libnuma's manual it's what</span><br></blockquote><blockquote type="cite"><span>is used in numactl to find out the total number of nodes in the system [6]. I</span><br></blockquote><blockquote type="cite"><span>could not find a function that would return that number readily. I asked to</span><br></blockquote><blockquote type="cite"><span>libnuma ML if a better solution exists [7].</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>The following webrev implements the proposed changes on jdk9 (backport to 8 is</span><br></blockquote><blockquote type="cite"><span>simple):</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>webrev: <a href="http://cr.openjdk.java.net/~gromero/8175813/">http://cr.openjdk.java.net/~gromero/8175813/</a></span><br></blockquote><blockquote type="cite"><span>bug: <a href="https://bugs.openjdk.java.net/browse/JDK-8175813">https://bugs.openjdk.java.net/browse/JDK-8175813</a></span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>Here are the logs with "-Xlog:gc*,gc+heap*=trace":</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span><a href="http://cr.openjdk.java.net/~gromero/logs/pristine.log">http://cr.openjdk.java.net/~gromero/logs/pristine.log</a> (current state)</span><br></blockquote><blockquote type="cite"><span><a href="http://cr.openjdk.java.net/~gromero/logs/numa_patched.log">http://cr.openjdk.java.net/~gromero/logs/numa_patched.log</a> (proposed change)</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>I've tested on 8 against SPECjvm2008 on the aforementioned machine and</span><br></blockquote><blockquote type="cite"><span>performance improved ~5% in comparison to the same version packaged by</span><br></blockquote><blockquote type="cite"><span>the distro, but I don't expect any difference on machines where nodes</span><br></blockquote><blockquote type="cite"><span>are always consecutive and where nodes always have memory.</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>After a due community review, could you sponsor that change?</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>Thank you.</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>Best regards,</span><br></blockquote><blockquote type="cite"><span>Gustavo</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>[1] <a href="http://hg.openjdk.java.net/jdk9/hs/hotspot/file/636271c3697a/src/share/vm/gc/parallel/mutableNUMASpace.cpp#l241">http://hg.openjdk.java.net/jdk9/hs/hotspot/file/636271c3697a/src/share/vm/gc/parallel/mutableNUMASpace.cpp#l241</a></span><br></blockquote><blockquote type="cite"><span>[2] <a href="http://hg.openjdk.java.net/jdk9/hs/hotspot/file/636271c3697a/src/os/linux/vm/os_linux.cpp#l2745">http://hg.openjdk.java.net/jdk9/hs/hotspot/file/636271c3697a/src/os/linux/vm/os_linux.cpp#l2745</a></span><br></blockquote><blockquote type="cite"><span>[3] <a href="http://hg.openjdk.java.net/jdk9/hs/hotspot/file/636271c3697a/src/share/vm/gc/parallel/mutableNUMASpace.cpp#l243">http://hg.openjdk.java.net/jdk9/hs/hotspot/file/636271c3697a/src/share/vm/gc/parallel/mutableNUMASpace.cpp#l243</a></span><br></blockquote><blockquote type="cite"><span>[4] <a href="http://hg.openjdk.java.net/jdk9/hs/hotspot/file/636271c3697a/src/os/linux/vm/os_linux.cpp#l2761">http://hg.openjdk.java.net/jdk9/hs/hotspot/file/636271c3697a/src/os/linux/vm/os_linux.cpp#l2761</a></span><br></blockquote><blockquote type="cite"><span>[5] <a href="http://hg.openjdk.java.net/jdk9/hs/hotspot/file/636271c3697a/src/share/vm/runtime/os.hpp#l356">http://hg.openjdk.java.net/jdk9/hs/hotspot/file/636271c3697a/src/share/vm/runtime/os.hpp#l356</a></span><br></blockquote><blockquote type="cite"><span>[6] <a href="https://github.com/numactl/numactl/blob/master/numactl.c#L251">https://github.com/numactl/numactl/blob/master/numactl.c#L251</a></span><br></blockquote><blockquote type="cite"><span>[7] <a href="http://www.spinics.net/lists/linux-numa/msg01173.html">http://www.spinics.net/lists/linux-numa/msg01173.html</a></span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>Thanks,</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>Sangheon</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>Thank you.</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>Best regards,</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>Gustavo</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>[1] <a href="https://bugs.openjdk.java.net/browse/JDK-8163796">https://bugs.openjdk.java.net/browse/JDK-8163796</a></span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>[2] <a href="https://da.gd/4vXF">https://da.gd/4vXF</a></span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote><blockquote type="cite"><span></span><br></blockquote></div></blockquote></body></html>