#define S_DOCUMENTATION
#define S_DOCUMENTATION_DEVELOPER
#define PAGE Documentation
#define SUBPAGE Developer
#define TITLE InfiniBand

#include "header.shtml"

<p>
  Author: <a href="mailto:eilemann@gmail.com">eilemann@gmail.com</a><br>
  State: Implemented in 0.3
</p>

<h2>Overview</h2>
<p>
  InfiniBand (IB) is supported using the SDP layer. This layer is transparent to
  socket-based applications, such as the Equalizer network layer. This document
  describes the setup to use IB hardware with Equalizer applications.
</p>

<h2>Windows XP</h2>
<h3>Overview</h3>
<p>
  The usage of InfiniBand requires a Mellanox card, since Mellanox is the only
  vendor providing a SDP implementation. Often cards from other vendors are
  Mellanox cards with a different firmware and can be 'converted' by flashing a
  Mellanox firmware. The Mellanox Windows SDP implementation is described in
  great detail in the
  <a href="http://www.mellanox.com/pdf/whitepapers/SDPCluster2006.pdf">Architecture
  and Implementation of Sockets Direct Protocol in Windows</a> paper.
</p>
<p>
  The Mellanox Win32 implementation allows currently no mixing between SDP
  sockets and TCP/IP sockets. That means that all machines have to be
  connected using InfiniBand, and need to have the full installation as decribed
  below. Furthermore, only SDP IP addresses are to be used for all connections.
</p>

<h3>Installation and Configuration</h3>
<p>
  The following steps are necessary to use SDP on Windows:
</p>
<ul>
  <li>Download and install the WinIB Software from Mellanox
    (<a href="http://www.mellanox.com">www.mellanox.com</a>, go to
    Product: Software: WinIB). The version tested was 1.3.0.</li>
  <li>At the end of the installer, select 'Enable Socket Direct Protocol'</li>
  <li>Setup the network ports with private IP addressing, e.g., address
    10.1.1.x, netmask 255.255.255.0, no default gateway or DNS.</li>
  <li>If you are running a switchless configuration, start one subnet
    manager per IB port (%ProgramFiles%/Mellanox/WinIB/bin/OpenSM.exe).</li>
  <li>Start the SDP service: <code>net start sdp</code>.</li>
  <li>Set all Equalizer applications to be 'SDP Applications'. This is done
    using the environment
    variable <code>SdpApplications</code>. For <code>eqPly</code> it has to be
    set to <code>eqServer.exe;eqPly.exe</code>.</li>
  <li>Optional: Verify network performance using <code>netPerf</code>. Set
    SdpApplications to netPerf.exe, run one server (<code>netPerf.exe -s
    IB_ADDR</code>) on one machine a client (<code>netPerf.exe -c
      SERVER_IB_ADDR</code>) on another machine. On SDR IB hardware, the
    performance is around 700-800 MB/s using SDP, as compared to 100-200 MB/s
    using IPoIB.</li>
  <li>Use the IP addresses, as configured above, as the hostname for the node
    connection settings in the configuration files.</li>
  <li>Start the server using the SDP server address, for
    example: <code>eqServer.exe config.eqc --eq-listen 10.1.1.x</code>.</li>
  <li>Start the application so that it connects to the SDP server address and
    uses the local SDP as listening address (instead of localhost), for
    example: <code>eqPly.exe -- --eq-server 10.1.1.x --eq-listen
    10.1.1.x</code>.</li>
  <li>Depending on the CPU speed, disabling image compression might improve
    performance. Compression is enabled in <code>definesWin32.h</code>
    by <code>EQ_USE_COMPRESSION</code>.</li>
</ul>

<a name="explicit"></a>
<h3>Using Explicit SDP Connections</h3>
<p>
  Mellanox is working on providing a patch to allow applications to explicitly
  specify the socket type, which allows mixing of TCP and SDP sockets. If you
  have access to this patch, you can use TCP and SDP concurrently. Revisions 
  <a href="/cgi-bin/viewvc.cgi?view=rev&revision=1001">1001</a> to 
  <a href="/cgi-bin/viewvc.cgi?view=rev&revision=1010">1010</a> implement
  support for multiple node connections, which means that if a node (or the
  server) has multiple connections specified in the configuration file, they
  now listen to <em>all</em> of them, allowing to use mixed connection
  types. When a node connects to another node, the connections are tried in the
  order they are specified. Therefore connections should be configured in the
  fastest-to-slowest order.
</p>
<p>
  Transparent SDP support, as described above, still works the same
  way. Explicit SDP support is used by not setting the variable SdpApplications,
  and explicitly configure SDP socket connections:
  <ul>
    <li>Unset SdpApplications.</li>
    <li>Set the global connection type to SDP (EQ_CONNECTION_IATTR_TYPE SDP in
      config file globals).</li>
    <li>Start the server as usual, i.e., without the <code>--eq-listen</code>
      parameter. The server will create one default connection (SDP), unless
      connections are specified explicetely in the configuration file.</li>
    <li>Use SDP IP addresses for the nodes in the config file.</code>
    <li>Start the client with <code>eqPly.exe -- --eq-server 10.1.1.x:SDP 
        --eq-listen 10.1.1.x:SDP</code>. Note the :SDP qualifier, which is
      needed for the client to use SDP (it is unaware of the configuration file
      during startup!).</li>
    <li>Optionally configure (secondary) TCPIP connections. Note that when a
      node has no SDP capabilities, the server and all nodes he connects to
      (potentially all in the configuration) need to have a secondary TCPIP
      connection configured.</li>
  </ul>
</p>

<h3>Implementation</h3>
<p>
  The current (1.3.0) Mellanox Windows SDP implementation only supports a subset
  of the Windows socket interface. Most notably, the eq::net socket connection was
  rewritten to use overlapped (asynchronous) IO with 
  <a href="/cgi-bin/viewvc.cgi?view=rev&revision=951">svn
    revision 951</a>. TCP/IP connections also exhibit a performance increase due
  to the use of overlapped IO.
</p>
<p>
  The <code>--eq-listen</code> parsing was extended to support a simplified
  connection description in the form <code>hostname(:port)(:type)</code>. Port
  is an unsigned short value, type is either <code>SDP</code>
  or <code>TCPIP</code>.
</p>
<p>
  The interpretation of the node configuration has changed. When multiple
  connections are described, they are all set up as a listening connection
  during Node::initLocal(). Therefore, nodes can now listen on multiple sockets
  for incoming connections.
</p>
<p>
  The connection setup code between two nodes has been improved.
</p>
<p>
  The config file now accepts connection descriptions for the server section.
</p>

<p>
  Related SVN changes:
  <a href="/cgi-bin/viewvc.cgi?view=rev&revision=951">951</a>
  <a href="/cgi-bin/viewvc.cgi?view=rev&revision=952">952</a>
  <a href="/cgi-bin/viewvc.cgi?view=rev&revision=953">953</a>
  <a href="/cgi-bin/viewvc.cgi?view=rev&revision=1001">1001</a>
  <a href="/cgi-bin/viewvc.cgi?view=rev&revision=1002">1002</a>
  <a href="/cgi-bin/viewvc.cgi?view=rev&revision=1006">1006</a>
  <a href="/cgi-bin/viewvc.cgi?view=rev&revision=1007">1007</a>
  <a href="/cgi-bin/viewvc.cgi?view=rev&revision=1009">1009</a>
  <a href="/cgi-bin/viewvc.cgi?view=rev&revision=1010">1010</a>
</p>

<h3>Known Bugs</h3>
<p>
  When using SDP, the client process hangs upon exit. We are investigating this
  issue. Defining EQ_WIN32_SDP_JOIN_WAR in
  <a href="/cgi-bin/viewvc.cgi/trunk/src/lib/base/definesWin32.h?view=markup">definesWin32.h</a> enables a workaround.
</p>

<h2>Linux</h2>
<p>
  The Linux usage of SDP is very similar to the Windows usage described
  above. For the implicit, transparent mode just LD_PRELOAD the SDP library
  (typically libsdp.so) instead of setting SdpApplications. For explicit
  support, follow the same configuration as for Windows.
</p>

#include "footer.shtml"
<!-- $Id$ -->