4) Interconnect Network
On the extreme parallel Cluster system, performence and scalability stability depend on interconnect network construction for message passing between cluster node.
Generally, small-scale cluster consists of construction , management in low cost and simple consturction ethernet , but as Cluster’s scale is larger, whole cluster performence’s effency is decreased by Ethernet’s large latency , and cluster scalibility is also limited, large cluster system consists of network which are Infiniband, Quadric, Myrinet between computer node.

From large scale Cluster example On Top500.org, InfiniBand instead of existing Myfinet is universally used for Cluster’s Interconnect Network.
InfiniBand is that intergrated result of both Future I/O by Compaq, IBM, HP and Next Generation I/O(ngio) by Sun, Intel, MS, and mainly used HPC as Switched fabric. It includes quality of service(QoS) and failover function and especially offers scalability. It is mainly used as interface between preocessor node or between high performance I/O node on HPC.
IB offers below Theoretical throughput according to Signalling rate.

Effective theoretical throughput in different configurations

Because serial connection’s signalling rate is 2.5 Gbps/per connection, It offers 10Gbps in theory in case of 4X IB( SDR(Single Data Rate)) and offers 20Gbps in case of DDR(Double Data Rate).
Because of IB’s 8B/10B encoding, Data throughput is 4/5 of signalling rate which is 8Gbps, 16GBps.
From Top500, the reason that IB is choosen as Cluster intercinnect is due to high throughput, low Latency, and relative low construct cost.

InfiniBand Switch’s main vendors areCisco which takes over Topspin and Voltaire, SilverStorm.

Source/Kaynak : http://blogs.sun.com/jjson/entry/3_hpc_components_3_interconnect