SourceForge.net Logo

Tsunami UDP Protocol

Tsunami UDP Protocol: A fast user-space file transfer protocol that uses TCP control and UDP data for transfer over very high speed long distance networks (≥ 1 Gbps and even 10 GE), designed to provide more throughput than possible with TCP over the same networks. The project is based on original Indiana University 2002 Tsunami source code, but has been significantly improved and extended. As such, large portions of the program today are courtesy by Aalto University Metsähovi Radio Observatory.

Includes FTP-like client and server command line applications for normal file transfers. It has additionally been extended for high rate real-time data streaming in eVLBI radio astronomy and geodesy (VSIB, PCEVN DAQ). Licensed under the original IU open source license.

Updates

Version 1.1 cvsbuild 40 has automatic achieved rate memorization and several changes for smoother transfers. Beginning with version 1.1 cvsbuild 29 Tsunami now (09/2007) supports transfers between different-endian machines. Listing shared files with dir is supported since cvsbuild 27. Mac OS X / Darwin works beginning with cvsuild 36. For more changes, see the changelog.

The previous CVS update was on 22 December 2009.

Background

The original early Tsunami UDP transfer protocol developed and released to the public in 2002 by Mark Meiss et al. at the Pervasive Technology Labs at Indiana University (IU) can still be found at http://anml.iu.edu/research.shtml?prim=lab_research.

This project is a derived work - if not a complete rewrite - of the original IU source code. The new Tsunami includes a large number of bug fixes, improvements and new features. File releases in the form of compressed source code archives can be downloaded from the project page at http://sourceforge.net/projects/tsunami-udp, but I recommend you'd use the CVS version.

Source code access

Currently no binaries are distributed, only the source code is available and should be compiled on the target system. It should compile fine on all POSIX systems and on GNU/Linux, no special libraries or toolkits are required. Microsoft Windows and Vista are not supported (XP offers an optional POSIX layer, but some parts of Tsunami code would have to be slightly adapted neverheless, and unfortunately in Vista Microsoft has abandoned POSIX altogether).

Any current version of source code pulled from CVS HEAD will compile and work, and has been tested somewhat extensively. The CVS version is likely to contain new features. There are no guarantees any new features are going to work correctly with older versions of the Tsunami client/server. Nevertheless, in the basic features, Tsunami generally is backwards compatible.

If you want to get a non-changing stable release version, you may check out the following CVS tags: v10_RTM_protorev20061025 (v1.0 released 31. Nov 2006), v11_build11_protorev20061025 (v1.1 build 11 released 5. Jan 2007), v11_build16_protorev20061025 (v1.1 build 16 released 24. May 2007), v11_build29_protorev20061025 (v1.1 build 29 released 29. Aug 2007), v11_build36_protorev20061025 (v1.1 build 36 released 26. May 2008). There are other tags, please see ViewCV. Alternatively, some probably significantly less up to date source code .tar.gz files can be downloaded from file releases.

Source code and compile cheatsheet
Anonymous CVS checkout:
current version
cvs -z3 -d:pserver:anonymous@tsunami-udp.cvs.sourceforge.net:/cvsroot/tsunami-udp co -P tsunami-udp
older versions by tag, for example
cvs -z3 -d:pserver:anonymous@tsunami-udp.cvs.sourceforge.net:/cvsroot/tsunami-udp co -P -r v11_build29_protorev20061025 tsunami-udp
ViewCVS repository source: http://tsunami-udp.cvs.sourceforge.net/tsunami-udp/tsunami-udp/
ViewCVS repository docs: http://tsunami-udp.cvs.sourceforge.net/tsunami-udp/docs/
Compiling: COMPILING.txt
Changelog: ChangeLog
TODO and known bugs: BUGS-and-TODO.txt
 
Users Guide, Troubleshooting and Realtime
Quick Guide: USAGE.txt including Troubleshooting
How Tsunami works: howTsunamiWorks.txt

09Nov07: Those with a sluggish Petabit/sec network connection ;) can try the protocol version 1.2 in CVS branch protocol_v12_branch. Still unstable and experimental. The set rate value in v1.1 has only 32 bits and thus limits the user-settable rate to 4 Gbps. Not so in v1.2. Checkout like above but with tag -r protocol_v12_branch tsunami-udp. Note that v1.2 can't connect to v1.1 protocol version software and vice versa. Later goals are to add blocking send_data() and receive_data() API functions to make Tsunami less a file and more a data transfer protocol.


Hardware requirements

Here is a short summary of vague hardware requirements. It's nothing fancy apart from the RAID-0 disk requirement.

Typical transfer rate examples

Below are a few examples of transfer rates that Tsunami can achieve over the Internet. On a local LAN, for disk-to-disk transfers rates near the limit of the PC NIC or on-board LAN are easily achieved, which is usually around 940 Mbps for 1G LAN. This is provided that the disk system in both PCs is adequately fast, e.g. simple software RAID-0 with three or four SATA disks (no RAID controller necessary).

Disk-to-disk averaging 800 Mbps over 
Internet with cheap PCs and motherboard 1G LAN

Figure 1 - international transfer over the Internet, averaging 800 Mbit/second

Figure 1 shows a throughput plot of a disk-to-disk transfer where several multi-gigabyte files are transferred one after another from Helsinki, Finland to Bonn, Germany. Both PCs have a fast (>1 Gbps capable) RAID-0. Linux software RAID-0 with 4 SATA disks on the sending side, with a Dell Optiplex GX620 cheap PC. SATA RAID card at the receiving side. The server side had 10 Gbps fibre to the Finnish 'Funet' GEANT2 NREN, the client side had a shared fibre to the Viola GEANT2 NREN with ~1 Gbps allocated capacity.

Steady Tsunami v1.2 transfer plot of a 
20GB filetransferred over 10G LAN to disk
Figure 2 - a short 20GB file transferred with Tsunami v1.2 over local 10 Gbps LAN at Metsähovi.

Figure 2 shows a Tsunami v1.2 test transfer on the local 10G LAN at Metsähovi. RAID-0 were built with Samsung SpinPoint F1 disks on dual and quad-core consumer computer systems with Chelsio 10G cards connected via a HP6400cl switch. The transfer rate is limited to 4.3 Gbps by disk I/O. With diskless transfer a rate of 7 Gbps could be achieved.

 


Project admin and developer: Jan Wagner,