~mbuffer/
home of measuring buffer

Description

mbuffer is a tool for buffering data streams with a large set of unique features:

direct support for TCP based network targets (IPv4 and IPv6)
ability to send to multiple targets in parallel (distribution mode)
support for multiple volumes
I/O rate limitation
high/low watermark based restart criteria
configurable buffer size
on the fly MD5 hash calculation
highly efficient, multi-threaded implementation

It is especially useful, if you are writing backups to fast tape drives or libraries. Those drives tend to stop and rewind if they have a buffer underrun. This so called tape screwing reduces the lifetime of the motors.
mbuffer can prevent buffer underruns, if used correctly and speed up the whole backup process. Please read the man page for details, how to use it.

Technology/Features

mbuffer is written in C and uses the POSIX API. Initially, I developed it under Linux to learn multi-threaded programming and speed up my own backup process, which used a DAT tape. Later it got enhanced features like network support. Today I develop mbuffer on Solaris and only use it from time to time to mirror some diretory tree from one machine to another.

Usage/Command line options

-b <num> : use <num> blocks for buffer (default: 157)
-s <size> : use blocks of <size> bytes for processing (default: 2097152)
-m <size> : memory <size> of buffer in b,k,M,G,% (default: 2% = 314M)
-L : lock buffer in memory (unusable with file based buffers)
-d : use blocksize of device for output
-D <size> : assumed output device size (default: infinite/auto-detect)
-P <num> : start writing after buffer has been filled more than <num>%
-p <num> : start reading after buffer has been filled less than <num>%
-i <file> : use <file> for input
-o <file> : use <file> for output (this option can be passed MULTIPLE times)
--append : append to output file (must be passed before -o)
--truncate : truncate next file (must be passed before -o)
-I <h>:<p> : use network port <port> as input, allow only host <h> to connect
-I <p> : use network port <port> as input
-O <h>:<p> : output data to host <h> and port <p> (MUTLIPLE outputs supported)
-n <num> : <num> volumes for input, '0' to prompt interactively
-t : use memory mapped temporary file (for huge buffer)
-T <file> : as -t but uses <file> as buffer
-l <file> : use <file> for logging messages
-u <num> : pause <num> milliseconds after each write
-r <rate> : limit read rate to <rate> B/s, where <rate> can be given in b,k,M,G
-R <rate> : same as -r for writing; use either one, if your tape is too fast
-f : overwrite existing files
-a <time> : autoloader which needs <time> seconds to reload
-A <cmd> : issue command <cmd> to request new volume
-v <level> : set verbose level to <level> (valid values are 0..6)
-q : quiet - do not display the status on stderr
-Q : quiet - do not log the status
-c : write with synchronous data integrity support
-e : stop processing on any kind of error
--direct : open input and output with O_DIRECT
-H
--md5 : generate md5 hash of transfered data
--hash <a> : use alogritm <a>, if <a> is 'list' possible algorithms are listed
--pid : print PID of this instance
-W <time> : set watchdog timeout to <time> seconds
-4 : force use of IPv4
-6 : force use of IPv6
-0 : use IPv4 or IPv6
--tcpbuffer: size for TCP buffer
--tapeaware: write to end of tape instead of stopping when the drive signals
the media end is approaching (write until 2x ENOSPC errors)
-V
--version : print version information

Download

Sources

LICENSE:

GNU GPL 3

DESCRIPTION:

mbuffer is a replacement for buffer with additional functionality:

multi-target network output
on the fly hash calculation
display of I/O speed
optional use of memory mapped I/O for huge buffers
multithreaded instead of sharedmemory ipc
multi volume support
autoloader support
networking support
command-line options compatible to buffer

TESTING:

to test the program I run an integrity check with gpg

ARCHITECTURES:

This software has been successfully tested under the following operatingsystems:

Solaris (primary target)
FreeBSD
Linux
Tru64

Thomas Maier-Komor

~mbuffer/ home of measuring buffer