= fireperf(8)

== NAME
fireperf - A Network Benchmarking Tool

== SYNOPSIS
[verse]
`fireperf --server ...`
`fireperf --client HOST ...`

== DESCRIPTION
`fireperf` can perform network benchmarks to test throughput and quality of
a network between a fireperf client and fireperf server.

The client decides what kind of test is being run and tells the server when it
connects. A server does not need any options for that and can serve different tests
at the same time. `fireperf` can also talk to `iperf` (version 2) and `iperf3`, see
INTEROPERABILITY below.


== COMMON OPTIONS

--ipv4::
-4::
	Only uses IPv4. The server only listens on IPv4, and the client connects to an
	IPv4 address of HOST.

--ipv6::
-6::
	Only uses IPv6. The server only listens on IPv6, and the client connects to an
	IPv6 address of HOST.

--backend=NAME::
-b NAME::
	Selects how `fireperf` performs its I/O. NAME is one of:
	+
	* `auto`: Use `io_uring` if it is available, and `epoll` otherwise (this is the
	  default).
	* `io_uring`: Use io_uring. Over a network, it sends without copying at less than
	  half of the CPU time of `epoll`. It is not available if `fireperf` has been built
	  without it, on old kernels, or where it has been disabled (which is common in
	  containers). Over loopback, the kernel copies what it sends anyway, which makes
	  `epoll` the faster one there.
	* `epoll`: Use epoll, which works everywhere. It sends without copying, too
	  (see `--zerocopy`).

--debug::
-d::
	Enables debugging output
	+
	If `fireperf` has been compiled with debugging support, this command will print
	various debug messages that will help to find bugs in the program.

--help::
-h::
	Shows a short summary of all options and exits.

--json::
-j::
	Prints one JSON object per line for other programs to read instead of showing
	the interactive terminal interface. See JSON OUTPUT below.

--theme=MODE::
	Chooses colours that can be read on a terminal with a dark or a light background.
	MODE is one of `auto` (this is the default), `dark` and `light`. With `auto`,
	`fireperf` asks the terminal what its background looks like, or looks at
	`COLORFGBG` if that has been set, and uses dark colours if neither tells.

--no-tui::
-n::
	Prints plain text reports instead of showing the interactive terminal interface.
	See OUTPUT below.

--zerocopy=MODE::
	Most of what it costs to send data is copying it from the program into the kernel,
	which `fireperf` avoids. How depends on the backend:
	+
	* `epoll` keeps what it sends in a file in memory and asks the kernel to send it
	  from there (`sendfile(2)`). This always works.
	* `io_uring` asks the kernel to send straight from the memory of `fireperf`. That
	  pays off with a network card that can take the data as it is. Over loopback and
	  some virtual devices the kernel has to copy it after all, which is more work
	  than copying it in the first place.
	+
	MODE is one of:
	+
	* `auto`: Send without copying, but go back to copying if the kernel says that it
	  had to copy most of the data anyway (this is the default)
	* `always`: Never go back to copying
	* `never`: Copy what is being sent like most other programs do, which is useful to
	  see what difference it makes
	+
	The memory that `io_uring` sends from has to be locked while the kernel is busy
	with it. Unless `fireperf` runs as root, that counts towards how much memory a
	user may lock (`ulimit -l`), which is only a couple of MiB by default. `fireperf`
	then sends from as much of its data as it may lock, and copies if it runs out.
	Raising the limit (for example with `LimitMEMLOCK=` in systemd or in
	`/etc/security/limits.conf`) allows to send from more data, but makes no
	difference to how fast it is.

--parallel=N::
-P N::
	If this option is set on the client side, it will instruct the client to open N
	connections and keep them open (the default is one).
	+
	On the server side, it will set a limit how many connections the server is willing
	to accept. N can be up to 1048576.

--port=N::
-p N::
--port=N:M::
-p N:M::
	Both server and client take a single port (N) or a port range (N:M).
	+
	The server will open a listening socket the port given or each port in the range
	and waits for the client to connect.
	+
	The client will spread its connections evenly across all ports in the range or use
	the specified port to connect to the server.
	+
	A single port is all that is needed for almost everything: all workers of the server
	listen on the same port and the kernel spreads the connections across them. A range
	is only there for one thing: a client cannot have more connections open to the same
	address and port of a server than it has local ports available (about 28,000 by
	default, see `net.ipv4.ip_local_port_range`), and a port range multiplies that.
	More source addresses on the client (`--bind`) do the same and leave you with a
	single port to allow on a firewall.

--rate=N::
-r N::
	Makes the client open N connections per second (up to ten million).
	+
	Without this option, the client opens a new connection whenever one has been closed,
	as fast as it can. If whatever it is talking to slows down, the client slows down
	with it, and the result only tells which one of them was the slowest.
	+
	With this option, every connection is opened when it is due, no matter what has
	happened to the ones before. If the other side cannot keep up, connections take
	longer to be established, pile up and fail, all of which is shown in the output.
	To find out how many connections per second something can handle, increase the
	rate until that happens.
	+
	`--parallel` limits how many connections may be open at the same time. It is not
	limited by default if a rate is given. Connections that are due while that limit
	has been reached are skipped. With `--connections`, the client makes exactly that
	many attempts, whether they succeed or not.
	+
	At the end, the client shows how well it has been able to follow the schedule:
	+
	----
	Schedule:    10000 due, 9990 on time, 10 late, 0 skipped
	----
	+
	A connection is _late_ if it was opened more than ten milliseconds after it was
	due, and _skipped_ if the client was more than a second behind or was not allowed
	to open another connection. If more than one percent were not on time, the client
	itself was the limit and the result does not say much about anything else.

--timeout=T::
-t T::
	If set, `fireperf` will automatically terminate itself after T seconds.
	Otherwise it will run for forever.

--version::
-V::
	Prints the version of this program

--workers=N::
-w N::
	Sets the number of worker threads. By default, `fireperf` launches one worker for
	every processor, but never more than the client has connections to open.
	+
	Every worker handles its own share of the connections independently of all others.
	The server opens one listening socket per worker for every port and the kernel
	distributes incoming connections across them.

--close::
-x::
	If set, all connections will be closed as soon as they have been established.
	No data will be sent.
	+
	This is useful to test how many TCP handshakes per second are possible. The client
	will open a new connection for every one that has been closed, so that it is
	enough to pass this option to either the server or the client.
	+
	NOTE: This opens tens of thousands of connections per second. Every stateful
	firewall on the way (including connection tracking on the client and the server
	themselves) has to keep an entry for each of them for a while after it has been
	closed, and will start to drop packets of any other connection once its table
	is full. Use `--connections` or `--timeout` to limit the test.

--zero::
-z::
	Sends zeroes instead of random data, in both directions.
	+
	This is useful for testing compression between the client and server. On the
	server, it applies to all clients, whether they have asked for it or not.


--keepalive-interval=T::
	How often connections that are being kept open (`--keepalive`) are checked, in
	seconds (1 to 32767). The default is 10. See `--keepalive`.

--keepalive-count=N::
	How many TCP keepalives may go unanswered before the kernel gives up on a
	connection (1 to 127). The default is 3.

== SERVER-SPECIFIC OPTIONS

--server::
-s::
	This starts the program in server mode which makes it listen on the specified
	port and wait for clients to connect.
	+
	Without `-4` or `-6`, the server listens on IPv6 and IPv4. If the system does not
	have IPv6, it listens on IPv4 only.

--keepalive::
-k::
	Instead of sending data, this will configure fireperf to only send keepalive
	packets.
	+
	If a large number of connections is being created, it might become undesirable to
	saturate the link between client and server. This option will send some packets to
	keep the connection alive through for example NAT gateways.
	+
	This is an option of the client like any other: it asks the server not to send
	any data on its connections. Given to the server, it makes the server never send
	any data, whatever its clients ask for.
	+
	A connection that nothing is sent over does not tell if it still works: if a
	firewall in between has forgotten about it, both ends still have it open. So
	the client sends one byte over every connection once per interval (see
	`--keepalive-interval`), which the server sends back. A connection is _lost_ if
	that has not come back by the time the next one is due. It is then closed and
	the client opens a new one. The _Alive_ column shows how many of the open
	connections have last been found to work.
	+
	This only works if the server is `fireperf`. The kernel is being asked to send TCP
	keepalives as well.


== CLIENT-SPECIFIC OPTIONS

--client HOST::
-c HOST::
	This starts the program in client mode. In this mode, the client will connect to
	the server at HOST and start a throughput test.
	+
	HOST can be an IPv4 or IPv6 address, or a name. If a name has more than one
	address, the client connects to the first one that the system prefers, unless
	`-4`, `-6` or `--bind` tell otherwise. If that one does not work, it tries the
	next one, until it has reached the server. After that, all connections go to the
	address that has worked.

--bind=ADDRESS::
-B ADDRESS::
	Makes the client open its connections from ADDRESS, which must be an address of
	this host. This option may be given more than once and the connections are then
	spread across all of the addresses.
	+
	All addresses must be of the same family (IPv4 or IPv6), and the client connects
	to an address of the server of that family.
	+
	Every address can open as many connections to the same address and port of the
	server as there are local ports (see `--port`). More addresses allow more
	connections, and what is in between sees connections from more than one client.
	+
	The client then shows what has happened for every one of these addresses instead
	of for the server (see OUTPUT), including how many connections could not be
	established from it - which shows if something in between treats one of them
	differently.

--connections=N::
-C N::
	Limits the number of connections that the client opens in total. The client will
	terminate as soon as it has opened N connections and all of them have been closed.
	+
	Together with `--close`, this makes the client perform exactly N handshakes.

--connect-timeout=T::
-T T::
	The client gives up on a connection if it could not be established after T seconds
	and counts it as failed (timed out). The default is 5 seconds. 0 leaves it to the
	kernel, which keeps trying for about two minutes.
	+
	A SYN that is lost on the way is being sent again after one second, then after
	another two, and so on. A timeout of one second therefore counts every connection
	that has lost a single packet as failed, and with the default the first two
	attempts may be lost.

--exchange=N[:M]::
-X N[:M]::
	The client sends a request of N bytes on every connection, waits for the server
	to answer with a response of M bytes (or N, if only one number has been given),
	and closes the connection. Like with `--close`, it then opens a new one.
	+
	A connection only counts if all of the response has arrived. If it is closed
	before that, it is counted as failed (_incomplete_).
	+
	This is what a firewall gets to see from a short transaction like a DNS query over
	TCP or a small HTTP request, and it is a better test than `--close`: there, a
	connection is established and closed without a single byte going through, which
	is the easiest case there is. Both sizes can be zero, which still exchanges the
	two messages in which the client and the server tell each other what they want.
	Neither can be larger than one megabyte.
	+
	At the end, the client shows how many exchanges have been completed and how long
	they took, from asking for the connection until the last byte of the response:
	+
	----
	Exchanges:   10000 completed, min 71us, median 119us, 99% 1.24ms, max 1.03s
	----
	+
	The server does not need this option, but it has to be `fireperf`.

--reset::
-R::
	Makes the client reset connections (RST) instead of closing them properly (FIN)
	when it is done with them, which is with `--close` and `--exchange`.
	+
	Something that keeps state for a connection may forget about it straight away when
	it has been reset, but has to remember it for a while after it has been closed.
	The server counts these connections as _Reset_.

--duplex::
-D::
	Duplex mode enables the client to send data instead of only receiving it from the
	server. The same TCP connection will be used for traffic in both directions.


== OUTPUT

If `fireperf` is started on a terminal, it shows an interactive interface:

* At the top, what is running, and for how long.
* A graph of the traffic, with the latest second on the right hand side. What has
  been received grows upwards and what has been sent grows downwards. If only one
  direction carries any data, it gets all of the space.
* On terminals that are at least 100 columns wide, the current, average and peak
  rates and the totals in both directions, the connections, how long it takes to
  establish them, and how much of a CPU the busiest worker is using.
* Below that, every address that we are talking to: the clients of a server, or
  the server of a client, with what it is, its connections, and its traffic. The
  busiest one comes first, and those without any open connections are grey.
* The latest messages.

Press `q` or Ctrl-C to quit, `p` or the space bar to pause and resume, and Ctrl-L
to redraw the screen. If there are more addresses than fit, the arrow keys (or `j`
and `k`), Page Up, Page Down, Home and End scroll through them. `s` sorts them by
the next column, and `n` shows their names instead of the addresses (which are being
looked up in the background, so it might take a moment until they show up). With a UTF-8 locale, lines and the graph are drawn with what
UTF-8 has for them.

To find out who has connected, the server asks the kernel for the address of
every connection that it accepts. That costs a system call, which does not make a
difference that can be measured, even with hundreds of thousands of connections per
second.

If standard input or standard output is not a terminal (for example when the output
is redirected into a file or `fireperf` has been launched by a script), or if
`--no-tui` has been passed, one line is printed every second instead:

----
     Time     Open    Alive      Total      New/s   Failed    Reset    Median       99%   CPU        RX Rate     RX Total        TX Rate     TX Total
     1.0s        1        0          1       1.00        0        0      85us      85us   52%      9.41 Gb/s     1.10 GiB       0.00 b/s       0.00 B
----

The columns show:

* the time since the start,
* the number of connections that are currently open,
* how many of those are known to work (see `--keepalive`),
* the number of connections that have been opened in total and per second,
* the number of connections that could not be established (Failed),
* the number of connections that were established, but then failed instead of being
  closed properly (Reset). This is what happens if a firewall in between loses its
  state of a connection - or if the other side terminates while data is on its way,
* how long it took the client to establish the connections that have been opened
  since the previous line: the median, and the time that 99% of them stayed below,
* how much of a CPU the busiest worker has been using,
* the current rate and the total amount of data that has been received (RX) and
  sent (TX).

When `fireperf` terminates, a final line shows the totals and the average rates over
the whole runtime, followed by a summary. The client shows what has happened to the
connections it has tried to open:

----
Workers:     16 using io_uring, the busiest one was using 23% of a CPU
Transferred: 0 bytes sent, 1181116006 bytes received
Connections: 1000 attempted, 990 established, 10 failed (8 timed out, 2 rejected, 0 local limit, 0 incomplete), 0 reset, 0 lost
Handshake:   min 45us, median 120us, 99% 1.03s, max 3.02s
----

The server shows its workers, and how many connections the kernel has dropped because
they have not been accepted in time. That is counted for the whole host and includes
all other programs.

Both show everybody that they have been talking to - the clients of a server, or the
server of a client - with what it is, how many connections it has had, how many of
them have been reset, and how much has been received from it and sent to it, the one
that has transferred the most first. The client also shows how many connections could
not be established, and with `--bind`, it shows the addresses that it has connected
from instead of the server:

----
Clients:     2
             Address    Type           Total   Reset      Received          Sent
             127.0.0.1  fireperf           2       0        0.00 B     10.51 GiB
             ::1        fireperf           1       0      2.92 GiB      3.07 GiB
----

Every worker keeps up to 64 addresses apart and adds up everything else, which is
shown as _Others_.

A connection has _timed out_ if nobody has answered, it was _rejected_ if somebody has
answered that it cannot be established (refused, reset, unreachable), and _local limit_
means that the client itself ran out of something (ports, file descriptors, memory)
before anything was sent. _incomplete_ is a connection that has been closed before the
response to a request has arrived (see `--exchange`), and a connection is _lost_ if it
was open, but the server has stopped answering on it (see `--keepalive`).

The time of a handshake is measured from the moment the client asks for the
connection until it is established, and is accurate to about six percent. If a packet
of the handshake is lost, it is being sent again after one second, which shows up as a
99th percentile or a maximum of slightly more than that.

== JSON OUTPUT

With `--json`, `fireperf` prints one JSON object per line (JSON Lines) to standard
output, and all messages go to standard error. Every object has a `type`:

`start`::
	is printed first and describes the test: `version` (of the format, currently 1),
	`fireperf` (the version of the program), `mode` (`client` or `server`), `backend`,
	`workers`, `port` and `ports`. The client adds the server as it has been given
	(`host`) and the `address` that it connects to, and the options of the test:
	`parallel`, `rate`, `connections`, `timeout`, `duplex`, `keepalive`, `close`,
	`zero` and `exchange` (with the `request` and `response` sizes).

`interval`::
	is printed every second.

`summary`::
	is printed last and covers the whole runtime.

Intervals and the summary have the same fields:

* `time` is the time since the start and `interval` the time this object covers, both
  in seconds,
* `connections` has the number of connections that are `open` and `alive`, how many
  are `new` in this interval, the `total`, `new_per_s`, and how many have been
  `attempted`, have `failed`, have been `resets` or `lost`,
* `received` and `sent` have the `bytes` of this interval, the `total_bytes` and the
  rate in `bits_per_s`,
* `handshake` has the `count`, `min_us`, `median_us`, `p99_us` and `max_us` of the
  handshakes of this interval in microseconds, or is `null` if there were none,
* `cpu` is how much of a CPU the busiest worker has been using (1.0 is one CPU),
* `remotes` has everybody that we have been talking to so far, the one that has
  transferred the most first: its `address` (or `null` for everything that did not
  fit in), what it is (`type`: `fireperf`, `iperf2`, `iperf3`, `unknown`, or `null` if
  we don't know, yet), its `connections` (`open`, `total`, `failed` and `resets`),
  and the `received_bytes` and `sent_bytes` in total. With `--bind`, these are the
  addresses that the client has connected from, which the `start` object lists as
  `bind`.

The summary of the client also has the kind of `server` it has been talking to, why
connections have `failed` (`timeout`, `rejected`, `local` and `incomplete`), the
number of `exchanges` and their latency in `exchange` (see `--exchange`), and whether
it has kept its `schedule` (`due`, `late` and `skipped`, see `--rate`). The summary of
the server has the `listen_drops`.

----
{"type":"interval","time":1.000000,"interval":1.000000,"connections":{"open":1,...},"received":{"bytes":5165447638,...},...}
----

New fields may be added at any time, so programs should ignore what they do not know.
Fields will not be removed or change their meaning without changing `version`.
Numbers are always written the same way, no matter what the language is set to.

== INTEROPERABILITY

The first thing that a client sends on a connection is a short message that says what
it wants, which the server answers with what it is going to do. Both sides also use it
to find out what they are talking to, so that nothing has to be configured for this.
The client opens a single connection to find out before it opens all others.

*fireperf*::
	The client and the server do what the client has asked for.

*iperf (version 2)*::
	`iperf` only ever sends data from its client to its server.
	+
	A `fireperf` server receives what an `iperf` client sends.
	+
	A `fireperf` client that finds an `iperf` server sends data to it (as if `--duplex`
	had been given), says so, and does not expect to receive anything. Recent versions
	of `iperf` identify themselves straight away. Older ones do not answer at all, which
	the client takes for an answer after one second.

*iperf3*::
	A `fireperf` server serves `iperf3` clients that run a test over TCP: by default
	(the client sends), with `--reverse`, with `--bidir` and with `--parallel`. It
	can serve many of them at the same time, and `fireperf` clients next to them.
	The results that `iperf3` shows for the server are what `fireperf` has counted.
	It does not know anything about retransmits or the usage of the CPU.
	+
	A `fireperf` client that finds an `iperf3` server runs a test with it: the server
	sends (which is what `iperf3` calls reverse), and with `--duplex` the client does,
	too. `--parallel` sets the number of connections, which is doubled with `--duplex`
	as `iperf3` only sends in one direction on each of them. The test runs until
	`fireperf` terminates.
	+
	An `iperf3` test has an extra connection that controls it, which is counted like
	all others. All of its connections are handled by one worker of the client, so the
	_CPU_ column tells whether that was enough. An `iperf3` server only talks to one
	client at a time, and `--keepalive`, `--close`, `--rate` and `--connections` have
	no meaning with it.

Connections that are closed as soon as they have been established (`--close`) never
carry any of this. Connections that exchange a request and a response (`--exchange`)
and the checks of connections that are being kept open (`--keepalive`) need `fireperf`
on both sides.

== TESTING WHAT IS IN BETWEEN

`fireperf` is made to test something that sits between two hosts, like a firewall or a
NAT gateway. The client and the server must run on hosts on either side of it and not
on the device itself. If they are running on the same host, or are connected to each
other directly, the numbers only show how fast that host, its network stack and
`fireperf` are.

That is still worth doing once: connect the client and the server back to back and
run the test that you are planning to run. The result is the most this pair of hosts
can generate. Anything close to it that you measure later with the device in between
says nothing about the device.

=== Is it the device or is it me?

The client and the server run into limits of their own long before a decent firewall
does. `fireperf` prints a warning at start for the ones it knows about, and the
output shows a few things that tell you who was the limit:

* *CPU*: If a worker uses almost all of a CPU, this side cannot go any faster. Use
  more workers or a faster host. Time that the kernel spends on packets outside of
  `fireperf` is not included, so a low number does not prove that the host has
  capacity to spare.
* *local limit* in the summary of the connections: The client ran out of local ports
  or file descriptors before it has sent anything.
* *Schedule* (with `--rate`): If connections were late or have been skipped, the
  client could not open them at the rate that it has been asked to.
* *Accept* in the summary of the server: The server did not accept connections fast
  enough.

These settings of the hosts matter:

* A client can only open as many connections to one address and port of the server
  as it has local ports (`net.ipv4.ip_local_port_range`, about 28,000 by default).
  Use a port range (`--port`) on both sides and more source addresses (`--bind`).
  A connection that the client closes keeps its port for another minute, so let the
  server close the connections if you are opening many.
* Every connection needs a file descriptor (`ulimit -n`, `fs.nr_open`).
* If the client or the server are tracking connections themselves (if they are
  running a firewall with connection tracking, which most distributions do), that
  has a limit of its own (`net.netfilter.nf_conntrack_max`), slows them down, and
  will be full long before the table of a dedicated firewall is. Turn it off on both
  hosts or exclude the test traffic from it.
* The server can only have so many connections waiting to be accepted
  (`net.core.somaxconn`, `net.ipv4.tcp_max_syn_backlog`).

=== How much data can it forward?

----
server# fireperf --server
client# fireperf --client SERVER --parallel 16 --duplex
----

The server sends data to the client and, with `--duplex`, the client sends data to
the server, as fast as they can. One connection is usually handled by one CPU on every
device on the way, so use as many connections as you want to see CPUs being used.
Random data is being sent by default; `--zero` shows the effect of compression.

=== How many connections can it keep open?

----
server# fireperf --server --keepalive --port 5001:5064
client# fireperf --client SERVER --keepalive --port 5001:5064 --parallel 1000000
----

The client opens connections until it has as many as it has been asked for and no
data is being sent. Watch _Open_ go up and where it stops, and whether connections
start to fail (_Failed_) or are being lost again (_Reset_).

The client checks every connection once per interval (ten seconds by default, see
`--keepalive-interval`) by sending a byte that the server sends back. _Alive_ shows how
many connections have passed that check, and a connection that does not is counted as
lost, closed and opened again. If the device silently forgets about connections, _Alive_
falls behind _Open_ and the number of lost connections goes up. Keep the test running
for a while after the number of open connections has stopped changing: a firewall
might only drop what it has not seen any traffic on for some time.

=== How many connections per second can it take?

----
server# fireperf --server
client# fireperf --client SERVER --exchange 100:1000 --rate 10000 --timeout 30
----

The client opens connections at a fixed rate, sends a small request on each of them,
waits for the response and closes it. Run this a couple of times and increase the
rate. The device is at its limit when:

* connections fail (_Failed_), or
* it takes longer to establish them (_Median_ and _99%_). A step to about one second
  means that packets are being dropped and sent again.

Do not use the highest rate at which nothing failed as the result without checking
that the client was able to open connections at that rate (_Schedule_ and _CPU_).

Without `--rate`, the client opens a new connection whenever one has been closed. That
is easier, but it only finds whatever is slowest and slows down with it: you get a
number, but you cannot tell whose it is.

Everything on the way that keeps state has to remember each of these connections for a
while after it has been closed - on a Linux firewall for up to two minutes. A test
with 10,000 connections per second needs space for more than a million of them, and
when a table is full, connections of everybody else are being dropped, too. Do not
run this through anything that others depend on.

== LANGUAGE

`fireperf` speaks the language of the user, as far as it has been translated (see
`LANG` and `LC_ALL` in locale(7)). That includes the plain text output, in which numbers
are written the way the language writes them, too. Scripts that read the output should
run `fireperf` with `LC_ALL=C`.

== SIGNALS

`fireperf` terminates cleanly when it receives SIGINT or SIGTERM.

== EXIT STATUS

*0*::
	The test has finished: the timeout was reached, the user has quit, or the
	process has been asked to terminate by a signal.

*1*::
	An error has occurred.

*2*::
	The command line was invalid.

== BUGS
Please report all bugs to the bugtracker at https://bugzilla.ipfire.org/

== SEE ALSO
`fireperf-design`(7), which explains how `fireperf` works and how it compares to
`iperf3`(1) and `iperf`(1).

== AUTHORS
Michael Tremer
