= fireperf-design(7)

== NAME
fireperf-design - How fireperf works, and why it works that way

== DESCRIPTION
`fireperf`(8) is a tool to measure what a network - and above all, what is in
between two hosts - can do: how much data it can move, how many connections it can
open, how long that takes, and whether it keeps the connections that it has. It has
been written to test firewalls, which is what IPFire is.

That is a different question from what most tools for this have been built for.
They measure a link: they open a connection or a couple of them, send as much as
they can, and report a number. A firewall is not a link. It keeps state for every
connection, it has a table of them that can fill up, it may forget about a
connection that has been quiet for a while, and the number of new connections per
second matters to it at least as much as the number of bytes. A tool that tests a
firewall has to be able to push the firewall harder than the firewall can take,
which means that the tool itself must never be what is holding the test back. And
it has to tell when it is.

This page explains how `fireperf` is built to do that.

== ONE WORKER PER CPU, AND NOTHING SHARED
`fireperf` starts one worker thread for every CPU (see `--workers`). Every worker
runs its own event loop and has its own connections. Workers don't share any locks,
lists or buffers on the way of the data, so that adding a CPU adds a worker that can
do as much as the others.

The server opens one listening socket per worker on the same port (with
`SO_REUSEPORT`), and the kernel spreads new connections across them. That means that
all clients connect to a single port, which keeps firewall rules simple, and that no
worker has to hand connections to another one. The client spreads the connections
that it should open across its workers in the same way.

What the workers count (bytes, connections, failures, how long it took to establish
them) is kept in counters that only they write to. Once a second, the main thread
adds them up. It is the only thread that talks to the terminal, handles signals
(through a `signalfd`) and keeps time, so that the workers never wait for any of
that.

== TWO WAYS TO DO I/O: IO_URING AND EPOLL
Every worker does its I/O through a backend (see `--backend`):

`io_uring`::
	The kernel is being asked to do things (accept, receive, send, connect,
	close) and tells us when they are done, without a system call for every one of
	them. `fireperf` accepts all connections of a listening socket with a single
	request, and receives on a connection with a single request for as long as it is
	open, from a ring of buffers that the kernel picks from. It also opens and closes
	connections through the ring. This is what is being used if it is available.

`epoll`::
	The classic way of being told when a socket is ready and then doing what has to
	be done with a system call. It works everywhere, which `io_uring` does not: it is
	often disabled in containers and on older kernels.

Both backends do the same things. Everything else in `fireperf` does not know which
one is being used.

== SENDING WITHOUT COPYING
Most of what it costs to send data is copying it from the program into the kernel.
`fireperf` doesn't copy:

* What it sends comes from a pool of 32 MiB of random data (or zeroes, see
  `--zero`), so that nothing in between can compress it or recognise it. Random data
  that does not fit into a cache is also what a real network carries, unlike the
  same couple of kilobytes being sent again and again.

* The pool is a file in memory (a `memfd`). With `epoll`, the kernel is being asked
  to send from that file (`sendfile`(2)), which never copies.

* With `io_uring`, the pool is registered with the kernel once, and the kernel sends
  straight from it (`IORING_OP_SEND_ZC`) and tells us when it is done with it. The
  kernel also tells us when it had to copy after all - which is what happens over
  loopback and with some virtual devices - and then `fireperf` goes back to copying,
  which is less work for the same result (see `--zerocopy`).

Registering memory counts towards how much memory a user may lock, which is only a
couple of MiB by default. `fireperf` registers as much of the pool as it may, and if
that is not enough, it copies rather than failing.

Received data is thrown away by the kernel without being copied out (`MSG_TRUNC`),
as there is nothing that `fireperf` would do with it.

== OPENING CONNECTIONS
Testing how many connections a firewall can handle needs a client that can open
many more than that. `fireperf` keeps a number of connections open (`--parallel`),
opens them at a fixed rate (`--rate`), closes them straight away (`--close`), resets
them (`--reset`), or exchanges a request and a response on every one of them, as a
web server would (`--exchange`).

With `--rate`, connections are being opened on a schedule, whether the ones before
have been established or not. A client that waits for every connection before it
opens the next one measures how fast the slowest part is, not whether everything
keeps up with the rate. `fireperf` reports how many connections it has opened late,
or not at all.

Every connection that could not be established is counted, and why: it has timed
out (see `--connect-timeout`), it has been rejected, or the client itself has run
out of something (ports, file descriptors). Established connections that end
without being closed properly are counted as reset, which is what happens if a
firewall loses its state. With `--keepalive`, the client checks every connection
that it keeps open once in a while, and counts those that don't answer any more as
lost.

How long it takes to establish a connection is recorded for every connection in a
histogram that is accurate to about six percent, so that the median, the 99th
percentile and the maximum can be shown for any number of connections without
keeping all of them.

Before the test, `fireperf` checks whether the system can do what it has been asked
to do (file descriptors, local ports, the connection tracking table, the listen
backlog) and says so if it cannot. With `--bind`, connections are opened from more
than one address, so that a client is not limited by the number of its local ports.

== KNOWING WHAT HAS BEEN MEASURED
A result is only worth something if it is clear what has limited it. `fireperf`
measures how much of a CPU every worker uses. If one has been busy all of the time,
the result shows what `fireperf` could do on this side, not what the network or the
firewall can do, and `fireperf` says so.

On the server, `fireperf` counts how many connections the kernel has dropped
because they could not be accepted in time.

== ONE PROTOCOL, AND THE OTHERS, TOO
When a client connects, it sends 37 bytes that tell the server what kind of test it
wants. The server answers, and both start. That is all there is to the protocol.
The server does not need any options for any test, and can run different tests for
different clients at the same time.

Those 37 bytes look like what `iperf3` sends first, and they start in a way that
makes an `iperf` (version 2) server answer. So the client finds out what the server
is from what it answers: `fireperf`, `iperf3` or `iperf`. The server finds out what
a client is from what it sends first. `fireperf` speaks the protocol of `iperf3`
(including its JSON messages), and it understands what `iperf` sends. So a
`fireperf` client can test against an `iperf3` server, and an `iperf3` client can
test against a `fireperf` server, without anybody having to say so.

== SHOWING WHAT IS GOING ON
On a terminal, `fireperf` shows a graph of the traffic, its numbers, and every
address that it is talking to (see `fireperf`(8)). Without a terminal, it prints a
line every second, and with `--json`, it prints a JSON object every second for other
programs to read.

== COMPARED TO IPERF3 AND IPERF
`iperf3` and `iperf` (version 2) are the tools that most people use to measure a
network, and they are good at measuring a link. Where `fireperf` is different:

* An `iperf3` server runs one test at a time and turns everybody else away until it
  is done. A `fireperf` server takes everybody, and every client can run a test of
  its own.

* `iperf3` and `iperf` measure how much data a couple of connections move. Neither
  can open connections at a fixed rate, tell why connections could not be
  established, find out whether connections that are being kept open still work,
  or show how long it takes to establish connections across many of them.

* By default, `iperf3` and `iperf` copy what they send. Sending from another host
  over a network, `fireperf` needs less than half of the CPU time per GiB that
  `iperf3` needs with `--zerocopy`, and about a seventh of what `iperf3` and `iperf`
  need by default (see `benchmarks/results.csv` in the source).

* `fireperf` can be the client or the server for both of them, so it can be used
  where one of them is already running.

== SEE ALSO
`fireperf`(8), `iperf3`(1), `iperf`(1), `io_uring`(7), `epoll`(7), `sendfile`(2)

== AUTHORS
Michael Tremer
