Benchmarks
==========

This directory keeps what we have measured, so that we can tell what a change
has done to the performance of fireperf, and how it compares to iperf3 and iperf.

  results.csv   Everything that tools/benchmark has measured, one line per test


Running
-------

Build fireperf and run this in the top directory of the source tree:

  tools/benchmark --output benchmarks/results.csv --note "what this run is about"

iperf3 and iperf are being measured as well if they are installed. Set IPERF3 and
IPERF if they are not in the PATH. A run takes about five minutes.

To measure over a real network, let another host send to this one:

  tools/benchmark --remote ms@people01.haj.ipfire.org --output benchmarks/results.csv --note "..."

That host is reached over SSH. fireperf is copied to /tmp/fireperf-benchmark
there (see --remote-directory, home directories are often mounted noexec), and
iperf3 and iperf are being measured if they are installed on both hosts.

Commit results.csv together with the change that has been measured (or right
after it), and run the benchmark before and after a change that is meant to make
something faster on the same machine. Numbers from different machines cannot
be compared with each other.


What is being measured
----------------------

Everything runs over loopback on one host, unless --remote is given.

Throughput (1 and 8 connections)

  One side sends as fast as it can for a couple of seconds. We record how much
  has arrived and how much CPU time (user and system) the sending and the
  receiving process have used for it. The latter is given in seconds per GiB,
  which does not depend on how long the test ran or how fast it was: less is
  better.

  The sender is the server for fireperf and for iperf3 (which is being run in
  reverse mode for that), and the client for iperf.

Throughput from another host (--remote)

  The same, but the other host sends to this one over the network, so that
  what it costs to send is not mixed up with what it costs to receive on the
  same CPUs. The sender is the fireperf server, and the iperf3 and iperf client
  on the other host. The receiver here is always the same (fireperf with epoll),
  as only the sender is being compared. io_uring is measured with
  --zerocopy=always, too, as that is what matters over a real network card.

  The CPU time of the sender is what bash's "times" says on the other host. The
  "remote" columns say which host that was. Only the throughput tests are run.

Connections

  The client opens a number of connections, 16 at a time, and either closes
  them straight away ("handshake") or sends 100 bytes, waits for 1000 bytes and
  closes them ("exchange"). We record how many it gets done per second and how
  much CPU time the client and the server have used per connection (in
  microseconds, which includes what it takes to start them). Only fireperf can
  do this.

  Every connection stays in the connection tracking table of a host for a
  couple of minutes after it has been closed, and opening enough of them to
  measure anything would fill up that table. So these tests run in a network
  namespace of their own, which has its own table and does not track connections
  to begin with. That needs root. Without it, only 10,000 connections are being
  opened per test, which takes a fraction of a second and does not say much.

  The first results in results.csv were taken that way (the "isolated" column
  is empty) and cannot be compared with the later ones.


What this does not tell
-----------------------

 * Loopback (without --remote) is not a network card: there are no checksums to compute, no
   interrupts, no segmentation and nothing that is ever lost. Most of the time
   is spent copying memory, so that this mainly shows how good we are at
   avoiding that. In particular, sending without copying with io_uring only
   pays off with a real network card, and falls back to copying over loopback.

 * The client and the server compete for the same CPUs.

 * CPU time that the kernel spends on packets outside of the two processes is
   not included.

 * Results vary by a couple of percent between runs, and by a lot more if the
   machine is busy with something else. The throughput tests are run three
   times and the median is what is being recorded.
