Nova service concurrency
========================

For a long time nova services relied almost exclusively on the Eventlet library
for processing multiple API requests, RPC requests and other tasks that needed
concurrency. Since Eventlet is not expected to support the next major cPython
version the OpenStack TC set a `goal`__ to replace Eventlet and therefore Nova
has started transitioning its concurrency model to native threads. During this
transition Nova maintains the Eventlet based concurrency mode while building
up support for the native threading mode.

.. __: https://governance.openstack.org/tc/goals/selected/remove-eventlet.html

.. note::

   Native threading mode is now the default and considered production-ready for
   all services except the console proxy services (nova-novncproxy,
   nova-serialproxy, nova-spicehtml5proxy). For the console proxy services,
   native threading mode is available but still experimental. Please ensure
   you test the threading mode in pre-production before deploying to a production
   environment. If you do so please let us know how it went on the mailing list
   openstack-discuss@lists.openstack.org.

.. warning::

   Eventlet based concurrency mode is deprecated and will be removed in a
   future release, not earlier than 2027.2. Native threading is now the default
   and recommended mode for all Nova services except the console proxy services
   (nova-novncproxy, nova-serialproxy, nova-spicehtml5proxy) which still
   default to eventlet mode and require explicit configuration to enable native
   threading.

Selecting concurrency mode for a service
----------------------------------------

Since nova 34.0.0 (2026.2 Hibiscus) the nova-scheduler, nova-api,
nova-metadata, nova-conductor, and nova-compute are using native threading by
default. The rest of the services are using eventlet by default in this
release.

The concurrency mode can be configured in two ways: via a configuration option
or an environment variable. When both are set the environment variable takes
precedence.

Using the ``concurrency_backend`` config option
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The :oslo.config:option:`concurrency_backend` option under ``[DEFAULT]``
allows operators to select the concurrency backend in the Nova configuration
file. It accepts the following values:

``auto`` (default)
    Each Nova binary applies its own built-in deployment default. On Nova
    master all services default to native threading except the novnc/serial/
    spice console proxy services (nova-novncproxy, nova-serialproxy,
    nova-spicehtml5proxy) and CLI entry points (nova-manage, nova-policy,
    nova-status) which still default to eventlet.

``threading``
    Start the service with native threading. Eventlet monkey patching is not
    applied. Use this to switch a service to native threading without changing
    the environment.

``eventlet``
    Start the service with eventlet coroutine-based concurrency. Eventlet
    monkey patching is applied. This value is **deprecated** and will be
    removed no earlier than the 2027.2 release.

Example — switch a service to threading in ``/etc/nova/nova.conf``::

    [DEFAULT]
    concurrency_backend = threading

This option is read very early at startup, before oslo.config is fully
initialised, so it takes effect before any other imports occur. The
``OS_NOVA_DISABLE_EVENTLET_PATCHING`` environment variable takes precedence
over this option.

Using the ``OS_NOVA_DISABLE_EVENTLET_PATCHING`` environment variable
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The concurrency mode can also be configured per service via setting the
environment variable ``OS_NOVA_DISABLE_EVENTLET_PATCHING``. Setting that
variable to ``true`` requests the native threading mode while setting it to
``false`` requests the eventlet mode. If the variable is not set the above
default is applied.

CLI commands (nova-manage and nova-status)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The ``nova-manage`` and ``nova-status`` CLI commands also support native
threading mode via the same ``OS_NOVA_DISABLE_EVENTLET_PATCHING`` environment
variable. This is particularly useful for commands that perform cross-cell
operations using scatter-gather patterns (e.g., ``nova-manage cell_v2 list_hosts``,
``nova-status upgrade check``, ``nova-manage placement heal_allocations``) as
these benefit from the concurrent execution provided by native threading.

To run CLI commands in native threading mode::

    OS_NOVA_DISABLE_EVENTLET_PATCHING=true nova-manage cell_v2 list_hosts
    OS_NOVA_DISABLE_EVENTLET_PATCHING=true nova-status upgrade check

.. note::

   Nova is transitioning from ``eventlet`` to native threading across all
   services:

   * **Nova 32.0.0 (2025.2 Flamingo):** Optional native threading support added
     for ``nova-scheduler``, ``nova-metadata``, and ``nova-api``.
   * **Nova 33.0.0 (2026.1 Gazpacho):** Native threading became the **default**
     for the above services. Support was also added for ``nova-conductor`` and
     ``nova-compute``.
   * **Nova 34.0.0 (2026.2 Hibiscus):** Native threading became the **default**
     for ``nova-conductor`` and ``nova-compute``. Support has also been
     extended to ``nova-novncproxy``, ``nova-serialproxy``, and
     ``nova-spicehtml5proxy``, though these services require manual
     configuration to enable it.

   Operators can still manually revert to ``eventlet`` mode if needed,
   however eventlet mode is now deprecated and will be removed in a future
   release.

Tunables for the native threading mode
--------------------------------------
As native threads are more expensive resources than greenthreads Nova provides
a set of configuration options to allow fine tuning the deployment based on
load and resource constraints. The default values are selected to support a
basic, small deployment without consuming substantially more memory resources,
than the legacy Eventlet mode. Increasing the size of the below thread pools
means that the given service will consume more memory but will also allow more
tasks to be executed concurrently.

* :oslo.config:option:`cell_worker_thread_pool_size`: Used to execute tasks
  across all the cells within the deployment.

  E.g. To generate the result of the ``openstack server list`` CLI command, the
  nova-api service will use one native thread for each cell to load the nova
  instances from the related cell database.

  So if the deployment has many cells then the size of this pool probably needs
  to be increased.

  This option is only relevant for nova-api, nova-metadata, nova-scheduler, and
  nova-conductor as these are the services doing cross cell operations.

* :oslo.config:option:`executor_thread_pool_size`: Used to handle incoming RPC
  requests. Services with many more inbound requests will need larger pools.
  For example, a single conductor serves requests from many computes as well
  as the scheduler. A compute node only serves requests from the API for
  lifecycle operations and other computes during migrations.

  This option is only relevant for nova-scheduler, nova-conductor, and
  nova-compute as these are the services acting as RPC servers.

* :oslo.config:option:`default_thread_pool_size`: Used by various concurrent
  tasks in the service that are not categorized into the above pools.

  This option is relevant to every nova service using ``nova.utils.spawn()``.

* :oslo.config:option:`sync_power_state_pool_size`: Used by the
  nova-compute service to sync the power state of each instance on the host
  between the hypervisor and the DB. Since nova 33.0.0 (2026.1 Gazpacho) the
  default value of this option is changed from 1000 to 5 to have a sane default
  in native threading mode. Increasing this value in native threading mode
  increases the nova-compute memory consumption on a host that has many
  instances.

* :oslo.config:option:`max_concurrent_live_migrations`: Used by the
  nova-compute service to limit the number of outgoing concurrent live
  migrations from the host. It is implemented via a thread pool. So increasing
  the the number of concurrent live migrations will increase the nova-compute
  service memory consumption in native threading mode. It is almost always
  a bad idea to use change this config option from its default value, 1. If
  more performant live migration is needed then enable
  :oslo.config:option:`libvirt.live_migration_parallel_connections` instead.

* :oslo.config:option:`max_concurrent_builds` and
  :oslo.config:option:`max_concurrent_snapshots`: In native threading mode
  both types of operations using a common shared executor to free up the RPC
  handler workers. Therefore both type of operations are counted against the
  same shared maximum limit. If the two options are set to different values
  then the shared limit will be the bigger of the two values.

Seeing the usage of the pools
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

When new work is submitted to any of these pools in both concurrency modes
Nova logs the statistics of the pool (work executed, threads available,
work queued, etc).
This can be useful when fine tuning of the pool size is needed.
The parameter :oslo.config:option:`thread_pool_statistic_period` defines how
frequently such logging happens from a specific pool in seconds. A value of
60 seconds means that stats will be logged from a pool maximum once every
60 seconds. The value 0 means that logging happens every time work is submitted
to the pool. The default value is -1 meaning that the stats logging is
disabled.

Preventing hanging threads
~~~~~~~~~~~~~~~~~~~~~~~~~~

Threads from a pool are not cancellable once they are executing a task,
therefore it is important to ensure external dependencies cannot hold up a
task execution indefinitely as that will lead to having fewer threads in the
pool available for incoming work and therefore reduced overall capacity.

Nova's RPC interface already uses proper timeout handling to avoid hanging
threads. But adding timeout handling to the Nova's database interface is
database server and database client library dependent.

For mysql-server the `max_execution_time`__ configuration option can be used
to limit the execution time of a database query on the server side. Similar
options exist for other database servers.

.. __: https://dev.mysql.com/doc/refman/8.4/en/server-system-variables.html#sysvar_max_execution_time

For the pymysql database client a client side timeout can be implemented by
adding the `read_timeout`__ connection parameter to the connection string.

.. __: https://pymysql.readthedocs.io/en/latest/modules/connections.html#module-pymysql.connections

We recommend using both in deployments where Nova services are running in
native threading mode.

Upgrading to Nova 33.0.0 (2026.1 Gazpacho) or newer
---------------------------------------------------

In Nova 33.0.0 (2026.1 Gazpacho), the default concurrency mode of
nova-scheduler, nova-api, and nova-metadata services has been switched to
native threading. We recommend decoupling the upgrade from the concurrency
mode change to reduce the risk of issues. To do that either test and tune the
native threading mode of these services already in 2025.2 (Flamingo), or ensure
that your service configuration is explicitly using the eventlet mode before
you upgrade and only change to threading mode after the upgrade was successful.
In Nova 34.0.0 (2026.2 Hibiscus), the default concurrency mode of
nova-conductor and nova-compute services has also been switched to native
threading. So similar precaution is needed when planning an upgrade. These
services can be tested in native threading mode already in the 33.0.0
(2026.1 Gazpacho) release.
