Benchmarking Performance

We provide a benchmarking script to enable users to easily analyze the performance of CARLA in their own environment. The script can be configured to run a number of scenarios that combine different maps, sensors and weather conditions. It reports the average and standard deviation of FPS under the requested scenarios.

In this section we detail the requirements to run the benchmark, where to find the script, the flags available to customize the scenarios that are run and examples on how to run the commands.

We have also included our results of a separate benchmark which measures performance in CARLA in a specific environment when using different combinations of number of vehicles, enabling physics and/or enabling Traffic Manager. The results are presented alongside the CARLA version used and the environment the test was performed in.

The benchmark script

The benchmark script can be found in PythonAPI/util. It has several flags available to customize the scenarios to be tested which are detailed in the synopsis below.

Before you begin

The benchmarking script requires some dependencies to be installed before you can run it:

python -m pip install -U py-cpuinfo==5.0.0 psutil python-tr gpuinfo GPUtil

Synopsis

python3 performance_benchmark.py [--host HOST] [--port PORT] [--file FILE] [--tm] [--ticks TICKS] [--sync] [--async]) [--fixed_dt FIXED_DT] [--render_mode]) [--no_render_mode] [--show_scenarios]) [--sensors SENSORS [SENSORS ...]]) [--maps MAPS [MAPS ...]]) [--weather WEATHER [WEATHER ...]]

Flags

  • --host: IP_ADDRESS Default: Localhost.
    Configures the host of the server.

  • --port: PORT
    Default: 2000
    Configures the TCP port to listen to.

  • --file: filename.md Default: benchmark.md
    Writes results in markdown table format to a file.

  • --tm Switch to Traffic Manager benchmark

  • --ticks Default: 100
    Sets the number of ticks to use for each scenario.

  • --sync Default mode.
    Runs benchmark in synchronous mode.

  • --async Runs benchmark in asynchronous mode.

  • --fixed_dt Default: 0.05
    For use with synchronous mode if you would like to set the delta timestep.

  • --render_mode Runs benchmark in rendering mode.

  • --no_render_mode Default mode.
    Runs benchmark in non-rendering mode.

  • --show_scenarios When the script is run with only this flag you will see a list of all the scenario parameters available. When combined with other flags you will see a preview of the scenarios that will be run without actually executing them.

  • --sensors: INTEGER

    Default: All

Sensors to be used in the benchmark. Chose between LIDAR and RGB camera:

  • 0: cam-300x200
  • 1: cam-800x600
  • 2: cam-1900x1080
  • 3: cam-300x200 cam-300x200 (two cameras)
  • 4: LIDAR: 100k
  • 5: LIDAR: 500k
  • 6: LIDAR: 1M

  • --maps: TownName Default: All maps
    All CARLA maps, both layered and sub-layered, are available.

  • --weather: INTEGER Default: All weather conditions Change the weather conditions:

  • 0: ClearNoon
  • 1: CloudyNoon
  • 2: SoftRainSunset

How to run the benchmark

  1. Start CARLA:

    # Linux:
./CarlaUnreal.sh
# Windows:
CarlaUnreal.exe
# Source:
cmake --build Build --target launch

  2. In a separate terminal navigate to PythonAPI/util to find the performance_benchmark.py script:

  • Show all possible scenarios without running them:
python3 performance_benchmark.py --show_scenarios
  • Show what scenarios will run when configurations are applied without actually executing them:
python3 performance_benchmark.py --sensors 2 5 --maps Town03 Town05 --weather 0 1 --show_scenarios`
  • Execute the performance benchmark for those scenarios:
python3 performance_benchmark.py --sensors 2 5 --maps Town03 Town05 --weather 0 1
  • Perform the benchmark for asynchronous mode and rendering mode:
python3 performance_benchmark.py --async --render_mode

CARLA forum