Troubleshooting Simulations =========================== This guide covers tips and strategies for troubleshooting simulation failures. If you encounter a failure not covered here, please consider contributing it to the `openfe-failure-cases `_ repository or opening an `issue `_. .. contents:: :local: :depth: 1 .. _troubleshooting-periodic-box-size-nonbonded-cutoff-errors: Periodic box size / nonbonded cutoff errors ------------------------------------------- You may see errors like: .. code:: bash openmm.OpenMMException: The periodic box size has decreased to less than twice the nonbonded cutoff. .. code:: bash openmm.OpenMMException: NonbondedForce: The cutoff distance cannot be greater than half the periodic box size. These errors have the same cause: the system is too small for the chosen nonbonded cutoff. **Possible causes** - Insufficient solvent padding. - Incorrect periodic box vectors. - A solvated system that does not leave enough space around the solute. **Suggested fixes** - If the system is solvated by the OpenFE protocol, increase the protocol's ``solvent_padding`` setting. - If the system is explicitly solvated, e.g. when using ``SolvatedPDBComponent`` or ``ProteinMembraneComponent``: - Verify that the correct box vectors were supplied. - Ensure there is sufficient solvent around the system. We do **not** recommend changing the nonbonded cutoff to fix this issue, since force fields are typically parameterized for these values and changing them may affect accuracy. If you have modified the nonbonded cutoff, it may be that it is too large for the simulation box. .. _troubleshooting-smirnoff-force-field-parameter-assignment-errors: SMIRNOFF force field parameter assignment errors ------------------------------------------------ You may see errors like: .. code:: bash openff.interchange.exceptions.UnassignedBondError: BondHandler was not able to find parameters for the following valence terms: - Topology indices (0, 1): names and elements (C1 C), (Si1 Si) This usually means the SMIRNOFF-style force field does not have parameters for part of the chemistry in the system. In the example above, the force field was unable to assign bond parameters for bonds involving silicon. **Possible causes** - The system contains atoms or functional groups that are not covered by the selected force field. **Suggested fixes** - Check whether the chemistry in the system is expected to be supported by the chosen force field. - Consider using a molecule specific force field with custom parameters to guarantee coverage of the chemistry in your system. If the error lists specific bonds or valence terms, those terms are the ones that could not be assigned parameters. .. _troubleshooting-nan-errors-during-simulation: ``NaN`` errors during simulation -------------------------------- You may see errors like: .. code:: bash openmm.OpenMMException: Particle coordinate is NaN. .. code:: bash openmmtools.multistate.utils.SimulationNaNError: Propagating replica 0 at state 10 resulted in a NaN! For more information on the OpenMM error, see the OpenMM FAQ entry on NaNs: https://github.com/openmm/openmm/wiki/Frequently-Asked-Questions#nan These errors usually mean that the simulation became numerically unstable during minimization or propagation. In some cases the simulation can recover by restarting from the last stable state. By default the protocols will attempt this up to ``20`` times before giving up, so you may see multiple ``NaN`` errors in the logs even if the simulation ultimately completes. A successful restart does not necessarily mean that the resulting free-energy estimate is unreliable. However, repeated ``NaN`` errors may indicate an underlying stability problem. Before trusting the result, investigate the transformation and input structures for likely causes of instability and check the usual diagnostics, including agreement between independent repeats. **Possible causes** - Missing capping groups or other issues in the receptor structure. - A poor atom mapping, including mappings that break bonds, map too few heavy atoms or map atoms whose hybridization changes due to a single to double/triple bond transformation. - An initial clash between the ligand and a crystal water, the receptor, or another part of the system that the minimizer could not relax. - A poor input structure for the ligand that the minimizer could not fix. **Suggested fixes** - Run the `validation script `_ on the transformation JSON to try to identify the source of the problem. - Inspect the receptor for missing residues or missing capping groups. - Review the atom mapping for bond-breaking transformations or mappings with too few heavy atoms, try to improve the mapping by aligning the ligands before generating the mapping. - Consider running the transformation using the `SepTop `_ protocol which does not require an atom mapping and is robust to poor ligand alignment. - Check the input ligand and starting pose for clashes or poor geometry, try to relax the ligand in the receptor before running the protocol. - If possible, rebuild or re-prep the input structures before rerunning the protocol. The state of the system and integrator before the error are often saved in a ``nan-error-logs`` directory, which can help with debugging. .. _troubleshooting-validate-inputs-with-plain-md: Validating your inputs with a plain MD simulation ------------------------------------------------- Many of the errors above (e.g. parameter assignment failures, ``NaN`` errors during simulation, box size errors) are due to problems with the input structures which are not identified until protocol execution. A quick way to check this *before* running an expensive free energy protocol is to run a short MD simulation of your system with the :class:`.PlainMDProtocol` (see the :doc:`Plain MD protocol guide `). This helps check that: - The receptor can be parameterized together with the ligand (e.g. no missing residues, caps or unsupported chemistry) by the same force fields you intend to use in your production protocol. - The **non-alchemical** system can be minimized and equilibrated producing a stable simulation without clashes or ``NaN`` errors. If a system is stable in plain MD it is much more likely to be suitable for other protocols, for failures see sections :ref:`troubleshooting-nan-errors-during-simulation` and :ref:`troubleshooting-smirnoff-force-field-parameter-assignment-errors` for next steps. We recommend running this for a representative subset of the ligands with the receptor, using the same force fields and solvent settings you intend to use in your production protocol. The simplest way to run this check is via the Python API with the example below which uses short equilibration and production lengths, which can catch common setup problems: .. code-block:: python import openfe from gufe.protocols import execute_DAG from openff.units import unit from openfe.protocols.openmm_md.plain_md_methods import PlainMDProtocol import pathlib ligand = openfe.SmallMoleculeComponent.from_sdf_file("ligand.sdf") # load your ligand from your SDF file protein = openfe.ProteinComponent.from_pdb_file("protein.pdb") # load your protein from your PDB file solvent = openfe.SolventComponent() # use the same settings as your production run # If you have cofactors in the system load them here as SmallMoleculeComponents and include them in the ChemicalSystem below. # cofactor = openfe.SmallMoleculeComponent.from_sdf_file("cofactor.sdf") system = openfe.ChemicalSystem( { "ligand": ligand, "protein": protein, "solvent": solvent, # "cofactor": cofactor # include any cofactors here }, name="validation", ) settings = PlainMDProtocol.default_settings() settings.simulation_settings.equilibration_length_nvt = 1 * unit.picosecond settings.simulation_settings.equilibration_length = 1 * unit.picosecond settings.simulation_settings.production_length = 1 * unit.picosecond # You will need a GPU with CUDA installed to run this quick check, we recommend using the same platform as your production simulations. settings.engine_settings.compute_platform = "CUDA" protocol = PlainMDProtocol(settings=settings) dag = protocol.create(stateA=system, stateB=system, mapping=None) output_dir = pathlib.Path("plain_md_check") output_dir.mkdir(exist_ok=True, parents=True) result = execute_DAG( dag, shared_basedir=output_dir, scratch_basedir=output_dir, keep_shared=True, raise_error=True, n_retries=0 ) # Check for any errors in the execution if not result.ok(): for failure in result.protocol_unit_failures: # If we find a failure print the traceback to help with debugging print(f"Protocol unit {failure.name} failed with error: {failure.traceback}") Resuming Interrupted Simulations -------------------------------- In the case where your ``openfe quickrun`` execution has timed out, was cancelled, or was otherwise interrupted, you can attempt to resume the simulation using the ``--resume`` flag, which is available as of openfe v1.11.0. See :ref:`Resuming Interrupted Simulations ` for more details. .. _troubleshooting-log-debug-information: Log Debug information --------------------- .. note:: When using a scheduler (e.g. SLURM), be sure to specify output files for standard out and standard error. For example, when using SLURM both ``--output=`` and ``--error=`` must be set to view errors. One of the first troubleshooting steps is to increase the verbosity of the logging. ``openfe`` uses Python's native logging library which can be `configured `_ either using a Python API or a configuration file. .. warning:: **We do not recommend setting the log level to debug for production runs,** as the logging may slow down the simulation and add a lot of noise to the output. When using ``openfe quickrun``, the configuration file is more convenient. Below is an example logging configuration file that can be used to set the log level to ``DEBUG``: .. code-block:: ini [loggers] keys=root [handlers] keys=stdout [formatters] keys=standardFormatter,msgOnly [handler_stdout] class=StreamHandler level=DEBUG formatter=standardFormatter args=(sys.stdout,) [logger_root] level=DEBUG handlers=stdout [formatter_standardFormatter] format=%(asctime)s %(levelname)s %(name)s: %(message)s [formatter_msgOnly] format=%(message)s Save this configuration file as ``debug_logging.conf`` and then run ``openfe quickrun`` with the ``--log`` flag, for example: .. code-block:: bash $ openfe --log debug_logging.conf quickrun -d results/ -o results/result_lig_ejm_31_solvent_lig_ejm_42_solvent.json transformations/rbfe_lig_ejm_31_solvent_lig_ejm_42_solvent.json Note that the ``--log debug_logging.conf`` argument goes between ``openfe`` and ``quickrun`` on the command line. This will cause every package to log at the debug level, which may be quite verbose and noisy but should aid in identifying what is going on right before the exception is thrown. .. _troubleshooting-jax-warnings: JAX warnings ------------ We use ``pymbar`` to analyze the free energy of the system. ``pymbar`` uses JAX to accelerate computation. The JAX library can utilize a GPU to further accelerate computation. If the necessary libraries for GPU acceleration are not installed and JAX detects a GPU, JAX will print a warning like this: .. code-block:: bash WARNING:2025-06-10 09:01:40,857:jax._src.xla_bridge:966: An NVIDIA GPU may be present on this machine, but a CUDA-enabled jaxlib is not installed. Falling back to cpu. This warning does not mean that the *molecular dynamics* simulation will fall back to using the CPU. The simulation will still use the computing platform specified in the settings. .. _troubleshooting-pymbar-disable-jax: PYMBAR_DISABLE_JAX ------------------ Due to a suspected memory leak in the JAX acceleration code in ``pymbar`` we disable JAX acceleration by default. This memory leak may result in the simulation crashing, wasting compute time. The error message may look like this: .. code-block:: bash LLVM compilation error: Cannot allocate memory LLVM ERROR: Unable to allocate section memory! We have decided to disable JAX acceleration by default to prevent wasted compute. However, if you wish to use the JAX acceleration, you may set ``PYMBAR_DISABLE_JAX`` to ``FALSE`` (e.g. put ``export PYMBAR_DISABLE_JAX=FALSE`` in your submission script before running ``openfe quickrun``). For more information, see these issues on github: - https://github.com/choderalab/pymbar/issues/564 - https://github.com/OpenFreeEnergy/openfe/issues/1534 - https://github.com/OpenFreeEnergy/openfe/issues/1654 CUDA-related errors ------------------- You may see CUDA errors, such as ``openmm.OpenMMException: Error loading CUDA module``, especially when running across multiple machines, e.g. HPC systems. Refer to :ref:`installation-troubleshooting` for guidance on debugging your installation and :ref:`installation-cuda-errors` for suggestions for how to resolve CUDA-specific errors.