Experimental Design#

This study tests whether machine-learning force fields recover after their input atomic coordinates are deliberately perturbed. Mathematical definitions are provided separately in Equations.

Workflow#

Every adversarial experiment uses the same sequence:

  1. Load an initial atomic structure.

  2. Relax it using the selected MLFF.

  3. Save the relaxed structure as the reference.

  4. Apply an adversarial coordinate perturbation.

  5. Measure the immediate response.

  6. Relax the perturbed structure with the same MLFF.

  7. Compare the final structure with the reference.

Initial structure
       ↓
First relaxation
       ↓
Reference structure
       ↓
Adversarial perturbation
       ↓
Immediate measurements
       ↓
Second relaxation
       ↓
Recovery measurements

The first relaxed structure is the reference. The original unrelaxed input is not used as the recovery target.

Datasets#

Two structure sets are evaluated:

  • Two-dimensional Materials Project: 20 low-dimensional structures.

  • LiCOHPF: 20 Li-C-O configurations associated with solid-electrolyte-interphase modelling.

The two datasets test whether observed behaviour is specific to one chemistry or structural domain.

Models#

Five MLFFs are compared:

  • MACE, trained using LiCOHPF data (MACE).

  • MTP, trained using LiCOHPF data (MTP).

  • MACE-MH, a pretrained MACE-family model (MACE-MH).

  • UMA, Meta’s pretrained Universal Model for Atoms (UMA).

  • CHGNet, a pretrained crystal graph neural network (CHGNet).

The experiment compares model behaviour, not only computational speed or ordinary test-set accuracy.

Adversarial attacks#

Three coordinate attacks are evaluated:

  • FGSM: one sign-gradient update (FGSM).

  • I-FGSM: repeated sign-gradient updates (I-FGSM).

  • PGD: repeated updates projected into the allowed perturbation region (PGD).

The attacks modify atomic coordinates. Model parameters remain fixed.

Attack variables include:

  • epsilon;

  • normalized epsilon;

  • number of attack steps;

  • attack step size;

  • random seed.

The attack equations are defined in Attack methods.

Relaxation#

The structure is relaxed before and after the attack using the same model and relaxation settings.

The configured convergence threshold is:

fmax = 0.01 eV/Å

The maximum relaxation length is:

300 optimizer steps

A run reaching 300 steps reached the configured limit and should not automatically be described as converged.

Relaxation uses the Atomic Simulation Environment optimization interface (ASE optimization).

Evaluation stages#

Metrics are recorded at two stages:

Immediate response

The attacked structure is compared with the pre-attack relaxed reference before the second relaxation.

Post-relaxation response

The final structure is compared with the same reference after the second relaxation.

This separates attack magnitude from recovery behaviour.

Baseline#

Contour exploration is the non-adversarial baseline. It explores a targeted energy contour instead of selecting an adversarial gradient direction.

The implementation uses ASE contour exploration (contour exploration).

Contour exploration is a structured baseline, not a random-noise baseline.

Experiment coverage#

The benchmark evaluates two sets of 20 structures: LiCOHPF Li-C-O configurations and two-dimensional Materials Project structures. Each structure is tested using MACE, MACE-MH, UMA, CHGNet, and MTP.

The adversarial evaluation includes FGSM, I-FGSM, and PGD. Attack configurations span nominal \(\epsilon\) values from 0.001 to 10 and 1 to 100 attack steps where applicable. Contour exploration provides a 500-step non-adversarial constant-energy baseline.

Experiments use random seeds 42 through 46. MACE, MACE-MH, UMA, and CHGNet are evaluated in float32 and float64, while MTP is evaluated in float64. Reported epsilon values may additionally be normalized by each structure’s minimum lattice-vector length for cross-structure comparison.

Experimental comparisons#

The broader experiment compares:

  • random seeds 42 through 46;

  • float32 and float64 where supported;

  • epsilon sweeps;

  • attack-step sweeps;

  • attack step-size sweeps;

  • immediate and post-relaxation outcomes;

  • primitive structures and supercells;

  • adversarial attacks and contour exploration.

The experiment seeds control stochastic experimental operations. They do not retrain the downloaded model artifacts.

Supercell stress test#

The optional supercell workflow repeats the experiment using expanded periodic structures. It tests whether observed behaviour persists when the represented atomic system becomes larger.

Supercells are constructed using ASE structure-building operations (ASE supercell).

Measured outcomes#

The analysis includes:

  • displacement;

  • delta force and force angle;

  • relaxation steps;

  • final energy;

  • neighbour-edge changes;

  • neighbour-set Jaccard distance;

  • coordination changes;

  • RDF L1 distance;

  • space-group change;

  • symmetry-operation retention;

  • unique-site change;

  • float32/float64 agreement;

  • cross-seed variability.

Energy agreement alone does not guarantee correct structural recovery, as small errors in the learned potential energy surface can alter relaxation trajectories and lead to different local minima (ASE supercell). Structure-based metrics were chosen to be more approxpriate metrics for evaluating MLFFs.

Every mathematical definition is listed on the Equations page.

Interpretation#

The experiment evaluates robustness under deliberately challenging coordinate perturbations. It does not directly measure error against DFT unless an explicit DFT reference is included.

A persistent post-relaxation difference indicates failure to recover the pre-attack relaxed configuration under the selected model and settings. It does not, by itself, prove that the final structure is physically impossible.

The validation context is discussed in the MLFF validation.