Results#
This page summarizes the measured outcomes and principal findings. Blank fields should be completed only after the corresponding summary files have been validated.
Index#
2D Materials Project Cases#
Model |
Device |
Floating-point precision |
Attacks |
Epsilon values |
Iterative attack steps |
Random seeds |
Cases per material |
|---|---|---|---|---|---|---|---|
MACE-MH-1 |
CPU |
|
FGSM, I-FGSM, PGD |
0.001–0.005, 0.01–0.05, 0.1–0.5, 1–5, 10 |
1, 2, 3, 4, 5, 10, 20, 30, 40, 50, 100 |
42, 43, 44, 45, 46 |
850 |
UMA-S-1p1 |
CPU |
|
FGSM, I-FGSM, PGD |
0.001–0.005, 0.01–0.05, 0.1–0.5, 1–5, 10 |
1, 2, 3, 4, 5, 10, 20, 30, 40, 50, 100 |
42, 43, 44, 45, 46 |
850 |
CHGNet |
CPU |
|
FGSM, I-FGSM, PGD |
0.001–0.005, 0.01–0.05, 0.1–0.5, 1–5, 10 |
1, 2, 3, 4, 5, 10, 20, 30, 40, 50, 100 |
42, 43, 44, 45, 46 |
850 |
Total |
CPU |
6 model/precision combinations |
3 attacks |
21 epsilon values |
11 step values |
5 seeds |
2,550 per material |
The 2D Materials Project dataset contains 20 materials, giving a total of 51,000 benchmark cases.
LiCoHPF Database Cases#
Model |
Device |
Floating-point precision |
Attacks |
Epsilon values |
Iterative attack steps |
Random seeds |
Cases per material |
|---|---|---|---|---|---|---|---|
MACE-MH-1 |
CPU |
|
FGSM, I-FGSM, PGD |
0.001–0.005, 0.01–0.05, 0.1–0.5, 1–5, 10 |
1, 2, 3, 4, 5, 10, 20, 30, 40, 50, 100 |
42, 43, 44, 45, 46 |
850 |
UMA-S-1p1 |
CPU |
|
FGSM, I-FGSM, PGD |
0.001–0.005, 0.01–0.05, 0.1–0.5, 1–5, 10 |
1, 2, 3, 4, 5, 10, 20, 30, 40, 50, 100 |
42, 43, 44, 45, 46 |
850 |
CHGNet |
CPU |
|
FGSM, I-FGSM, PGD |
0.001–0.005, 0.01–0.05, 0.1–0.5, 1–5, 10 |
1, 2, 3, 4, 5, 10, 20, 30, 40, 50, 100 |
42, 43, 44, 45, 46 |
850 |
MTP |
CPU |
|
FGSM, I-FGSM, PGD |
0.001–0.005, 0.01–0.05, 0.1–0.5, 1–5, 10 |
1, 2, 3, 4, 5, 10, 20, 30, 40, 50, 100 |
42, 43, 44, 45, 46 |
425 |
MACE Model |
GPU |
|
FGSM, I-FGSM, PGD |
0.001–0.005, 0.01–0.05, 0.1–0.5, 1–5, 10 |
1, 2, 3, 4, 5, 10, 20, 30, 40, 50, 100 |
42, 43, 44, 45, 46 |
850 |
Total |
CPU and GPU |
9 model/precision combinations |
3 attacks |
21 epsilon values |
11 step values |
5 seeds |
3,825 per material |
The LiCoHPF dataset contains 20 materials, giving a total of 76,500 benchmark cases.
For each model, precision, material, and random seed, the benchmark contains:
63 epsilon-sweep cases: 21 epsilon values multiplied by 3 attacks.
22 step-sweep cases: 11 step values multiplied by I-FGSM and PGD.
85 combined epsilon- and step-sweep cases.
Outcomes#
Outcome |
Stage |
Result |
|---|---|---|
Delta force |
Post-attack |
MACE and MTP break off to 106 – 109 after 1 min lattice |
Delta force |
Post-attack + relaxation |
MACE and MTP remain at 108`eV/Å after perturbation of ~1% min. lattice, while other models ``<` 1eV/Å |
Displacement |
Post-attack |
Smooth exponential increase as epsilon sizes increase |
Displacement |
Post-attack + relaxation |
Models begin to diverge, MACE and MTP experience a greater initial spike at ~1% min lattice |
Relaxation steps |
Initial relaxation |
CHGNet takes most steps ~250, MACE and MTP take the least |
Relaxation steps |
Post-attack + relaxation |
Fill in |
Neighbour Jaccard distance |
Post-attack |
Fill in |
Neighbour Jaccard distance |
Post-attack + relaxation |
Fill in |
Maximum coordination change |
Post-attack |
Fill in |
Maximum coordination change |
Post-attack + relaxation |
Fill in |
RDF L1 distance |
Post-attack |
Fill in |
RDF L1 distance |
Post-attack + relaxation |
Fill in |
Space-group change |
Post-attack |
Fill in |
Space-group change |
Post-attack + relaxation |
Fill in |
Symmetry-operation retention |
Post-attack |
Fill in |
Symmetry-operation retention |
Post-attack + relaxation |
Fill in |
Unique-site change |
Post-attack |
Fill in |
Unique-site change |
Post-attack + relaxation |
Fill in |
Key takeaways#
Finding |
Question |
Result |
|---|---|---|
Recovery threshold |
At what normalized epsilon does recovery begin to fail? |
1% of min lattice parameter |
Largest force response |
Which model/attack/material produces the largest delta force? |
MACE and MTP pretrained by LiCOHPF dataset |
Slowest convergence |
Which model most frequently reaches the 600-step limit? |
CHGNet |
Most recoverable model |
Which model has the smallest post-relaxation changes? |
Fill in |
Least recoverable model |
Which model has the largest persistent changes? |
Fill in |
Attack comparison |
How do FGSM, I-FGSM, and PGD differ under matched budgets? |
Fill in |
Chemistry dependence |
Do LiCOHPF and 2D materials exhibit different failure patterns? |
Fill in |
Topology transition |
When do connectivity or coordination changes become persistent? |
Fill in |
Precision dependence |
Do float32 and float64 support the same conclusions? |
Fill in |
Seed dependence |
Are conclusions stable across seeds 42 through 46? |
No abnormalities which suggest results are stable and reliable. |
Contour exploration baseline |
Are adversarial directions more damaging than contour motion? |
Slight increase but many magnitudes less than deliberate attacks, suggesting that atoms merely moving away from equilibrium do not lead to failure modes compared to adversarial perturbations that maximize model loss. |
Supercell dependence |
Do conclusions persist for expanded periodic systems? |
Fill in |