# WBM-MatPES

A benchmark for universal machine-learning interatomic potentials (uMLIPs),
built entirely at **MatPES PBE settings**, with a high-pressure extension that
reaches 150 GPa.

Every energy in this release was computed with one uniform set of VASP
settings, with **no Hubbard _U_** and with **no empirical energy corrections**
applied anywhere. A potential trained on MatPES can therefore be scored
against a reference that is consistent with its own training data. Energies
from this release are **not** interchangeable with Materials Project or
Alexandria energies.

---

## Files

| file | contents | entries |
|---|---|---|
| `matpes_wbm_test_00000.json.bz2`, `matpes_wbm_test_00001.json.bz2` | the test set: candidates to relax and score | 182,427 |
| `matpes_mp_ref_annotated_00000.json.bz2` | Materials-Project-equivalent reference set | 89,008 |
| `matpes_mp_convex_hull.json.bz2` | hull derived from that reference set (vertices only) | 30,873 |
| `matpes_alex_convex_hull.json.bz2` | Alexandria hull at the same settings | 166,133 |
| `matpes_P025_test_00000.json.bz2` … `matpes_P150_test_00000.json.bz2` | six pressure test sets, 25–150 GPa | 47,792 total |
| `testset_flags.csv` | `mat_id`, `wbm_id`, prototype-uniqueness flags for the test set | 182,427 |
| `testset_nsites.csv` | `mat_id`, cell sizes, DFT energy | 182,427 |
| `pressure_null_baseline.csv` | volume error of the unrelaxed input, per pressure | 6 |

All `.json.bz2` files are compressed JSON holding a list of `pymatgen`
`ComputedStructureEntry` objects under the key `entries`.

```python
import bz2, json
from pymatgen.entries.computed_entries import ComputedStructureEntry

with bz2.open("matpes_wbm_test_00000.json.bz2", "rt") as fh:
    entries = [ComputedStructureEntry.from_dict(e) for e in json.load(fh)["entries"]]
```

---

## Computational settings

VASP, PBE exchange and correlation, PAW pseudopotentials, following the MatPES
settings:

| | |
|---|---|
| plane-wave cutoff | 680 eV |
| _k_-point grid | Γ-centered, spacing 0.22 Å⁻¹ |
| smearing | Gaussian, σ = 0.05 eV |
| spin | polarized throughout |
| convergence | 10⁻⁷ eV (energy), 5 meV Å⁻¹ (forces) |
| Hubbard _U_ | none |
| energy corrections | none |

The PAW setups are more recent than those used by the Materials Project and by
the ambient-pressure Alexandria database, and were chosen for their improved
accuracy. The smearing correction proposed as part of MatPES was not applied.

---

## The test set

Built from the WBM dataset, which was generated by successive elemental
substitution into known prototypes. 196,636 WBM structures were recomputed at
the settings above; 191,096 converged. From those we removed 7,488 structures
that also belong to the reference set, since a compound already present in the
reference is not a discovery, and 1,181 entries whose unrelaxed geometry was
unavailable or described a supercell of the relaxed one. **182,427 structures
remain.**

Each entry carries the unrelaxed geometry in `data["initial_structure"]` and
the DFT-relaxed geometry in `structure`.

**The benchmark task is initial-structure-to-relaxed-energy.** A model is given
`data["initial_structure"]`, relaxes atomic positions and cell, and its final
energy is converted into a distance to one of the hulls. `structure` is the
reference for the structural comparison only; relaxing from it instead would
be a different, easier task.

Report metrics on the **unique-prototype subset**, 178,971 structures, selected
by `data["unique_prototype"]`. This follows the Matbench Discovery convention
and keeps results comparable with it. The flag removes only 1.9% of candidates
here, against 16% of full WBM, because duplicates were already rejected when
entries entered our database.

---

## The two reference hulls

Both are computed at the settings above and should be used side by side. A
candidate that is stable against the first but not the second is a false
positive that a single-hull benchmark structurally cannot see.

**`matpes_mp_convex_hull.json.bz2`** is equivalent to the Materials Project.
Every entry of the 154,718-structure Materials Project snapshot used by
Matbench Discovery was structure-matched against our database of calculations
performed at Materials Project settings, and the matched material was then
recomputed at MatPES settings. Matching at the original settings is
deliberate: relaxation at MatPES settings can move a geometry far enough that
it no longer matches its parent. 125,214 entries matched; after a
root-mean-square cut and the removal of entries claimed by more than one
Materials Project material, 116,675 distinct structures remained, of which
89,008 have a converged MatPES counterpart. The other 27,667 did not converge
within our computational budget, mostly because of the size of their cells.
30,873 of the 89,008 lie on the hull. Every elemental endpoint the test set
needs is present; nothing was added by hand.

**`matpes_alex_convex_hull.json.bz2`** is the full Alexandria hull recomputed
at the same settings, 166,133 entries spanning 89 elements. It is
substantially deeper.

### Which stability metric to use

Fraction of the test set that is thermodynamically stable in DFT, on the
unique-prototype subset:

| hull | by phase separation energy | by distance above hull |
|---|---|---|
| Materials-Project-derived | 21.54% | 17.95% |
| Alexandria | 12.54% | 7.23% |

**Against the Alexandria hull, use the phase separation energy only.** The WBM
structures are themselves part of Alexandria, so a candidate can lie exactly on
that hull, where its distance above the hull is zero and any approximate
estimate falls on either side of the threshold with roughly equal probability,
however accurate the model. The phase separation energy measures the distance
to the hull formed by all *other* compositions and is well defined in that
case. Against the Materials-Project-derived hull both metrics are valid,
because the overlapping candidates were removed from the test set.

The gap between the two hulls is the quantity this release exists to expose:
roughly four in ten of the compounds that look like discoveries against a
Materials-Project-like reference are already beaten by an entry of Alexandria.

---

## The pressure test sets

Six sets at 25, 50, 75, 100, 125 and 150 GPa, 47,792 structures in total
(8,154 / 7,897 / 7,929 / 8,166 / 8,173 / 7,473). The compared quantity is the
**enthalpy** _H_ = _E_ + _PV_, stored in `energy`, and **no convex hull is
involved**. Hulls at finite pressure are of uneven quality and would not be
comparable with the ambient construction, so the analysis is restricted to the
enthalpy and the relaxed geometry.

The target pressure was imposed through the stress tensor during the DFT
relaxation. Each DFT relaxation was itself started from a geometry relaxed at
that pressure with GRACE, and it is that geometry, in
`data["initial_structure"]`, that should be handed to a model, so that it
receives exactly the input the reference calculation received.

Because that input is already close to the answer, `pressure_null_baseline.csv`
gives the volume error of returning it unchanged, 1.28% at 25 GPa rising to
2.08% at 150 GPa. A relaxation has to beat that to be useful.

Useful per-entry fields: `volume_per_atom_dft` is the reference volume,
`ambient_energy_total` the ambient-pressure total energy of the same compound,
and `initial_volume_per_atom` the volume of the starting geometry.

---

## Entry fields

Present on every entry:

| field | meaning |
|---|---|
| `mat_id` | identifier in our database |
| `formula`, `elements`, `nsites`, `spg`, `prototype_id` | composition and symmetry |
| `energy_total` | total energy in eV (enthalpy for the pressure sets) |
| `energy_corrected` | equal to `energy_total`; no corrections are applied |
| `stress` | in kB, VASP sign convention |
| `total_mag`, `band_gap_ind`, `band_gap_dir`, `dos_ef` | electronic properties |
| `e_form`, `e_above_hull`, `e_phase_separation`, `decomposition` | thermodynamics |

Test set only:

| field | meaning |
|---|---|
| `initial_structure` | the unrelaxed WBM geometry, the input for the task |
| `wbm_id` | Matbench Discovery identifier, e.g. `wbm-1-10450` |
| `unique_prototype` | Matbench Discovery's flag; 178,971 true |
| `unique_prototype_agm` | our native `(prototype, chemical system)` flag; 181,310 true |
| `wyckoff_spglib` | Aflow-style prototype label |
| `mb_e_form_mp2020`, `mb_e_above_hull_mp2020` | Matbench Discovery's values for the same structure, **at Materials Project settings**. Cross-check columns only. Never mix them with MatPES hull distances. |

Pressure sets only: `pressure_GPa`, `volume_per_atom_dft`,
`initial_volume_per_atom`, `ambient_energy_total`.

---

## Citation

TODO: add the citation and DOI on publication.