Source research

Quantifying the advantage of domain-specific pre-training on named entity recognition tasks in materials science

Amalie Trewartha, Nicholas A. Walker, Haoyan Huo, Sang-Hoon Lee, Kevin Cruse, John M. Dagdelen et al. · 2022 · 163 citationsRead the paper

A bottleneck in efficiently connecting new materials discoveries to established literature has arisen due to an increase in publications. This problem may be addressed by using named entity recognition (NER) to extract structured summary-level data from unstructured materials science text. We compare the performance of four NER models on three materials science datasets. The four models include a bidirectional long short-term memory (BiLSTM) and three transformer models (BERT, SciBERT, and MatBERT) with increasing degrees of domain-specific materials science pre-training. MatBERT improves over the other two BERT BASE -based models by 1%∼12%, implying that domain-specific pre-training provides measurable advantages. Despite relative architectural simplicity, the BiLSTM model consistently outperforms BERT, perhaps due to its domain-specific pre-trained word embeddings. Furthermore, MatBERT and SciBERT models outperform the original BERT model to a greater extent in the small data limit. MatBERT's higher-quality predictions should accelerate the extraction of structured data from materials science literature.

What the full text shows

Working prototype

MatBERT outperforms SciBERT, BERT, and BiLSTM models across three materials science datasets. — 1% to 12% F1-score improvement over other BERT-base models (Across solid-state, doping, and gold nanoparticle datasets)
Domain-specific pre-training (MatBERT) provides a measurable advantage over general scientific pre-training (SciBERT). — 1% to 4% F1-score improvement (Across the three materials science datasets)
Scientific pre-training (SciBERT) provides a significant advantage over general-purpose pre-training (BERT). — 3% to 9% F1-score improvement (Across the three materials science datasets)
BiLSTM with domain-specific embeddings outperforms the original BERT model. — '' (Across the tested materials science datasets)
MatBERT and SciBERT show greater performance gains over BERT in low-data regimes. — '' (Small training set sizes)

Setup: Three datasets: 800 solid-state abstracts, 455 doping-related sentences, and 85 gold nanoparticle characterization paragraphs.
How: The researchers compared four NER models (BiLSTM, BERT, SciBERT, and MatBERT) using an IOBES tagging scheme and a Conditional Random Field (CRF) layer for decoding. MatBERT was pre-trained from scratch on 8.8 billion tokens from 2 million materials science papers using Masked Language Modeling (MLM).
Limitations: MatBERT pre-training corpus is biased toward experimental synthesis and trending topics. · Under-representation of computational papers (DFT results) and theoretical stoichiometries. · Performance of all models remains poor in the extreme low-data limit.

Summarized by Gemma 4 from the open-access full text · 3 Oct 2026. Read it