Resources

4 results for "pretraining"

Model Credentialed Access

EntityBERT: BERT-based Models Pretrained on MIMIC-III with or without Entity-centric Masking Strategy for the Clinical Domain

Chen Lin, Steven Bethard, Guergana Savova, et al.

Transformer-based neural language models have led to breakthroughs for a variety of natural language processing (NLP) tasks. However, most models are pretrained on general domain data. We propose a methodology to produce a model focused on the clini…
Published: March 17, 2022. Version: 1.0.1
Model Credentialed Access

Clinical BERT Models Trained on Pseudo Re-identified MIMIC-III Notes

Eric Lehman, Sarthak Jain, Karl Pichotta, et al.

This project contains the weights of 7 different BERT models trained over a surrogate re-identified set of MIMIC-III (v1.4) notes. Each model is suitable for direct compatibility with the HuggingFace framework; users can easily load and apply the mo…
Published: April 28, 2021. Version: 1.0.0
Model Credentialed Access

Clinical-T5: Large Language Models Built Using MIMIC Clinical Text

Eric Lehman, Alistair Johnson

Recent advances in scaling large language models (LLMs) has resulted in significant improvements over a number of natural language processing benchmarks. There has been some work to pretrain these language models over clinical text. These works demo…
Published: Jan. 25, 2023. Version: 1.0.0
Database Credentialed Access

MIMIC-Ext-SynNotes: Large-Scale Synthetic Clinical Notes via Rephrasing MIMIC-III/IV with LLMs

Jinghui Liu, Sarvesh Soni, Anthony Nguyen

MIMIC-III and MIMIC-IV are foundational resources for clinical NLP, but their scale is modest relative to the data volumes needed to train language models. Building on recent advances in synthetic data generation through LLM-based rephrasing, we con…
Published: Oct. 5, 2026. Version: 1.0