Last updated 07-27-2026
Category:
Reviews:
Join thousands of AI enthusiasts in the World of AI!
UniLM
UniLM is a pre-trained language model from Microsoft Research that handles both natural language understanding and text generation from one shared Transformer. You fine-tune a single checkpoint for reading tasks like question answering and writing tasks like summarization or dialogue, without maintaining separate encoder-only and decoder-only models. Code and pretrained weights ship through the microsoft/unilm GitHub repo under an MIT license.
BERT-style models excel at reading but need a separate decoder stack for generation. UniLM trains one Transformer with three attention-mask modes: unidirectional, bidirectional, and sequence-to-sequence. That design let the same weights compete with BERT on GLUE and SQuAD while setting summarization and question-generation benchmarks in 2019, a split that most contemporaries treated as two problems.
ML researchers and NLP engineers use UniLM when they want published benchmarks, training scripts, and checkpoint files for both understanding and generation in one codebase. The repository now spans later releases like UniLMv2 (Pseudo-Masked Language Model, ICML 2020), but v1 remains the reference for the original unified masking approach described in the NeurIPS 2019 paper.
Three pre-training objectives (unidirectional, bidirectional, sequence-to-sequence) share one Transformer backbone
CNN/DailyMail abstractive summarization ROUGE-L of 40.51, a 2.04-point gain over prior work
CoQA generative question answering F1 score of 82.5 on the published benchmark
SQuAD question generation BLEU-4 of 22.12 with beam search decoding
Pre-trained checkpoints and PyTorch training scripts in the microsoft/unilm repository (22.2k GitHub stars)
MIT license with UniLM v1 and UniLMv2 code paths in the same open-source repo
One pre-trained model covers both understanding and generation tasks after fine-tuning.
Published benchmark numbers on GLUE, SQuAD, CNN/DailyMail, and CoQA with reproducible scripts.
MIT-licensed code and checkpoint downloads through the microsoft/unilm GitHub repo.
Repository grew into a broader Microsoft foundation-model hub with UniLMv2 and related releases.
Requires PyTorch, CUDA GPUs, and manual setup rather than a hosted API.
Original UniLM v1 training scripts target older pytorch-transformers v0.4.0 dependencies.
No managed inference service; you run fine-tuning and decoding on your own hardware.
What is UniLM?
UniLM is a unified pre-trained language model from Microsoft Research that fine-tunes one Transformer for both natural language understanding and generation. It was introduced in the NeurIPS 2019 paper and ships as open-source code on GitHub.
Is UniLM free to use?
Yes, UniLM is free. The microsoft/unilm repository is released under the MIT license, and you can download pre-trained checkpoints and training scripts without payment.
What tasks can you fine-tune UniLM for?
UniLM supports fine-tuning for NLU tasks like GLUE and SQuAD question answering, plus NLG tasks like CNN/DailyMail summarization, SQuAD question generation, CoQA generative QA, and DSTC7 dialog response generation.
How is UniLM different from BERT?
BERT uses bidirectional attention for understanding only. UniLM adds unidirectional and sequence-to-sequence attention masks so the same pre-trained weights handle both reading and generation after fine-tuning.
Where do you download UniLM pre-trained models?
Pre-trained UniLM v1 checkpoints and fine-tuning code live in the unilm-v1 folder of the microsoft/unilm GitHub repository. The README links to Google Drive checkpoint files for base and large models.
What license covers the UniLM code?
UniLM code is released under the MIT license in the microsoft/unilm repository. Portions build on pytorch-transformers v0.4.0, and the LICENSE file in the repo root defines the terms.