Distil* vs GPT-4

Discover cutting-edge machine learning with Hugging Face Transformers, which offers state-of-the-art models for Pytorch, TensorFlow, and JAX. Dive into the 'distillation' research project on GitHub to explore how knowledge distillation techniques can compress large, complex models into smaller, faster counterparts without significantly sacrificing performance.

This part of the Hugging Face Transformers repository contains examples and scripts that demonstrate the training and implementation of distilled models such as DistilBERT, DistilRoBERTa, and DistilGPT2. Learn from the detailed documentation and updates about the ongoing improvements, and understand how these models can be used in practical applications for efficient natural language processing.

GPT-4 is the latest milestone in OpenAI’s effort in scaling up deep learning.

GPT-4 is a large multimodal model (accepting image and text inputs, emitting text outputs) that, while less capable than humans in many real-world scenarios, exhibits human-level performance on various professional and academic benchmarks. For example, it passes a simulated bar exam with a score around the top 10% of test takers; in contrast, GPT-3.5’s score was around the bottom 10%. We’ve spent 6 months iteratively aligning GPT-4 using lessons from our adversarial testing program as well as ChatGPT, resulting in our best-ever results (though far from perfect) on factuality, steerability, and refusing to go outside of guardrails.

GPT-4 is more creative and collaborative than ever before. It can generate, edit, and iterate with users on creative and technical writing tasks, such as composing songs, writing screenplays, or learning a user’s writing style.