GPT4o (Omni) vs ChatGPT Plugins

GPT-4o ("o" for "omni") represents a significant leap towards more natural interactions between humans and computers. It's designed to handle a mix of text, audio, image, and video inputs, and can output text, audio, and images. Impressively, GPT-4o can process audio inputs in just 232 milliseconds on average, nearly matching human response times in conversation. This model not only retains the high performance of GPT-4 Turbo in English and coding tasks but also shows marked improvements in processing non-English languages, all while being faster and 50% more cost-effective via its API. Additionally, GPT-4o excels in understanding vision and audio better than previous models.

Model capabilities include:

• Two GPT-4os interacting and singing
• Interview preparation
• Playing Rock Paper Scissors
• Detecting sarcasm
• Mathematical discussions with figures like Sal and Imran Khan
• Harmonizing in music
• Language learning through interaction
• Real-time meeting translations
• Singing lullabies or birthday songs
• Humor with dad jokes
• Assisting visually impaired users in real-time through partnerships like BeMyEyes

Prior models like GPT-3.5 and GPT-4, in Voice Mode, involved a multi-step process with latencies up to 5.4 seconds. This process used separate models to transcribe audio to text, process the text, and then convert responses back to audio. This often resulted in a loss of nuanced information like tone, emotion, or background sounds.

GPT-4o simplifies this with a unified model that handles text, vision, and audio end-to-end, preserving the richness of the inputs and enabling more expressive outputs. As our first foray into such an integrated model, GPT-4o opens new avenues for exploring multimodal interactions and their potential applications.

OpenAI follows an iterative deployment philosophy, and as part of this approach, it is gradually releasing plugins for ChatGPT. The purpose of this gradual release is to study the real-world use of the plugins, assess their impact, and identify any safety and alignment challenges that may arise. Getting these aspects right is essential for OpenAI to achieve its mission.

Since the launch of ChatGPT, users have expressed interest in having access to plugins, and many developers are experimenting with similar ideas. OpenAI is initially rolling out plugins to a small set of users and plans to gradually increase access as it learns more. After an alpha period, API users who want to integrate plugins into their products will also have access. OpenAI is excited to build a community that will help shape the future of the human-AI interaction paradigm.

Developers who have been invited off the waitlist can use OpenAI's documentation to build a plugin for ChatGPT. Once the plugin is created, it is listed in the prompt shown to the language model, along with documentation that instructs the model on how to use each one. The first set of plugins has been created by Expedia, FiscalNote, Instacart, KAYAK, Klarna, Milo, OpenTable, Shopify, Slack, Speak, Wolfram, and Zapier.