OpenAI announces Realtime API, prompt catching at DevDay 2024
OpenAI, a top player in the AI game, just dropped some cool tools at its 2024 DevDay event. The star of the show? The public beta of the "Realtime API," which can help developers whip up apps that deliver low-latency, AI-generated voice replies. It's somewhat similar to ChatGPT's Advanced Voice Mode in what it can do.
Company resilient amid leadership changes
Despite some big names leaving, OpenAI is still on the fast track. Kevin Weil, the Chief Product Officer, said that the exits of Chief Technology Officer Mira Murati and Chief Research Officer Bob McGrew won't slow them down. He praised what they brought to the table, and reiterated OpenAI's focus on growth during a chat with journalists ahead of the event.
Realtime API: A game-changer for developers
The Realtime API is about to shake up app development by allowing super-fast, speech-to-speech interactions. It comes with six unique voices from OpenAI, which are different from the ones you get with ChatGPT. They've limited the use of third-party voices to dodge any copyright headaches. Romain Huet, OpenAI's Head of Developer Experience, showed off what this new tool can do with a trip planning app and a food ordering example.
Vision fine-tuning and prompt caching
Alongside the Realtime API, OpenAI also rolled out vision fine-tuning in its API at DevDay. This cool feature lets developers use both images and text to fine-tune their GPT-4o apps, which could really boost performance for tasks that need visual understanding. Plus, they've introduced a prompt caching feature, letting developers save frequently used context between API calls. This means lower costs and faster response times.
OpenAI launches model distillation feature and evaluation tool
OpenAI has rolled out a model distillation feature, letting developers use larger AI models like o1-preview and GPT-4o to refine smaller ones like GPT-4o mini. This could boost the performance of these smaller models while also saving some bucks. As part of this initiative, OpenAI is also launching a beta evaluation tool for developers, to check out how well their fine-tuning is performing within OpenAI's API.
