How was GPT-4 created and what are its limitations
OpenAI has cemented its place as the frontrunner in the AI race with the introduction of GPT-4 (Generative Pre-trained Transformer-4), its new large language model (LLM). Unlike GPT-3.5, its text-only predecessor, GPT-4 has multimodal capabilities. It can accept both texts and images as inputs. A close look at how GPT-4 was created will give us a better idea about OpenAI's "most advanced system."
GPT-4 is trained on a web-scale corpus of data
GPT-4, like its predecessors GPT, GPT-2, and GPT-3, is trained using data available on the internet. The data includes publicly available information and data licensed by OpenAI. According to the company, GPT-4 is created with more knowledge and more computation. Its 'food' is a "web-scale corpus of data" that includes varying ideas and ideologies, contradictory and consistent statements, and weak and strong reasoning.
OpenAI fine-tuned GPT-4's behavior using 'reinforcement learning with human feedback'
The amount of variable data the model is trained on means it can respond in different ways to a question. At times, the response could be entirely different from what the user intended. To address this, OpenAI used 'reinforcement learning with human feedback' (RLHF). With RLHF, the company fine-tuned GPT-4's behavior to align with the user's intent.
Over 50 human experts refined the AI's performance
OpenAI used more human feedback than before to improve GPT-4's behavior. This includes feedback from ChatGPT users and over 50 experts in AI safety and security. To train AI models with human feedback, OpenAI uses an algorithm developed in collaboration with DeepMind. The training process involves a three-step feedback cycle between the human, the AI agent, and the RL algorithm.
GPT-4's additional capabilities also increased risks posed by the model
Despite GPT-4's capabilities, it poses the same risks as previous models, including harmful advice and inaccurate information, among others. However, the model's new capabilities mean it comes with some additional risks. To address this, OpenAI incorporated an additional safety reward signal during the RLHF training. Due to the mitigations, GPT-4's tendency to respond to disallowed content was reduced by 82%.
GPT-4 suffers from social biases and hallucinations like predecessors
GPT-4 is OpenAI's most advanced AI system yet. However, it suffers from the same limitations as ChatGPT or the previous models. Users will come across social biases, hallucinations, and adversarial prompts while dealing with GPT-4.
