Cerebras creates world's largest AI-ready processor with 850,000 cores
An American artificial intelligence (AI) centric company called Cerebras Systems claims to have built a single processor about the size of an iPad that can achieve the computational equivalent of the human brain or the equivalent of 100 trillion synapses. The company also claims that this Cerebras Wafer Scale Engine-2 (WSE-2) is the largest chip ever built. Here's everything you should know about it.
The processor is made up of 2.6 trillion transistors
Cerebras's WSE-2 is a single processor based on TSMC's seven-nanometer process technology that packs 2.6 trillion transistors to construct 850,000 AI-optimized cores installed on a colossal 46,225 square millimeter surface area. The chip itself has 40GB of onboard SRAM memory with a memory bandwidth of 20 petabytes/second to interface with 2.4 petabytes of off-chip memory for calculations.
Computer can process AI models with over 120 trillion parameters
This absolute unit of a processor is housed inside a custom computer called the Cerebras CS-2. It easily occupies an entire server rack. The processor can make short work of complex AI models with north of 120 trillion parameters. For the uninitiated, parameters are blocks of machine learning algorithms that encapsulate what the machine has learned from the data used to train it.
Cerebras CEO Andrew Feldman's explanation of the WSE-2's significance
Explaining the processor's significance to VentureBeat, Cerebras CEO and founder Andrew Feldman said, "Larger networks, such as (Microsoft's) GPT-3, have already transformed the natural language processing (NLP) landscape, making possible what was previously unimaginable". "The industry is moving past... trillion-parameter models, and we are extending that boundary by two orders of magnitude, enabling brain-scale neural networks with 120 trillion parameters," he added.
Giant processor is made out of a single silicon ingot
Hundreds of conventional desktop computing processors are cut out of a 12-inch diameter ingot of silicon at chip foundries such as TSMC. The new WSE-2 processor is one processor made out of a single ingot. This is more efficient than effectively distributing a computational workload among several smaller, conventional workstation processors. Matters are helped by Cerebras's new software-side technological innovations.
Cerebras's associated technologies help WSE-2 perform optimally
Weight Streaming, a software execution architecture, helps researchers stream AI model parameters to the chip from its connected off-die memory sans latency and memory bandwidth issues. This allows researchers to use anywhere between one and 192 CS-2 computers seamlessly. This ties into Cerebras's MemoryX technology that helps the 2.4 petabytes of off-chip storage stream data to the processor efficiently.
However, each Cerebras CS-2 can't be upscaled easily
Tom's Hardware noted that while researchers can run several CS-2 computers together, upscaling the capabilities of an existing CS-2 computer is nigh impossible. This is because the mammoth processor draws 15kW of power from a custom power supply. It also calls for custom cooling solutions, thus making it challenging to cram another wafer-sized chip into a single computer.
Cerebras' WSE-2 can compute as much as a human brain
Cerebras was founded in 2016 and employs 350+ people. In comparison to the Cerebras chip, Google trained a 1.6 trillion parameter model in January. Feldman's Cerebras system can handle 1,000 times more data with 1,000 times more power than Google's system. While the WSE-2 can handle human-like 100 trillion synapses, in 2020, Intel crammed 768 Loihi "brain chips" to rival a mole rat's brain.
