Network-Attached Accelerator
Network-attached accelerators, such as graphics processing units (GPUs), field-programmable gate arrays (FPGAs) and application-specific integrated circuits (ASICs), are specialized computing devices used to improve the performance of networked systems. By connecting an accelerator directly to a computer or server’s internal network, it can be accessed by multiple applications, allowing them to use its powerful capabilities without needing to purchase additional hardware. This reduces costs and improves the efficiency of operations, making it ideal for both small businesses and large enterprises. Additionally, network-attached accelerators can provide increased speed and scalability when compared with traditional CPU-based solutions. With their enhanced performance and cost savings, they have become increasingly popular in many industries. As a result, more companies are turning to network-attached accelerators to gain the competitive edge they need in today’s market.
A Cerebras Systems CS-1 and CS-2 are network-attached accelerators coexisting with a CPU cluster that consists of one chief node and one or more worker nodes. In order to maximize the full computing power of the wafer scale engine, we highly recommend using multiple workers to send in data so that there is enough data to keep the wafer at full utilization. This means PyTorch training code would need to be adapted to enable multi-worker coordination, which is easily done through the context manager – cbtorch.Session that wraps around the “training” portion of the code. This allows multiple workers to call the exact same training script, but only the chief worker is given control of the training loop while the other workers are solely designated to sending data from the dataloader to the system.
