Dr Rohin Y, Founder-CEO, LightSpeed Photonics
The term AI has been used to denote the large language models and generative AI which demand some of the most resource intensive computing hardware formerly restricted to high performance computing (HPC)/ super computing systems.
However, the training of these compute intensive systems is already pushing the boundary of knowing and extracting all the knowledge of humans. These are the systems that are massively constructed by technology giants such as Google/OpenAI/Meta/Microsoft as hyperscale. The narrative is slowly moving to the fact that his consumers utilize this information to make inference queries. That is the Edge AI which has been taking the intelligence nearer to the devices of the end user.
The transfer of the tokenized content of the trained data takes high bandwidth highways in order to share it between the computing systems. This is equal to GPUs or Edge computing chips communicating data over the fastest data transfer medium available – optical/Photonic Interconnects in which copper can no longer support it. These edge system requirements are quite unlike the ones being cool in a hyperscale/data centre-controlled environment. These edge systems may be located anywhere between a factory floor, and almost at the roof top 6G antenna.
Conventional computing platforms like GPUs cannot be taken seriously as edge applications because they use too much power and extreme limitations on power usage and form factor. Consider a robot in a factory setting that is attempting to measure and handle an object in 3D line a human can, or a (6G) connected autonomous driving vehicle that must make billions of calculations based on the LiDAR as well as a whole array of sensors that require decisions to be made in split seconds. This edge hardware must be not only very low latency in computing, but be compact and operate on comparatively very low power and also can operate in austere environments.
This is one of the reasons why you require a different kind of hardware in such edge situations each with its own requirements. The new movie being streamed requires a secure OTT platform that has a fast storage / caching and high optical connectivity as opposed to a high speed trading application that needs a high speed real-time encryption of high bandwidth connection with ultra-low latency.
This exposes the youthful startups to an array of opportunities in order to solve specific application opportunities. Although Nokia, Huawei and hardware vendors prepare to support telecom ecosystem with gear to serve these near-edge “cloudlets” there are numerous applications that need/would make use of quantum encryption hardware, optical computers or simply low-power special-purpose ASICs with custom silicon and connected over next-generation interconnect architectures. An example, a Deep-tech startup (LightSpeed Photonics) is developing compact, low-power, next generation Near-packaged/co-packaged by optical interconnects which are generic enough to be utilized in both datacentre-controlled environment as well as a range of edge circumstances. We shall also experience a trend of significant power and efficiency in the consumer devices and the edge scenarios but it needs micro data centres/near-edge clouds as it is cost-efficient in power to shift compute-intensive functions to dedicated and low-latency connected on-prem/near telecom equipment.
There is one significant theme that exists with AI. Hardware – there is money to be made by hardware – and those with moat in terms of the IP/know-how and production capacity to support growing demands will certainly win big! The large bodies of software might not be able to profit off their AI algorithms as the next generation of open-source versions are quickly replacing the proprietary systems, however, in hardware advancements the future of AI is here.
Through these significant architectural developments in the manner that AI will be used, it is indeed intriguing to observe how the edge AI hardware sit squarely at the table as AI usage is focused upon. With the maturity of AI infrastructure, data transportation is starting to be as significant as the computation itself. Hyperscale training of large models is not the only step in intelligence lifecycle. The second and more and more dominant phase is inference, or trained models running on thousands or millions of distributed systems. This paradigm fundamentally changes the computing stack architecture and makes the industry reconsider the operation of compute, storage, and networking in the non-centralized data centre environments.
Compared to a centralized cloud infrastructure, edge deployments use distributed instances of compute, which are closer to the data generation source. The persistently growing streams of data generated by cameras, robots, vehicles, drones, telecom towers, and industrial sensors require processing, which is expected to be carried out in real time. It is costly, inefficient and in many cases impractical to transmit all this raw information all the way back to the cloud because of latency, privacy and bandwidth issues. The delays are even more than those introduced by the physics of distance in real-world applications where milliseconds could mean the difference between safety, reliability or financial results.
It is here that high bandwidth connection is established as something of a defining factor. The copper-based electrical interconnects have served computing over the decades with their place in the fundamental physical limits. Signal integrity decreases with the distance, power consumption increases exponentially with the speed and electromagnetic interference is more of a problem. Optical interconnects offer a way forward as AI workloads may require terabytes per second of data movement and thus support much greater bandwidth per link, less power per transmitted bit, longer ranges of transmission without signal degradation, and less thermal dissipation within the computer systems. In the case of distributed AI systems that are not run in climate-control environments, the benefits are not optional, but obligatory.
Micro data centers located in the proximity of telecom towers, factories, or enterprise campuses have to be small in thermal and power envelope. Optical connectivity allows these distributed compute nodes to be connected both to each other and to centralized cloud infrastructure in a manner which electrical interconnects alone cannot scale to support. Consequently, bandwidth is quickly becoming the new currency of AI infrastructure and data movement efficiency is becoming as useful as data computing.
The essence of this change is even more evident when considering robotics and autonomous systems. These platforms have to view the world, explain it, and take action in the form of milliseconds. There is a very thin margin of error and the cost of latency can be disastrous. The autonomous car, in particular, should combine information provided by cameras, radar, LiDAR, ultrasonic, GPS, and inertial measurement. Gigabytes of sensor data need to be processed every second in order to find pedestrians, determine trajectories, and take navigational decisions. It is not good to wait tens of milliseconds to get responses of a cloud. There has to be local and immediate decisions.
The same can be said of robotics in manufacturing. Machines are required to detect faultiness, to adjust to material changes and to work in a safe manner with the human employees. This demands continuous perception and immediate feedback loops which is only possible with highly optimized edge computing hardware. The transformative benefits of optical technologies in such settings are that optical signals have the capacity to support multiple wavelengths concurrently, which provides the opportunity to support mass parallelism. Reduced power consumption directly corresponds to reduced heat which is important when a small robotic system is required to work in a limited area. Electromagnetic interference also does not affect optical links, which is why they are very reliable in the industrial setting where electrical noise is prevalent.
The other area edge AI hardware will have a decisive role is in cybersecurity. The attack surface is increasing exponentially as more and more infrastructure becomes interconnected. Real time threat detection and response is no longer a luxury, but a need. The conventional methods of cybersecurity are deeply rooted in the centralized monitoring and analysis. Nevertheless, the threats nowadays are dynamic and may spread through networks within a few seconds. The damage can be completed before data is sent to a central server to be analyzed.
Threat detection is becoming a newer paradigm, distributed, real-time. Edge nodes which are provided with special AI accelerators can constantly observe the network traffic, identify anomalies and react immediately. These are real time encryption and decryption, detection of behavioral anomalies, intrusion prevention and safe caching and distribution of data. These operations need ultra-low latency and high throughput processing, and thus the advantages of specialized edge AI hardware perfectly match these needs. This can be further improved by using optical interconnects which allow high rate and secure communication among the distributed security nodes. With the digital infrastructure increasing, the hardware-accelerated intelligence at the edge will keep becoming a fundamental of cybersecurity.
The hardware of AI has developed fast and revolutionized itself. The introduction of GPUs into AI training transformed the way it is trained, since it has the ability to do it in a massively parallel fashion. Nonetheless, the upcoming step of AI implementation will require more energy efficient, smaller and more detailed hardware, and inference-specific approach. This has seen the development of inference-specific AI accelerators in the form of ASICs and NPUs. These chips are more efficient in the performance per watt than the general-purpose GPUs and can be programmed to do certain tasks such as vision processing or speech recognition.
The next accelerators are the developing photonic computing. Photonic chips involve the utilization of light as opposed to electricity to execute some computation. Since photons are faster than electrons and Photonic systems produce less heat, they can be extraordinarily efficient and throughput. Photonic computing is yet to reach the stage at which it can be used in high-speed matrix multiplication, signal processing, neural network acceleration, and high-speed movement of data. With the convergence of electronic and photonic technology, hybrid systems will establish new limits of performance and efficiency of AI hardware.
AI infrastructure is not a central system any more. It is instead becoming a stratified ecosystem that is comprised of intertwined layers. Large-scale model learning and worldwide coordination still remain the hub of hyperscale cloud environments. The near edge is the cloud-end devices connection, which consists of micro data centers located close to the telecom infrastructure to ensure access to AI services with low latency. On the far extreme is the consumer device, vehicles, sensors and robots that do real-time inference locally. These layers are connected with the help of optical connectivity that allows smooth integration throughout the computing hierarchy.
Edge AI is not the miniaturized form of cloud AI. It demands a radically new design of hardware. Edge systems need to be able to trade-off between performance, power efficiency, thermal requirements, reliability, physical size, and cost. Such a combination of needs cannot be fulfilled in the form of the reuse of the old data center hardware. In its place, a new paradigm of hardware is developing, one that is characterized by specialized AI accelerators, optical interconnects, photonic computing, secure hardware modules, and highly efficient memory architectures.
This change has gigantic innovations. In the past, semiconductor innovation has been preoccupied with large companies because the manufacturing of chips is capital-intensive. Nevertheless, AI usage is diversifying at a very fast pace and this leaves room to the agile start-ups to carve a niche. The application-specific AI accelerators, optical interconnect solutions, quantum-safe encryption hardware, robotics compute platforms, and edge data center infrastructure can be targeted by startups. The variety of applications on the edge does not allow a single company to control all the niches. In its place, it will be specialized innovators who solve very specific problems that will shape the ecosystem.
However, as much as software tends to capture imagination of the people, hardware is the foundation of technological revolutions. Hardware innovation has always led to the creation of massive economic value, starting with the personal computer, the smartphone, or any other device. The same is the case with AI. With the spread of open-source models of AI, software differentiation can decrease. Hardware, on the other hand, is hard to imitate because of the intellectual property barrier, the complexity of manufacturing, and its supply chain, and lengthy development cycle. Firms that achieve a high position in AI hardware will enjoy sustainable competitive advantages.
The coming decade will transform the way intelligence is being implemented and consumed. AI is not going to remain far away in remote data centres, it will be a part of devices and the environment around us. We will have smarter cities and infrastructure, complete autonomy in transportation systems, intelligent manufacturing environments, live human health monitoring, and devices that can understand consumers on a deeply personalized level. The epicenter of this change is the edge AI hardware a compact, efficient, and heavily intimately embedded in the physical world.
The convergence of a distributed intelligence is one of the most important technological shifts of our era as compared to the centralized computing. With this transformation being experienced, optical connectivity and specialized hardware will become a defining feature of the future of AI. The era of edge intelligence is here to stay and its influence will be experienced in all industries and in all spheres of the modern world.
