Azure Nvidia GPUs power Bing’s faster, next generation chatbots

Nvidia takes chatbots a step closer to conversational AI

By Cliff Saran

A new artificial intelligence (AI)-powered speech recognition from Nvidia is set to power the voice assistant search in Microsoft Bing and a new generation of chatbots. The technology could lead to the development of chatbot systems that can respond more like a real human.

Nvidia claimed the new system could power chatbots that operate more realistically than existing AI systems. To achieve this, it said the new Nvidia platform has been optimised to run queries on vast datasets.

According to Nvidia, it is extremely difficult for chatbots, intelligent personal assistants and search engines to operate with human-level comprehension because of the inability to deploy extremely large AI models in real time. To overcome this limitation, Nvidia said it had added key optimisations to its AI platform, which it said could deliver complete AI inference in just over two milliseconds.

“We achieved two times the latency reduction and five times throughput improvement during inference using Azure Nvidia GPUs compared with a CPU-based platform, enabling Bing to offer a more relevant, cost-effective, real-time search experience for all our customers globally.”

Using Nvidia’s T4 GPUs running its TensorRT library, the new platform performed inference on the Bert-Base SQuAD dataset in only 2.2 milliseconds – which is under the 10-millisecond processing threshold for many real-time applications, said Nvidia.