Google BARD AI based on LaMDA – How It Learned & Trained Bard from Free Web

Recently, there has been a significant increase in the use of language models for various natural language processing (NLP) tasks, including dialog systems. One of the key components of training effective language models for dialog applications is the quality of the pre-training data used to train the model.

To address this challenge, researchers have introduced LaMDA (Language Models for Dialog Applications), a language model specifically designed for dialog applications. One of the key features of LaMDA is its pre-training data, known as Infiniset.

Infiniset is a large dataset that is a combination of dialog data from public dialog data and other public web documents. The dataset consists of 2.97 billion documents, 1.12 billion dialogs, and 13.39 billion utterances.

The composition of the data is as follows:

• 50% dialogs data from public forums.
• 12.5% C4 data, 12.5% code documents from sites related to programming such as Q&A sites, tutorials, etc.
• 12.5% English Wikipedia articles.
• 6.25% English web documents.
• 6.25% non-English web documents.

The total number of words in the dataset is 1.56 trillion.

The choice of this composition was made with the goal of achieving a more robust performance on dialog tasks while still retaining the model’s ability to perform other NLP tasks such as code generation. This composition of pre-training data is a crucial factor in the performance of LaMDA, as it allows the model to understand and generate coherent and contextually appropriate responses in dialog systems.

However, it is worth noting that the choice of pre-training data composition may have an effect on the quality of other NLP tasks performed by the model. As future work, researchers can study how the choice of pre-training data composition affects the quality of other NLP tasks performed by the model.

In conclusion, LaMDA is a promising language model designed specifically for dialog applications. The quality of its pre-training data, Infiniset, is a key factor in its performance, and the choice of pre-training data composition is an important consideration for future research.