Like millions of users, I have been using Generative AI - especially Chat GPT - regularly to cope with my routine tasks. Have we considered what happens between giving a prompt and receiving the required response? What actually goes on behind the scenes?
The entire episode is nothing less than a journey and a fascinating one!
Let us review the various steps of this journey:
The moment we type a phrase (known as a prompt) and press the Enter key, the Tokeniser kicks in!
Now, what is a Tokenizer?
Think of it as a chef's chopping board knife! It chops the prompt into smaller parts - known as Tokens.
For example, if I give a prompt - "What is the capital of India?" - the Tokeneiser splits the phrase in the following format:
["What","is","the","capital","of","India","?"]
The tokenizer decides the most appropriate split based on its algorithm and vocabulary.
These chopped pieces are Tokens.
Then the Tokenizer refers or looks up in a pre-defined library known as the Vocabulary.
This library has a numeric reference for each token. Something like this:
"what" → 125
"is" → 864
" the" → 5421
"cpaital" → 13
"of" → 99
"India" → 199
"?" → 09
As soon as matches are found, the Tokenizer converts the tokens to corresponding numeric values - Token Ids - to be more specific.
Then comes the Embedding layer.
This layer converts all token IDs into a vector format.
Token: "India"
Embedding Vector (illustrative)
[0.83,-1.42, 0.91,2.13,-0.37,1.78,-0.55,0.24]
This list of numbers is called an embedding vector. It is the mathematical representation of the token "India."
The embedding vector is a learned numerical representation of the token that captures statistical relationships with other tokens
Transformer
The transformer can only understand vector-based inputs; it doesn't understand texts, unlike humans. It neither comprehends the token IDs. The Transformer then processes these vectors using its self-attention mechanism to understand the relationships between tokens, refine their representations through multiple layers, and predict the most appropriate next token. By repeating this prediction one token at a time, it gradually generates the complete response.
This response is again in the form of token Ids.
The predicted token IDs are then converted back into human-readable text by the tokenizer's decoding process, producing the response that appears on our screens.
The response is rightly termed as closure.
Contrary to popular belief, an LLM doesn't think of the complete answer first. It predicts one token at a time. After generating each token, it repeats the entire prediction process for the next one until the response is complete.
To us, an LLM appears to understand language. Under the hood, however, it is performing mathematics on vectors, predicting one token at a time until a coherent response emerges.
So, imagine millions of requests and responses going through the same pipeline daily!
