Parameters are one of the most common ways to measure a large language model’s (LLM) performance. When OpenAI upgraded GPT 3.5 to GPT-4, one of the most significant talking points was how the latter allegedly offered 1.7 trillion parameters compared to GPT 3.5’s 175 billion.
But what are artificial intelligence (AI) parameters exactly? And what difference do they make to a language model’s capabilities? Below, we will break down some of the most common FAQs surrounding these vital components.
What is a Parameter?
In its simplest terms, a parameter is a value that determines the behavior of a machine learning model, an algorithm designed to identify patterns in a dataset and make predictions based on that input.
Within the confines of a machine learning model, each parameter acts as a variable, which determines how the model will process and convert an input into an output.
This means that, in general, the more parameters a model has, the better it can capture details in a data set and the better its overall performance, particularly in tasks like text generation and responding to user questions.
What Types of Parameters Are There?
It’s worth noting that there are two main types of parameters in machine-learning models: parameters and hyperparameters. Often, these terms are used interchangeably, but they each denote different types of variables.
Parameters are variables that learn their own values from a dataset. These parameters are updated by the ML algorithm throughout the training process. The training process continues until the parameters find their optimal values.
On the other hand, hyper-parameters are variables defined by a human user, which determine how an ML model is trained. Hyper-parameters are used to determine the optimal values of the parameters outlined above.
Hyper-parameter values need to be specified by the developer before the model undergoes training and will remain fixed throughout the training process.
What’s the Benefit of Having More Parameters?
At a high level, the more parameters a model has, the more data it can process and the better it can summarize or translate text and respond to user questions.
However, it’s worth mentioning that having more parameters doesn’t necessarily make an AI model or LLM better. Other factors like faults in the training data and the types of techniques used to process it can also determine performance.
As an OpenAI study called Scaling Laws for Neural Language Models highlights, there is a diminishing point of returns: “Performance improves predictable as long as we scale up N [number of model parameters] and D [the size of the dataset[ in tandem, but enters a regime of diminishing returns if either N or D is held fixed while the other increases.”
So, while having more parameters may be a positive for many models, it is only a benefit to performance if the training data size and the amount of computing used for training also increases.
In any case, at a certain point, having too many parameters may be undesirable due to the higher computational requirements it needs to run, but also if it falls into the pitfall of overfitting.
What is Overfitting?
Overfitting is when a model has too many parameters, which are tethered to a particular set of training data, and can’t make accurate predictions based on a new dataset.
To avoid this scenario, AI vendors often need to provide sufficient parameters to offer a balance between generalization and specialization. This way, a model has enough parameters to make inferences from a dataset but doesn’t fall into the trap of overfitting to a particular dataset.
Can Models with Less Parameters Compete with Larger Models?
Yes, in certain tasks. Microsoft’s newly released small language model, Orca 2, has just 13 billion parameters but can perform similarly to or better than models 5-10 times larger on certain tasks.
It does this through the use of synthetic training data, which teaches Orca 2 reasoning techniques it can use to process tasks more effectively.
This highlights that you don’t necessarily need to have more parameters to outperform or remain competitive with another model.
In addition, smaller language models also have the advantage of requiring less computational power to run. This can make low-parameter models the more cost-effective option to run in certain scenarios.
As a result, it’s worth considering a model’s parameters alongside the type of data it is trained on, what techniques the vendor used, and the overall cost to run it.
