Weight pruning is a technique used to reduce the size of a neural network by removing certain weights, typically those with small magnitudes, without significantly affecting the model’s performance. The idea is to identify and eliminate connections in the network that contribute less to the overall computation. This process helps in reducing the memory footprint and computational requirements during both training and inference.

Initial Model: Let’s consider a simple fully connected neural network with one hidden layer. The architecture might look like this:
Input layer (features) -> Hidden layer -> Output layer (predictions)

Training: The network is trained on a dataset to learn the mapping from inputs to outputs. During training, weights are adjusted through optimization algorithms like gradient descent to minimize the loss function.

Pruning: After training, weight pruning involves identifying and removing certain weights. A common criterion for pruning is to set a threshold, and weights whose absolute values fall below this threshold are pruned. For example, let’s say we have a weight matrix connecting the input layer to the hidden layer:

If we set a pruning threshold of 0.2, weights smaller than 0.2 in absolute value may be pruned. After pruning, the weight matrix might become:

Here, the connections with weights below 0.2 are pruned by setting those weights to zero.

Fine-tuning: Optionally, the pruned model can undergo fine-tuning to recover any loss in accuracy caused by pruning. During fine-tuning, the remaining weights may be adjusted to compensate for the pruned connections.

Weight pruning is an effective method for model compression, reducing the number of parameters in a neural network and making it more efficient for deployment in resource-constrained environments.

If you have any doubt/suggestion please feel free to ask and I will do my best to help or improve myself. Good-bye until next time.