This article gives a sense of how matrix operations such as matrix multiplication arise in machine learning.

Dense layers

Before we get started, there is a small bit of prerequisite knowledge about “dense layers” of neural networks.

1. What is a “dense layer”? Don’t worry about how “dense layers” fit into the big picture of a neural network. All we need to know is that a dense layer can be thought of as a collection of “nodes”.

2. Well now we’ve begged the question: what is a “node”? A node can be thought of as a machine, associated with a set of numbers called weights, that is responsible for producing a numeric output in response to receiving some number of binary (i.e. 0 or 1) inputs. Specifically, if a node N is associated with n weights w₁, …, w_n, then, when n binary numbers x₁, …, x_n are given to N as input, N will compute the following “weighted sum” as output:

For example, if a node has weights 3, 4, 5 and is passed the binary inputs 1, 0, 1, then that node will return the weighted sum 3 * 1 + 4 * 0 + 5 * 1 = 8.

That’s it! We are done with covering prerequisites. We now know what a dense layer of a neural network is responsible for accomplishing: a dense layer contains nodes, which are associated with weights; the nodes use the weights to compute weighted sums when they receive binary inputs.

Matrices

Now, we see what happens when we consider all of the nodes in a dense layer producing their outputs at once.

Suppose L is a dense layer. Then L consists of m nodes, N₁, …, N_m for some whole number m, where the ith node N_i has n weights* associated with it, w_i1, …, w_in. When N_i receives n binary (0 or 1) inputs x₁, …, x_n, it computes

Since the above weighted sum is computed by the ith node, let’s refer to it as f(N_i):

The nodes N₁, …, N_m will need to compute the weighted sums f(N₁), …, f(N_m):

Revolutionize Your Business With Generative AI

From product design and software development to virtual agents, content creation, and reporting, GenAI is transforming business. Our AI experts help you unlock GenAI’s full potential and drive growth.

Let’s Get Started

The above can be rewritten with use of so-called column vectors:

A column vector is simply a list of numbers written out in a column. Note that in the above we have made use of the notions of “column vector addition” and “scaling of a column vector by a number”.

The above expression can be expressed as a matrix-vector product:

(The matrix-vector product Wx of the above is literally defined to be the expression on the right side of the previous equation).

This matrix-vector product notation gives us a succinct way to determine what each node in a layer does to inputs x₁, …, x_n.

Generalization

The above approach generalizes in the following way. Assume that instead of n inputs x₁, …, x_n, we have p groups of inputs, where the ith group has inputs x_1i, …, x_ni. Additionally, define x_i := (x_1i, …, x_ni) and let f(N_j, x_i) denote the result of node N_j acting on x_1i, …, x_ni. Then we have

This is equivalent to

Further generalization

Now, assume that in addition to having multiple groups of inputs x₁, …, x_p, we want to pass these groups of inputs through multiple layers multiple layers L₁, …, L_l. If layer L_i has weight matrix w_i, then the result of passing x₁, …, x_p through L₁, …, L_l is the following product of matrices:

* You may object and remark that the number of weights associated with a given node depends on the node; in other words, that node each node N_i has n_i weights associated with it, not n weights. We are justified in assuming that n_i = n for all i because we can always associate extra weights of 0 to nodes that don’t already have the requisite n weights associated with them.