AutoGrad - Algorithmic Differentiation in C

Sat, 28 Dec 2024 00:00:00 +0000

Background

Let’s write an auto differentiation algorithm. We will take mathematical expressions in prefix notation, build an expression tree like the one below for

\[ \begin{align} 1338 * a + \frac{y^5}{x^{1335}} \end{align} \]

// expression in prefix notation
Expr e = ADD_EXPR(
 MUL_EXPR(INT_CONSTANT_EXPR(1338), VAR_EXPR("a", 1)),
 DIV_EXPR(VAR_EXPR("y", 5), VAR_EXPR("x", 1337))
);

graph TD
 node0["add"]
 node1["mul"]
 node2["1338"]
 node3["$a^{1}$"]
 node4["div"]
 node5["$y^{5}$"]
 node6["$x^{1337}$"]

 node0 --> node1
 node0 --> node4
 node1 --> node2
 node1 --> node3
 node4 --> node5
 node4 --> node6

and then write an algorithm to traverse the expression tree, taking derivatives of each sub-expression recursively and then generating a new expression tree that represents the derivative of the original expression tree, like the one below by just doing ExprGrad(&e)

Machine-Learning on Siddharth Mishra

AutoGrad - Algorithmic Differentiation in C

Background