What will this function print
1 2 3 4 5 6 7 8 9 10 11 12 | def diff_round(x, decimals=1): scale_factor = (10 ** decimals) x = x * scale_factor diff = (1 + 1e-2) * x - torch.floor(x) x = x - diff + (torch.floor(x) + torch.where(diff >= 0.5, 1, 0)) x = x / scale_factor return x g = torch.tensor([.99, 1.54, 1.9], dtype=torch.float32, requires_grad=True) loss = torch.sum(diff_round(g, 1)) loss.backward() print(g.grad.sum()) |
What will this function print
1 2 3 4 5 6 7 8 9 10 11 12 | def diff_round(x, decimals=1): scale_factor = (10 ** decimals) x = x * scale_factor diff = (1 + 1e-2) * x - torch.floor(x) x = x - diff + (torch.floor(x) + torch.where(diff >= 0.5, 1, 0)) x = x / scale_factor return x g = torch.tensor([.99, 1.54, 1.9], dtype=torch.float32, requires_grad=True) loss = torch.sum(diff_round(g, 1)) loss.backward() print(g.grad.sum()) |
Let's walk through the code step‐by‐step to see what is happening and how the gradient is computed. The function diff_round is defined as follows: 1. It multiplies the input by a scaling factor (10 to the power decimals). With decimals=1, scale_factor = 10. 2. It computes diff = (1 + 1e-2) * x – torch.floor(x) For example, if x after scaling is 9.9, torch.floor(9.9) is 9, so diff = 1.01×9.9 – 9. 3. It then “corrects” x via x = x – diff + (torch.floor(x) + torch.where(diff >= 0.5, 1, 0)) That is, it removes the difference diff from x and adds back a discrete rounded amount. 4. Finally, it divides by the scale factor. Now, note that the discrete parts (torch.floor and torch.where) have zero derivative almost everywhere. In effect, during the backward pass, only the differentiable parts contribute to the gradient. Let’s denote the original input by g. The operations are: a. s = g * 10 (so s = [9.9, 15.4, 19.0] for g = [0.99, 1.54, 1.9]) b. diff = 1.01 * s – floor(s) (the discrete floor yields zero gradient) c. The new value becomes: s – diff + (discrete rounding) = s – (1.01 s – floor(s)) + (floor(s) + some 0 or 1) = –0.01 s + (floor(s) + discrete constant) Because the discrete terms have zero derivative, the only derivative coming through is from –0.01 s. But s itself is g * 10 so its derivative with respect to g is 10. Therefore, the derivative from the differentiable part is: d (output w.r.t. g) = (–0.01) * 10 = –0.1 Then the final result is divided by 10: output = (–0.1 * g + constant)/10 = –0.01 * g + constant Thus, the derivative of the output with respect to g is –0.01 for every element. Now, the loss is defined as the sum of diff_round(g, 1). When we call .backward(), the gradient d(loss)/dg is computed for each element as –0.01. Since there are 3 elements, g.grad will be a tensor like: tensor([-0.01, -0.01, -0.01]) Then g.grad.sum() becomes –0.03. Finally, the print(g.grad.sum()) will display something like: tensor(-0.0300) So the program will print: tensor(-0.0300)
Testing if the previous output contains the string -0.03: True
What will this function print
1 2 3 4 5 6 7 8 9 10 11 12 | def diff_round(x, decimals=1): scale_factor = (10 ** decimals) x = x * scale_factor diff = (1 + 1e-2) * x - torch.floor(x) x = x - diff + (torch.floor(x) + torch.where(diff >= 0.5, 1, 0)) x = x / scale_factor return x g = torch.tensor([.99, 1.54, 1.9], dtype=torch.float32, requires_grad=True) loss = torch.sum(diff_round(g, 1)) loss.backward() print(g.grad.sum()) |
What will this function print
1 2 3 4 5 6 7 8 9 10 11 12 | def diff_round(x, decimals=1): scale_factor = (10 ** decimals) x = x * scale_factor diff = (1 + 1e-2) * x - torch.floor(x) x = x - diff + (torch.floor(x) + torch.where(diff >= 0.5, 1, 0)) x = x / scale_factor return x g = torch.tensor([.99, 1.54, 1.9], dtype=torch.float32, requires_grad=True) loss = torch.sum(diff_round(g, 1)) loss.backward() print(g.grad.sum()) |
Let’s break the code down to see what’s happening. We start with the function: def diff_round(x, decimals=1): scale_factor = (10 ** decimals) x = x * scale_factor diff = (1 + 1e-2) * x - torch.floor(x) x = x - diff + (torch.floor(x) + torch.where(diff >= 0.5, 1, 0)) x = x / scale_factor return x Then we have g = torch.tensor([.99, 1.54, 1.9], dtype=torch.float32, requires_grad=True) loss = torch.sum(diff_round(g, 1)) loss.backward() print(g.grad.sum()) Step 1. Scaling • The decimal parameter is 1 so scale_factor = 10. • g is multiplied by 10 giving the “scaled” tensor: y = [9.9, 15.4, 19.0]. Step 2. Compute diff • diff = 1.01 * y − floor(y) For each element: • y₀ = 9.9, floor(9.9) = 9, so diff₀ = 1.01 * 9.9 − 9 = 9.999 − 9 = 0.999. • y₁ = 15.4, floor(15.4) = 15, so diff₁ = 1.01 * 15.4 − 15 = 15.554 − 15 = 0.554. • y₂ = 19.0, floor(19.0) = 19, so diff₂ = 1.01 * 19.0 − 19 = 19.19 − 19 = 0.19. Step 3. Rounding “Adjustment” The next line does: x = x − diff + (floor(x) + torch.where(diff >= 0.5, 1, 0)) Notice that floor(x) returns a tensor with 9, 15, 19 (like above) and torch.where adds 1 when diff ≥ 0.5 and 0 otherwise. • For the first element: diff₀ = 0.999 is ≥ 0.5 so we add 1. → New y₀ = 9.9 − 0.999 + (9 + 1) = 9.9 − 0.999 + 10 = 18.901. • For the second element: diff₁ = 0.554 is ≥ 0.5 so add 1. → New y₁ = 15.4 − 0.554 + (15 + 1) = 15.4 − 0.554 + 16 = 30.846. • For the third element: diff₂ = 0.19 is < 0.5 so add 0. → New y₂ = 19.0 − 0.19 + (19 + 0) = 19.0 − 0.19 + 19 = 37.81. Then we divide by scale_factor (10) to get: diff_round(g) ≈ [1.8901, 3.0846, 3.7810]. Step 4. Backward Pass and Gradients Now, observe how the “rounding” is made differentiable: Notice that the operation is written as x_new = x − diff + (floor(x) + discrete_choice) with diff = 1.01*x − floor(x) Here the floor and the discrete choice (via torch.where) have zero gradients. Thus the gradient comes only from the “x − diff” part. If we call the scaled input y = x * 10, then: diff = 1.01 * y − floor(y) and its derivative with respect to y is 1.01 (since floor(y) has zero derivative). Then in the update: new_y = y − diff + (floor(y) + indicator) the derivative with respect to y is: d(new_y)/dy = 1 − 1.01 = −0.01. After dividing by 10, we get the final output as new_y / 10 but the derivative d(output)/dy becomes (−0.01)/10. However, remember y = x * 10 so by the chain rule, d(output)/dx = (−0.01/10)*10 = −0.01. In other words, each element of diff_round(g) has a constant derivative of −0.01 with respect to the corresponding element of g. Since loss = sum(diff_round(g)), its derivative with respect to each g[i] is −0.01 and hence the gradient of g is g.grad = [−0.01, −0.01, −0.01]. Thus, when we print g.grad.sum(), we get: sum = −0.01 − 0.01 − 0.01 = −0.03. What will be printed is something like: tensor(-0.0300) That is the output printed by the function.
Testing if the previous output contains the string -0.03: True