COCKPIT: A Practical Debugging Tool for Training Deep Neural Networks

Frank Schneider*

University of TübingenMaria-von-Linden-Straße 6

Tübingen, Germanyfschneider@tue.mpg.de

Felix Dangel∗

University of TübingenMaria-von-Linden-Straße 6

Tübingen, Germanyfdangel@tue.mpg.de

Philipp HennigUniversity of Tübingen &

MPI for Intelligent SystemsTübingen, Germany

ph@tue.mpg.de

Abstract

When engineers train deep learning models, they are verymuch “flying blind”. Commonly used approaches for real-time training diagnostics, such as monitoring the train/testloss, are limited. Assessing a network’s training processsolely through these performance indicators is akin to de-bugging software without access to internal states througha debugger. To address this, we present COCKPIT, a collec-tion of instruments that enable a closer look into the innerworkings of a learning machine, and a more informative andmeaningful status report for practitioners. It facilitates theidentification of learning phases and failure modes, like ill-chosen hyperparameters. These instruments leverage novelhigher-order information about the gradient distributionand curvature, which has only recently become efficientlyaccessible. We believe that such a debugging tool, whichwe open-source for PYTORCH, represents an important stepto improve troubleshooting the training process, reveal newinsights, and help develop novel methods and heuristics.

1. Introduction & motivationDeep learning represents a new programming paradigm:

instead of deterministic programs, users design models and“simply” train them with data. In this metaphor, deep learn-ing is a meta-programming form, where coding is replacedby training. Here, we ponder the question what a (better)debugger for this process should look like.

Debuggers are indispensable for traditional software de-velopment. When things fail unexpectedly, they give accessto the internal workings of the code, allowing a look “intothe box”. This is much more efficient than re-running theprogram with different inputs. And yet, the state-of-the-artin deep learning is arguably closer to the latter. If the attemptto train a deep net on a new data set fails, a machine learn-

*Equal contribution

ing engineer faces various options: Should they change thetraining hyperparameters (how?); the optimizer (to whichone?); the architecture (how?); or just re-run with a differ-ent seed? Machine learning toolboxes provide scant help toguide these decisions.

Of course, traditional debuggers can be applied to deeplearning. They will give access to every single weight of aneural network, or the individual pixels of its training data.But this rarely yields insights towards successful training.Extracting meaningful information requires a statistical ap-proach and distillation of the bewildering complexity into amanageable summary. Tools like TENSORBOARD [1] werebuilt in part to streamline this visualization. Yet, the quan-tities that are widely monitored (mainly train/test loss &accuracy), hardly provide useful feedback beyond a relativecomparison among multiple training runs. They do not showthe network’s internal state. Figure 1 illustrates how suchestablished learning curves can describe the current stateof the model – whether it is performing well or not – whilefailing to inform about training state and dynamics. Theytell the user that things are going well or badly, but not why.

The situation is similar to flying a plane by observingonly its altimeter. It is not surprising, then, that achievingstate-of-the-art performance in deep learning requires expertintuition, or plain trial and error.

Here, we try to enrich the deep learning pipeline with a vi-sual and statistical debugging tool. To this end, we proposenew observables, in addition to using several establishedones (Section 2). We leverage and augment recent exten-sions to automatic differentiation (in particular BACKPACK[11] for PYTORCH [29]) to efficiently access second-orderstatistical (e.g. gradient variances) and geometric (e.g. Hes-sian) information. We show how the proposed quantities canaid the deep learning software engineer in online tasks, likelearning rate selection, as well as detecting common prob-lems with data processing or model architectures (Section 3).

Concretely, we introduce COCKPIT, a flexible and effi-cient framework for monitoring these observables online dur-

1

arX

iv:2

102.

0660

4v1

[cs

.LG

] 1

2 Fe

b 20

21

0 4 8 12 16Iteration

0.7

0.8

0.9

1.0

1.1

Min

i-B

atch

Trai

ning

Los

s

Loss Curve

θ1

θ 2

Loss Landscape

−1.0 −0.5 0.0 0.5 1.0Normalized Step Length

0

1

Nor

mal

ized

Los

s

Alpha Distribution

Den

sity

Figure 1. Illustrative example: Learning curves do not tell the whole story. Two different optimization runs (—/—) can lead to virtuallythe same loss curve (left). However, the actual optimization trajectories (middle), exhibit vastly different behaviors. In practice, they areintractably large and cannot be visualized directly. Recommendable actions for both scenarios (increase/decrease the learning rate) cannot beinferred from the loss curve. The α- distribution, one COCKPIT instrument (right), not only clearly distinguishes the two scenarios, but alsoallows for taking decisions regarding how the learning rate should be adapted. See Section 3.3 for further details.

ing training in carefully designed plots we call “instruments”(see Figure 2). To be of practical use, such visualizationmust have manageable computational overhead. We pro-vide three different configurations of varying computationalcomplexity and demonstrate that their instruments keep thecomputational cost well below a factor of 2 in run time (Sec-tion 4). COCKPIT scales well to real-world deep learningproblems (see Figure 2 and Section 5). It is available at

https://github.com/f-dangel/cockpit,

open-source, extendable, and seamlessly integrates into con-ventional PYTORCH training loops.

2. COCKPIT’s instrumentsSetting: We consider supervised regression/classificationwith labeled data (x,y) ∈ X×Y generated by a distributionP (x,y). The training set D = {(xn,yn) | n = 1, . . . , N}consists of N i.i.d. samples from P and the deep modelf : Θ × X → Y maps inputs xn to predictions yn by pa-rameters θ ∈ RD. This prediction is evaluated by a lossfunction ` : Y × Y → R which compares to the labelyn. The goal is minimizing an inaccessible expected riskLP (θ) =

∫`(f(θ,x),y) dP (x,y) by empirical approx-

imation through LD(θ) = 1N

∑Nn=1 `(f(θ,xn),yn) :=

1N

∑Nn=1 `n(θ), which in practice though can only be

stochastically subsampled on mini-batches B ⊆ {1, . . . , N},

LB(θ) =1

|B|∑n∈B

`n(θ) . (1)

As is standard practice, we use first- and second-order infor-mation of the mini-batch loss, described by the mini-batchgradient as an average over individual gradients gn(θ),

gB(θ) =1

|B|∑n∈B∇θ`n(θ)︸ ︷︷ ︸gn(θ)

, (2)

and the mini-batch Hessian

HB(θ) =1

|B|∑n∈B∇2θ`n(θ) . (3)

Design choices: To minimize computational and designoverhead, we restrict the metrics to quantities that require noadditional model evaluations. This means that, at trainingstep t→ t+ 1 with mini-batches Bt,Bt+1 and parametersθt,θt+1, we may access information about the mini-batchlosses LBt

(θt) and LBt+1(θt+1), but no cross-terms that

would require additional forward passes.

Key point: LB(θ), gB(θ), and HB(θ) are just expectedvalues of a distribution over the batch. Only recently, thisdistribution has begun to attract attention [13] as its computa-tion has become more accessible [7, 11]. Contemporary op-timizers leverage only the mean gradient and neglect highermoments. One core point of our work is making extensiveuse of these distribution properties, trying to visualize themin various ways. This distinguishes COCKPIT from being“just a collection of plots” that could be built in tools likeTENSORBOARD. Leveraging these distributional quantities,we create instruments and show how they can help adapthyperparameters (Section 2.1), analyze the loss landscape(Section 2.2), and track network dynamics (Section 2.3). In-struments can sometimes be built from already-computedinformation or are efficient variants of previously proposedobservables. To keep the presentation concise, we highlightthe instruments shown in Figure 2 and listed in Table 1.Appendix B defines them formally and contains more exten-sions, such as the mean GSNR [24], the early stopping [25]and CABS [4] criterion, which can be used in COCKPIT.

2

Figure 2. Screenshot of COCKPIT’s full view while training the ALL-CNN-C [36] on CIFAR-100 with SGD using a cyclical learningrate schedule. Figure and labels are not meant to be legible, but serve to give an impression of how COCKPIT looks to the user. Gray panels(bottom row) show the information currently tracked by most practitioners. The individual instruments are discussed in Section 2, andobservations are described in Section 5. An animated version can be found in the accompanying GitHub repository.

2.1. Adapting hyperparameters

One of the biggest challenges in deep learning is set-ting the hyperparameters correctly. Currently, this choice ismostly done by trial and error through parameter searches.We aim to augment this process by providing a set of in-struments to inform the user about what effect the selectedhyperparameters have on the current training process.

Alpha: Are we crossing the valley? Using individualloss and gradient information at the start and end pointof each iteration, we build a noise-informed univariatequadratic approximation along the step direction (i.e. theloss as a function of the step-size), and assess to which pointon this parabola our optimizer moves. We standardize thisvalue α such that stepping to the valley-floor is assignedα = 0, the starting point is at α = −1 and updates to thepoint exactly opposite of the starting point have α = 1 (com-pare right panel of Figure 1). In Section 3.3 we demonstrateempirically that, counter-intuitively, it is generally not a goodidea to choose the learning rate such that α ≈ 0.

Figure 1 illustrates the scenarios α = ±1 and how moni-toring the α-distributions can help distinguish between twotraining runs with similar performance but distinct failuresources. By default, this COCKPIT instrument shows the

α-distribution for both the last 10 % of training as well asthe full training process (e.g. top left plot in Figure 2).

Distances: Are we making progress? Another way todiscern the trajectories in Figure 1 is by measuring the dis-tance to initialization in parameter space [28], and updatesize [2, 14]. Both are shown together in one COCKPIT in-strument (e.g. center left plot in Figure 2) and would besignificantly larger for the blue trajectory.

These distances are also able to disentangle phases in theblue trajectory. Using the same learning rate, it will continueto “jump back and forth” between the loss valley’s walls butat some point cease to make progress towards the minimum.During the initial phase of “surfing the wall”, distance frominitialization increases. Ultimately though, it will stagnate,while the update size remains non-zero, indicating diffusion.While the initial “surfing the wall”-phase benefits successfultraining (see Section 3.3), achieving stationarity may requireadaptation once the optimizer reaches that diffusion.

Gradient norm: How steep is the wall? The update sizewill show that the orange trajectory is stuck. But why? Suchslow-down can result from both a bad learning rate and fromloss landscape plateaus. The gradient norm (bottom left

3

Name Description Min. Config

Alpha Normalized step on a noisy quadratic interpolation between two iterates θt,θt+1 economyDistance Update size ‖θt+1 − θt‖ and distance from initialization ‖θt − θ0‖ economyGradNorm Mini-batch gradient norm ‖gB(θ)‖ economyNormTest Normalized fluctuations of the residual norms ‖gB − gn‖, proposed in [8] economyInnerTest Normalized fluctuations of the gn’s parallel components along gB, proposed in [6] economyOrthoTest Normalized fluctuations of the gn’s orthogonal components along gB, proposed in [6] economyGradHist1d Histogram of individual gradient elements, {gn(θj)}n∈B,j=1,...,D economyTICDiag Relation between (diagonal) curvature and gradient noise, inspired by [38] businessHessTrace Exact or approximate Hessian trace, Tr(HB(θ)), inspired by [44] businessHessMaxEV Maximum Hessian eigenvalue, λmax(HB(θ)), inspired by [44] fullGradHist2d Histogram of weights and individual gradient elements, {(θj , gn(θj))}n∈B,j=1,...,D full

Table 1. Overview of COCKPIT quantities. They range from cheap byproducts, to nonlinear transformations of first-order information andHessian-based measures. Some quantities have already been proposed, others are first to be considered in this work. They are categorizedinto configurations economy ≤ business ≤ full based on their run time overhead (see Section 4 for a detailed evaluation).

panel in Figure 2) distinguishes these two causes.

Gradient tests: How noisy is the batch? The batch sizetrades off gradient accuracy versus computational cost. Bol-lapragada et al. [6] and Byrd et al. [8] proposed adaptivesampling strategies based on testing geometric constraintsbetween mean and individual gradients. Their norm, innerproduct, and orthogonality tests use a standardized radiusand two band widths (parallel and orthogonal to the gradientmean) that indicate how strongly individual gradients scatteraround their mean. The original works use these values toadapt batch sizes. Instead, we use the standardized noiseradius and band widths for visualization. COCKPIT com-bines all three gradient tests into a single instrument (topmiddle plot of Figure 2), which also monitors their evolu-tion over time. These noise indicators can be used to guidebatch size adaptation on- and offline, or to investigate theinfluence of gradient alignment on optimization speed [33]and generalization performance [9, 10, 24].

2.2. Hessian properties for local loss geometry

An intuition for the loss landscape’s local geometry helpsin many ways. It can help diagnose whether training is stuck,to adapt the step size, and can explain optimization stabilityor regularization [16, 20]. The principal challenge is theextreme weight-space dimension. Low-dimensional projec-tions of high-dimensional surfaces can behave unintuitively[27], but tracking their most extreme or average behaviorsmay help in debugging, especially if first-order metrics fail.

Hessian eigenvalues: A gorge or a lake? In convex op-timization, the maximum Hessian eigenvalue crucially de-termines the appropriate learning rate [34]. Many workshave studied the Hessian spectrum in machine learning (e.g.,

[15, 16, 27, 31, 32, 44]). In short: curvature matters. Es-tablished [30] and recent advanced automatic differentiationframeworks [11] allow computing Hessian properties with-out computing the full matrix, and COCKPIT leverages thisfunctionality to provide access to the the Hessian’s largesteigenvalue and its trace (right top and middle plots in Fig-ure 2). While the former corresponds to the loss surface’ssharpest valley and can thus hint at optimization instabilities[20], a notion of “average curvature” is provided throughthe Hessian trace, since the eigenvalues λi relate to it by∑

i λi = Tr(HB(θ)).

TIC: How do curvature and gradient noise interact?There is an ongoing debate about curvature’s relationshipto generalization (e.g., [12, 19, 21, 38]). One recently pro-posed metric [38], the Takeuchi Information criterion (TIC)[37], estimates the generalization gap by a ratio betweenHessian and non-central second gradient moment. It alsoprovides intuition for changes in the objective function im-plied by gradient noise. Inspired by the approximations in[38], COCKPIT provides mini-batch TIC estimates duringtraining (bottom right plot of Figure 2).

2.3. Visualizing internal network dynamics

Histograms are a natural visual compression of the high-dimensional individual gradients (|B| ×D numbers). Theygive insights into the gradient distribution, and hence pro-vide a more detailed view of the learning signal, the gradientmean. In conjunction with the parameter associated to indi-vidual gradients, the entire model status (current parameterconfiguration and gradient distribution) and dynamics canbe visualized in a single plot, and monitored during train-ing. This provides a more fine-grained view of the modelcompared to tracking parameter and gradient norm [14].

4

Gradient and parameter-histograms: What is happen-ing in our network? COCKPIT offers a univariate his-togram of the gradient elements {gn(θ)j}n,j . Addition-ally, a combined histogram of parameter-gradient pairs{(θj , gn(θj)}n,j provides a two-dimensional look into thefrequencies of the network’s gradient and parameter valuesin a mini-batch. Both histograms are also able to show theirdata layer-wise. Section 3.1 shows an example use-caseof the gradient histogram; Section 3.2 makes the case forlayer-wise histograms.

3. ExperimentsThe diverse information provided by COCKPIT can help

users and researchers in many ways, some of which, just likefor a traditional debugger, only become apparent in practicaluse. In this section, we present a few motivating exampleapplications, selecting specific instruments and scenarios inwhich these instruments are practically useful. Specifically,we first show COCKPIT can help the user discern between,and thus fix, common training bugs (Sections 3.1 and 3.2)that are otherwise hard to distinguish as they lead to thesame failure: bad training. We demonstrate that COCKPITcan guide practitioners to choose efficient hyperparameterswithin a single training run (Sections 3.2 and 3.3). Finally,we highlight that COCKPIT’s instruments can provide re-search insights about the optimization process (Section 3.3).Our empirical findings are demonstrated on deep learningproblems from the DEEPOBS [35] benchmark collection.

3.1. Incorrectly scaled data

One prominent source of bugs is the data pipeline. To picka relatively simple example: For standard optimizers to workat the usual learning rates, inputs must be standardized (i.e.between zero and one, or have zero mean and unit variance[5, e.g.]). If the user forgets to do this, optimizer performancedegrades (by how much depends on the original data scaling).It can be difficult to identify the source of this problem as itdoes not cause obvious failures, NaN or Inf gradients, etc.

We now construct a semi-realistic example, to show howusing COCKPIT can help diagnose this problem upon ob-serving slow training performance. By default1, the popularimage data sets CIFAR-10 and CIFAR-100 [22] are pro-vided as NUMPY [18] arrays that consist of integers in theinterval [0, 255]. This raw data, instead of the widely usedversion with floats in [0, 1], changes the data scale by a fac-tor of 255 (and thus the training gradients). Therefore, theoptimizer’s optimal learning rate is scaled as well. In otherwords, the default parameters of popular optimization meth-ods may not work well anymore, or good hyperparametersmay take extreme values that are not discovered by a defaultparameter search.

1https://www.cs.toronto.edu/~kriz/cifar.html

(a)

100 104 108

−1

0

1

Gra

dien

tele

men

t

(b)

100 104 108

−1

0

1

Gra

dien

tele

men

t

Figure 3. Same inputs, different gradients. Catching data pro-cessing bugs with COCKPIT. (a) normalized ([0, 1]) and (b)raw ([0, 255]) images look identical in auto-scaled front-ends likeMATPLOTLIB’s imshow. The gradient distribution on the 3C3D

architecture, however, is crucially affected by this scaling.

Even if the user directly inspects the training images, thismay not be apparent (Figure 3). But the gradient histograminstrument of COCKPIT, which has a deliberate default plot-ting range around [−1, 1] to highlight such problems, imme-diately and prominently shows that there is an issue.

Of course, this particular data is only a placeholder forreal practical data sets. While this problem may not fre-quently arise in the highly pre-processed, packaged CIFAR-10, it is not a rare problem for practitioners who work withtheir personal data sets. This is particularly likely in domainsoutside standard computer vision, e.g. when working withmixed-variable data without obvious natural scale.

3.2. Vanishing gradients

The model architecture itself can be a source of trainingbugs. As before, such problems mostly arise with novel datasets, where well-working architectures are unknown. Thefollowing example shows how even small (in terms of code)architecture modifications may severely harm the training.

Figure 4 (a) shows the distribution of gradient values oftwo different network architectures in blue and orange. Al-though the blue network trains considerably better than theorange one, their gradient distributions look quite similar.The difference becomes evident when inspecting the gra-dient histogram layer-wise. We can see that many layershave a degenerated gradient distribution. Many elements arepractically zero (see Figure 4 (b), bottom row). Since thelast fully-connected layers have many more parameters (a

5

(a) (b)

100 102 104 106 108

−1

0

1

Gra

d.el

emen

t

Network

100 102 104 106 108

−1

0

1

Gra

d.el

emen

t

100 102 104

Parameter 0

100 103 106

Parameter 4

100 102 104

Parameter 10

100 102 104 100 103 106 100 102 104

Figure 4. Gradient distributionsof two similar architectures onthe same problem. (a) Distri-bution of individual gradient ele-ments summarized over the entirenetwork. Both seem similar. (b)Layer-wise histograms for a sub-set of layers. Parameter 0 is thelayer closest to the network’s input,parameter 10 closest to its output.Only the layer-wise view revealssignificant differences.

classic pattern many convolutional neural nets follow), theydominate the network-wide histogram. This obscures that aconsiderable network part is effectively unable to train.

Both the blue and orange networks follow DEEPOBS’s3C3D architecture. The only difference is the non-linearity:The blue network uses standard ReLU activations, whilethe orange one has sigmoid activations. Here, the layer-wise histogram instrument of COCKPIT highlights whichpart of the architecture makes training unnecessarily hard.Accessing information layer-wise is also essential due to thestrong overparameterization in deep models where trainingcan happen in small subspaces [17]. Once again, this is hardto do with common monitoring tools, such as the loss curve.

3.3. Tuning learning rates

Once the architecture is defined, the optimizer’s learningrate is the most important hyperparameter to tune. Gettingit right requires extensive hyperparameter searches at highresource costs. COCKPIT’s instruments can provide intuitionand information to streamline this process: In contrast tothe raw learning rate, the curvature-standardized step-sizeα-quantity (Section 2.1) has a natural scale.

Across multiple optimization problems, we observe, per-haps surprisingly, that the best runs and indeed all good runshave a mean α > 0 (Figure 5). This experiment illustrates afundamental difference between the stochastic optimizationtypical of contemporary machine learning and classic deter-ministic optimization tasks. Instead of locally stepping “tothe valley floor” (which is optimal in the deterministic case),stochastic optimizers should overshoot the valley somewhat.This necessity to “surf the walls” has been reported before(e.g. [42, 43]) as a property of neural network training.

Figure 5 also indicates a major challenge preventing sim-ple automated tuning solutions: The optimal α-value isproblem-dependent, and simpler problems, such as a multi-layer perceptron (MLP) on MNIST [23], behave more simi-larly to classic optimization problems. Algorithmic researchon small problems can thus produce misleading conclusions.

Learning rates are also frequently adapted during training.

−0.6 −0.4 −0.2 0.0 0.2 0.4 0.6Mean α

0.25

0.50

0.75

1.00

Test

Acc

urac

y

SVHN 3c3dMNIST MLP

F-MNIST MLPCIFAR-10 3c3d

Figure 5. Final test accuracy as a function of standardized step-size α. For multiple test problems, final test accuracy is shownagainst average α-value over the whole training period. Markersize indicates the magnitude of the raw learning rate used, markercolor identifies tasks (legend below). For each problem, the best-performing setting is highlighted by a vertical colored line.

This fits with the observation above about positive α-values.“Overshooting” allows fast early progression towards areasof lower loss, but it does of course not yield convergence inthe end. This kind of fine-tuning requires online, real-timevisualization of the training state, as is possible in COCKPIT.

Figure 5 shows that the α-gauge is not sufficient by itself:extreme overshooting with a too large learning rate leads topoor performance, which however can be prevented by tak-ing additional instruments into account. This makes the casefor the cockpit-metaphor of monitoring several instrumentsin conjunction. By combining the α-instrument with othergauges that capture the local geometry or network dynamics,the user can better identify good choices of the learning rateand other hyperparameters.

4. Benchmark

The previous section made a case for COCKPIT as aneffective debugging and tuning tool. To make the libraryuseful in a practical online setting, it must also have limitedcomputational cost. We now show that it is possible to com-pute all the necessary quantities at reasonable computationaloverhead. The user can control the absolute cost along two

6

Baseline Loss GradientNorm

Distance Norm Test InnerProduct Test

OrthogonalityTest

1DHistogram

Alpha Trace TIC0

1

2

3

Run

Tim

eO

verh

ead

Figure 6. Run time overhead for individualCOCKPIT instruments on a GPU. Run timeson CIFAR-10 3C3D are shown as multiples ofthe baseline (no tracking). Most instruments addlittle overhead. This plot shows the overheadfor computations in one iteration, determined byaveraging over multiple iterations and randomseeds. By reducing the tracking interval, the rela-tive overhead across the full training process canbe made orders of magnitude smaller (Figure 7).

dimensions, by reducing the number of instruments shown,or by reducing their update frequency.

Complexity analysis/expected performance: Comput-ing more information introduces computational overhead,of course. However, recent work [11] has demonstratedthat first-order information, like distributional statistics onthe batch gradients, can be computed on top of the meangradient at little extra cost. Similar savings apply for mostother quantities listed in Table 1, as they are (non-)lineartransformations of individual gradients.

A subset of COCKPIT quantities uses second-order infor-mation from the Hessian diagonal. For ReLU networks on aclassification task with C classes, the additional work is pro-portional to C gradient backpropagations (i.e. C = 10 forCIFAR-10, C = 100 for CIFAR-100). Parallel processingcan, to some extent, process these extra backpropagationsin parallel without significant overhead. If this is no longerpossible, we can fall back to a Monte Carlo (MC) samplingapproximation, which reduces the number of extra backproppasses to the number of samples (1 by default).2

While parallelization is possible for the gradient instru-ments, computing the maximum Hessian eigenvalue is in-herently sequential. Similar to [44], we use matrix-freeHessian-vector products by automatic differentiation [30],where each product has a cost proportional to one gradientcomputation. Regardless of the underlying iterative eigen-solver, multiple such products must be queried to computethe spectral norm (the required number depends on the spec-tral gap to the second-largest eigenvalue). This step is thuscostly and introduces the largest computational overhead.

Run time benchmark: Figure 6 shows the wall-clockcomputational overhead for individual instruments. Thesetimes were collected on a GPU for the CIFAR-10 3C3D testproblem (details in Appendix C).3 As expected, byproducts

2Indeed, such an MC-sampled approximation of the Hessian/generalizedGauss-Newton has been used in Figure 2 to reduce the prohibitively largenumber of extra backpropagations on CIFAR-100 (C = 100).

3We do not show run times for HessMaxEV because the associatedoverhead is large compared to the other quantities, as described above. Sur-prisingly, we observed significant cost for the two-dimensional histogram

1 4 16 64 256Track Interval

baseline

economy

business

fullC

onfig

urat

ion

1 1 1 1 1

1.6 1.2 1.1 1 1

2.4 1.4 1.1 1 1

32 8.8 3 1.5 1.11.0

1.5

2.0

2.5

3.0

Figure 7. Overhead of different COCKPIT configurations forvarying tracking intervals. Results collected while training onCIFAR-10 3C3D on a GPU. Blue fields mark settings which allowtracking without doubling the training time. More in Appendix C.2.

are virtually free, and quantities that rely solely on first-orderinformation add a small overhead (at most roughly 25 % onthis particular problem). Thanks to parallelization, the 10extra backward passes required for Hessian quantities re-duce to a less than 100 % overhead. As we will see now,the individual overheads also do not simply add up whenmultiple quantities are tracked, because quantities relying onthe same information share computations.

To allow a rough cost control, COCKPIT currently offersthree configurations, called economy, business, and full, inincreasing order of cost (cf. Table 1). As a rough guideline,we consider a factor of two to be an acceptable limit for theincrease in training time and benchmark the configurations’run times for different tracking intervals. Figure 7 showsa performance matrix for the CIFAR-10 3C3D problem,where settings that meet this limit are set in blue.

Improved performance due to shared computations is easyto read off from this figure: Summing all the individual over-heads shown in Figure 6 would result in a total overheadlarger than 200 %, while the joint overhead (business) re-duces to 140 %. The economy configuration can easily betracked at every step of this problem and stay well belowour threshold of doubling the execution time. COCKPIT’s

on GPU. It is caused by an implementation bottleneck for histogram shapesobserved in deep models. We thus also omit the run time of GradHist2dhere, as we expect it to be eliminated with future implementations (see Ap-pendix C.2 for a detailed performance analysis and additional benchmarks).

7

view shown in Figure 2 can be updated every 64-th iterationwithout a major increase in training time (this correspondsto about five updates per epoch). Finally, tracking any con-figuration about once per epoch – which is what many practi-tioners do – adds overhead close to zero (rightmost column).

This good performance is largely due to the efficiency ofthe BACKPACK package, which we leverage with customand optimized modification, that compacts information layer-wise and then discards unneeded buffers. Using layer-wiseinformation (Section 3.2) scales better to large networks,where storing the entire model’s individual gradients all atonce becomes increasingly expensive (see Appendix C).

To the best of our knowledge, many of the quantitiesin Table 1, especially those relying on individual gradients,have only been explored on rather small problems. WithCOCKPIT they can now be accessed at a reasonable rate fordeep learning models outside the toy problem category.

5. ShowcaseHaving introduced and benchmarked the tool, we can now

return to Figure 2 for a closer look. The figure shows a snap-shot from training the ALL-CNN-C [36] on CIFAR-100using SGD with a restarted cyclic learning rate schedule (seebottom left panel). Diagonal curvature instruments are con-figured to use an MC approximation in order to reduce therun time (remember that C = 100 here, compare Section 4).

A glance at all panels shows that the learning rate sched-ule is reflected in the metrics. However, the instruments alsoreveal interesting behavior in the early phase of training (first∼ 100 iterations), where the learning rate is still unaffectedby the schedule: There, the loss plateaus and the optimizertakes relatively small steps (compared to later, as can beseen in the small gradient norms, and small distance frominitialization). Based on these low-cost instruments, onemay thus at first suspect that training was poorly initialized;but training indeed succeeds after iteration 100! ViewingCOCKPIT entirely though, it becomes clear that optimizationis not stuck at all: While loss, gradient norms and distance inparameter space remain almost constant, curvature changes,which expresses itself in a clear downward trend of the max-imum Hessian eigenvalue (top right panel).

The importance of early phases in deep learning has re-cently been hypothesized in [14], suggesting a logarithmictimeline. Not only does our showcase support this hypothe-sis, it also adds evidence through the curvature-based met-rics, which in this particular case are the only meaningfulfeedback in the first few training steps. It also suggeststhat training should be monitored at log-spaced intervals.COCKPIT provides the flexibility to do so, indeed, Figure 2has been created with log-scheduled tracking events.

As a final note, we acknowledge that the approach takenhere promotes an amount of “manual” work (monitoringmetrics, taking decisions to intervene, etc.) that may seem

ironic, at odds with the paradigm of automation that is atthe heart of machine learning. While we recognize thisproblem, we argue that it is what is needed at this point inthe development of the field. Deep learning has to a largeextent been driven by scaling computational resources [39],and fully-automated, one-shot training may still be some wayout. To develop better optimization algorithms, researchers,not just users, need more direct insights and intuition aboutthe processes taking place “inside” neural networks. Weargue that the tool presented in this work provides just that.

To highlight how this may work, in Appendix D, we showthe COCKPIT view of two convex DEEPOBS problems: anoisy quadratic and logistic regression on MNIST. For theseproblems, the instruments behave quite differently comparedto the deep learning problem in Figure 2. In particular, thegradient norm increases (left column, bottom panel) overtraining, and individual gradients become less scattered (cen-ter column, top panel). This is diametrically opposed to theconvex problems and shows that deep learning differs evenquantitatively from well-understood optimization problems.

6. ConclusionIn contrast to classic deterministic optimization problems,

contemporary machine learning, in particular deep learning,remains a craft and an art. High dimensionality, stochastic-ity, and non-convexity require constant tracking and tuning,often resulting in a painful process of trial and error. Whenthings fail, popular performance measures, like the train-ing loss, do not provide enough information by themselves.These metrics only tell whether the model is learning, but notwhy. Alternatively, traditional debugging tools can provideaccess to individual weights and data. However, in modelswhose power only arises from the statistical properties ofmyriad weights, this approach is hopeless, like looking forthe proverbial needle in a haystack.

To alleviate this problem, we proposed COCKPIT, a vi-sual and practical debugging tool for deep learning. It offersinstruments to monitor the network’s internal dynamics dur-ing optimization, in real-time. Although these instrumentsutilize rich second-order information, efficient computationkeeps the necessary runtime overhead reasonable.

To demonstrate the tool’s utility, we showed how it canidentify bugs where traditional tools fail. The whole softwarepackage is available for PYTORCH, can be added to manyexisting training loops, and will be released as open source.

Of course, such a tool is never complete. Just like thereis no perfect universal debugger, the list of currently im-plemented instruments is necessarily incomplete. However,our experiments show that COCKPIT provides useful toolsand extracts valuable information not currently otherwiseavailable to the user. We believe that this information canhelp practitioners to simply make their architectures work,but also inspire researchers to build new intuitions for how

8

to improve optimization. The source code is structured flexi-bly, such that new instruments can be added easily withoutchanging the internals. Computational layer and graphicalfront-end are separated, so users who prefer other tools, e.g.TENSORBOARD, can pipe them there, too. And of course,instead of just displaying the information, the same numberscan also be used by novel algorithms directly, side-steppingthe human in the loop.

Acknowledgments

The authors gratefully acknowledge financial support bythe European Research Council through ERC StG Action757275 / PANAMA; the DFG Cluster of Excellence “Ma-chine Learning - New Perspectives for Science”, EXC2064/1, project number 390727645; the German FederalMinistry of Education and Research (BMBF) through theTübingen AI Center (FKZ: 01IS18039A); and funds fromthe Ministry of Science, Research and Arts of the State ofBaden-Württemberg. Moreover, the authors thank the Inter-national Max Planck Research School for Intelligent Systems(IMPRS-IS) for supporting Felix Dangel and Frank Schnei-der. Further, we are grateful to Agustinus Kristiadi, Alexan-dra Gessner, Christian Fröhlich, Filip de Roos, JonathanWenger, Julia Grosse, Lukas Tatzel, Marius Hobbhahn, andNicholas Krämer for providing feedback to the manuscript.

References[1] Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo,

Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jef-frey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfel-low, Andrew Harp, Geoffrey Irving, Michael Isard, YangqingJia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, JoshLevenberg, Dandelion Mané, Rajat Monga, Sherry Moore,Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens,Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker,Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, OriolVinyals, Pete Warden, Martin Wattenberg, Martin Wicke,Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scalemachine learning on heterogeneous systems, 2015. Softwareavailable from tensorflow.org. 1

[2] Ayush Manish Agrawal, Atharva Tendle, Harshvardhan Sikka,Sahib Singh, and Amr Kayid. Investigating learning in deepneural networks using layer-wise weight change, 2020. 3

[3] Achraf Bahamou and Donald Goldfarb. A dynamic samplingadaptive-SGD method for machine learning, 2019. 18

[4] Lukas Balles, Javier Romero, and Philipp Hennig. Couplingadaptive batch sizes with learning rates, 2017. 2, 11, 14

[5] Yoshua Bengio. Practical recommendations for gradient-based training of deep architectures, 2012. 5

[6] Raghu Bollapragada, Richard H. Byrd, and Jorge Nocedal.Adaptive sampling strategies for stochastic optimization.SIAM Journal on Optimization, 28:3312–3343, 2017. 4, 11,15, 16, 17, 18

[7] James Bradbury, Roy Frostig, Peter Hawkins, Matthew JamesJohnson, Chris Leary, Dougal Maclaurin, and SkyeWanderman-Milne. JAX: composable transformations ofPython+NumPy programs, 2018. 2

[8] Richard H. Byrd, Gillian M. Chin, Jorge Nocedal, and YuchenWu. Sample size selection in optimization methods for ma-chine learning. Math. Program., 134(1):127–155, 2012. 4,11, 15, 16

[9] Satrajit Chatterjee. Coherent gradients: An approach to under-standing generalization in gradient descent-based optimiza-tion. In International Conference on Learning Representa-tions, 2020. 4

[10] Satrajit Chatterjee and Piotr Zielinski. Making coherenceout of nothing at all: Measuring the evolution of gradientalignment, 2020. 4

[11] Felix Dangel, Frederik Kunstner, and Philipp Hennig. Back-PACK: Packing more into backprop. In International Confer-ence on Learning Representations, 2020. 1, 2, 4, 7

[12] Laurent Dinh, Razvan Pascanu, Samy Bengio, and YoshuaBengio. Sharp minima can generalize for deep nets. InDoina Precup and Yee Whye Teh, editors, Proceedings of the34th International Conference on Machine Learning, ICML2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70of Proceedings of Machine Learning Research, pages 1019–1028. PMLR, 2017. 4

[13] Fartash Faghri, David Duvenaud, David J. Fleet, and JimmyBa. A study of gradient variance in deep learning, 2020. 2

[14] Jonathan Frankle, David J. Schwab, and Ari S. Morcos. Theearly phase of neural network training, 2020. 3, 4, 8

9

[15] Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao. Aninvestigation into neural net optimization via Hessian eigen-value density, 2019. 4

[16] Boris Ginsburg. On regularization of gradient descent, layerimbalance and flat minima, 2020. 4

[17] Guy Gur-Ari, Daniel A. Roberts, and Ethan Dyer. Gradientdescent happens in a tiny subspace, 2019. 6

[18] Charles R. Harris, K. Jarrod Millman, Stéfan J. van derWalt, Ralf Gommers, Pauli Virtanen, David Cournapeau, EricWieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith,Robert Kern, Matti Picus, Stephan Hoyer, Marten H. vanKerkwijk, Matthew Brett, Allan Haldane, Jaime Fernán-dez del Río, Mark Wiebe, Pearu Peterson, Pierre Gérard-Marchant, Kevin Sheppard, Tyler Reddy, Warren Weckesser,Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant.Array programming with NumPy. Nature, 585(7825):357–362, Sept. 2020. 5

[19] Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Neu-ral Comput., 9(1):1–42, 1997. 4

[20] Stanislaw Jastrzebski, Maciej Szymczak, Stanislav Fort, De-vansh Arpit, Jacek Tabor, Kyunghyun Cho, and KrzysztofGeras. The break-even point on optimization trajectoriesof deep neural networks. In International Conference onLearning Representations, 2020. 4

[21] Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal,Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharpminima. In 5th International Conference on Learning Rep-resentations, ICLR 2017, Toulon, France, April 24-26, 2017,Conference Track Proceedings. OpenReview.net, 2017. 4

[22] Alex Krizhevsky. Learning multiple layers of features fromtiny images. Technical report, 2009. 5

[23] Yann Lecun, Léon Bottou, Yoshua Bengio, and PatrickHaffner. Gradient-based learning applied to document recog-nition. In Proceedings of the IEEE, volume 86, pages 2278–2324, 1998. 6

[24] Jinlong Liu, Yunzhi Bai, Guoqing Jiang, Ting Chen, andHuayan Wang. Understanding why neural networks gener-alize well through GSNR of parameters. In InternationalConference on Learning Representations, 2020. 2, 4, 11, 19

[25] Maren Mahsereci, Lukas Balles, Christoph Lassner, andPhilipp Hennig. Early stopping without a validation set, 2017.2, 11, 15

[26] Maren Mahsereci and Philipp Hennig. Probabilistic linesearches for stochastic optimization. In Advances in Neu-ral Information Processing Systems, volume 28, 2015. 14

[27] Rotem Mulayoff and Tomer Michaeli. Unique properties offlat minima in deep networks, 2020. 4

[28] Vaishnavh Nagarajan and J. Zico Kolter. Generalization indeep networks: The role of distance from initialization, 2019.3

[29] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer,James Bradbury, Gregory Chanan, Trevor Killeen, ZemingLin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, An-dreas Kopf, Edward Yang, Zachary DeVito, Martin Raison,Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, LuFang, Junjie Bai, and Soumith Chintala. Pytorch: An im-perative style, high-performance deep learning library. In H.

Wallach, H. Larochelle, A. Beygelzimer, F. d’ Alché-Buc, E.Fox, and R. Garnett, editors, Advances in Neural InformationProcessing Systems 32, pages 8024–8035. Curran Associates,Inc., 2019. 1

[30] Barak A. Pearlmutter. Fast exact multiplication by the Hes-sian. Neural Computation, 6(1):147–160, 1994. 4, 7, 18

[31] Levent Sagun, Leon Bottou, and Yann LeCun. Eigenvalues ofthe Hessian in deep learning: Singularity and beyond, 2017.4

[32] Levent Sagun, Utku Evci, V. Ugur Guney, Yann Dauphin,and Leon Bottou. Empirical analysis of the Hessian of over-parametrized neural networks, 2018. 4

[33] Karthik A. Sankararaman, Soham De, Zheng Xu, W. RonnyHuang, and Tom Goldstein. The impact of neural networkoverparameterization on gradient confusion and stochasticgradient descent, 2019. 4

[34] Mark Schmidt. Convergence rate of stochastic gradient withconstant step size. 2014. 4

[35] Frank Schneider, Lukas Balles, and Philipp Hennig. Deep-OBS: A Deep Learning Optimizer Benchmark Suite. In 7thInternational Conference on Learning Representations, ICLR,mar 2019. 5

[36] Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox,and Martin Riedmiller. Striving for simplicity: The all convo-lutional net, 2015. 3, 8

[37] Kei Takeuchi. The distribution of information statistics andthe criterion of goodness of fit of models. MathematicalScience, 153:12–18, 1976. 4, 18

[38] Valentin Thomas, Fabian Pedregosa, Bart van Merriënboer,Pierre-Antoine Mangazol, Yoshua Bengio, and Nicolas LeRoux. On the interplay between noise and curvature and itseffect on optimization and generalization, 2019. 4, 11, 18, 19

[39] Neil C. Thompson, Kristjan Greenewald, Keeheon Lee, andGabriel F. Manso. The computational limits of deep learning,2020. 8

[40] Sharan Vaswani, Aaron Mishkin, Issam Laradji, MarkSchmidt, Gauthier Gidel, and Simon Lacoste-Julien. Painlessstochastic gradient: Interpolation, line-search, and conver-gence rates. In Advances in Neural Information ProcessingSystems, volume 32, 2019. 14

[41] James Warsa, Todd Wareing, Jim Morel, John Mcghee, andRichard Lehoucq. Krylov subspace iterations for determinis-tic k-eigenvalue calculations. Nuclear Science and Engineer-ing - NUCL SCI ENG, 147, 05 2004. 18

[42] Yuhuai Wu, Mengye Ren, Renjie Liao, and Roger B.Grosse. Understanding short-horizon bias in stochastic meta-optimization. 6th International Conference on Learning Rep-resentations, ICLR 2018 - Conference Track Proceedings,mar 2018. 6

[43] Chen Xing, Devansh Arpit, Christos Tsirigotis, and YoshuaBengio. A Walk with SGD, 2018. 6, 14

[44] Zhewei Yao, Amir Gholami, Kurt Keutzer, and Michael Ma-honey. PyHessian: Neural networks through the lens of theHessian, 2019. 4, 7, 11, 18

10

A. COCKPIT instruments overviewTable 2 lists all quantities available in the first public release of COCKPIT. If necessary, we provide references to their

mathematical definition. This table contains additional quantities, compared to Table 1 in the main text. To improve thepresentation of this work, we decided to not describe every quantity available in COCKPIT in the main part and instead focuson the investigated metrics. Custom quantities can be added easily without having to understand the inner-workings.

Name Description Math

Loss Mini-batch loss at current iteration, LB(θ) (1)Parameters Parameter values θt at the current iteration -Distance Update size ‖θt+1 − θt‖ and distance from initialization ‖θt − θ0‖ -GradNorm Mini-batch gradient norm ‖gB(θ)‖ -Time Time of the current iteration (e.g. used in benchmark of Appendix C) -Alpha Normalized step on a noisy quadratic interpolation between two iterates θt,θt+1 (10)CABS Adaptive batch size for SGD, optimizes expected objective gain per cost, from [4] (12)EarlyStopping Evidence-based early stopping criterion for SGD, proposed in [25] (14d)GradHist1d Histogram of individual gradient elements, {gn(θj)}n∈B,j=1,...,D (15)GradHist2d Histogram of weights and individual gradient elements, {(θj , gn(θj))}n∈B,j=1,...,D (16)NormTest Normalized fluctuations of the residual norms ‖gB − gn‖, proposed in [8] (19c)InnerTest Normalized fluctuations of the gn’s parallel components along gB, proposed in [6] (22c)OrthoTest Normalized fluctuations of the gn’s orthogonal components along gB, proposed in [6] (25b)HessMaxEV Maximum Hessian eigenvalue, λmax(HB(θ)), inspired by [44] (26)HessTrace Exact or approximate Hessian trace, Tr(HB(θ)), inspired by [44] -TICDiag Relation between (diagonal) curvature and gradient noise, inspired by [38] (29)TICTrace Relation between curvature and gradient noise trace, inspired by [38] (28)MeanGSNR Average gradient signal-to-noise-ratio (GSNR), inspired by [24] (31b)

Table 2. Overview of all COCKPIT quantities with a short description and, if necessary ,a reference to mathematical definition.

11

B. Mathematical details

In this section, we want to provide the mathematical background for each instrument described in Table 2. This complementsthe more informal description presented in Section 2 in the main text, which focused more on the expressiveness of theindividual quantities. We will start by setting up the necessary notation in addition to the one introduced in Section 2.

B.1. Additional notation

Population properties: The population risk LP (θ) ∈ R and its variance Λ(θ) ∈ R are given by

LP (θ) = E(x,y)∼P [`(f(θ,x),y)] =

∫`(f(θ,x),y)P (x,y) dx dy , (4a)

ΛP (θ) = Var(x,y)∼P [`(f(θ,x),y)] =

∫(`(f(θ,x),y)− LP (θ))

2P (x,y) dx dy . (4b)

The population gradient gP (θ) ∈ RD and its variance ΣP (θ) ∈ RD×D are given by

gP (θ) = E(x,y)∼P [∇θ`(f(θ,x),y)] =

∫∇θ`(f(θ,x),y)P (x,y) dx dy , (5a)

ΣP (θ) = Var(x,y)∼P [∇θ`(f(θ,x),y)]

=

∫(∇θ`(f(θ,x),y)− gP (θ)) (∇θ`(f(θ,x),y)− gP (θ))

>P (x,y) dx dy .

(5b)

Empirical approximations: Let S denote a set of samples drawn i.i.d. from P , i.e. S = {(xi,yi) | i = 1, . . . , |S|}. With aslight abuse of notation the empirical risk approximated with S is

LS(θ) =1

|S|∑n∈S

`n(θ) (6a)

(later, S will represent either a mini-batch B, or the train set D). The empirical risk gradient gS(θ) ∈ RD on S is

gS(θ) = ∇θLS(θ) =1

|S|∑n∈S∇θ`n(θ) =

1

|S|∑n∈S

gn(θ) , (6b)

with individual gradients gn(θ) = ∇θ`n(θ) ∈ RD implied by a sample n. Population risk and gradient variancesΛP (θ),ΣP (θ) can be empirically estimated on S with the sample variances ΛS(θ) ∈ R, ΣS(θ) ∈ RD×D, given by

ΛP (θ) ≈ 1

|S| − 1

∑n∈S

(`n(θ)− LS(θ))2

:= ΛS(θ) , (7a)

ΣP (θ) ≈ 1

|S| − 1

∑n∈S

(gn(θ)− gS(θ)) (gn(θ)− gS(θ))>

:= ΣS(θ)

≈ 1

|S| − 1

[(∑n∈S

gn(θ)gn(θ)>

)− |S|gS(θ)gS(θ)>

].

(7b)

Often, gradient elements are assumed independent and hence their variance is diagonal (�2 denotes element-wise square),

diag(ΣP (θ)) ≈ 1

|S| − 1

∑n∈S

(gn(θ)− gS(θ))�2

= diag(ΣS(θ)

)∈ RD . (8)

Slicing: To avoid confusion between θt (parameter at iteration t) and θj (j-th parameter entry), we denote the latter as [θ]j .

12

θt θt+10.0

0.2

0.4

0.6

0.8

1.0

1.2

Los

s

Understepping: α < 0

θt θt+1

Minimizing: α ≈ 0

θt θt+1

Overshooting: α > 0

Figure 8. Motivational sketch for the α quantity. In each iteration of the optimizer we observe the loss function at two positions θt andθt+1 (shown in l). The black lines (—) show the observed slope at this position, which we can get from projecting the gradients onto thecurrent step direction θt+1 − θt. Note, that all four observations (two loss and two slope values) are noisy, due to being computed on amini-batch. With access to the individual losses and gradients (some samples shown in l/—), we can estimate their noise level and build anoise-informed quadratic fit (—). Using this fit, we determine whether the optimizer minimizes the local univariate loss (middle plot), orwhether we understep (left plot) or overshoot (right plot) the minimum.

B.2. Alpha (AlphaOptimized, AlphaGeneral)

Motivation: The goal of the α-quantity is to estimate and quantify the effect that a selected learning rate has on theoptimizer’s steps. Let’s consider the step that the optimizer takes at training iteration t. This parameter update from θt to θt+1

happens in a one-dimensional space, defined by the update direction θt+1 − θt = st. The update direction depends on theupdate rule of the optimizer, e.g. for SGD with learning rate η it is simply st = −ηgBt(θt).

We build a noise-informed univariate quadratic approximation along this update step (θt → θt+1) based on the two noisyloss function observations at θt and θt+1 and the two noisy slope observation at these two points. Examining this quadratic fit,we are able to determine where on this parabola our optimizer steps. Standardizing this, we express a step to the minimum ofthe loss in the update direction as α = 0. Analogously, steps that end short of this minimum result in α < 0, and a step overthe minimum in α > 0. These three different scenarios are illustrated in Figure 8 also showing the underlying observationsthat would lead to them. Figure 1 shows the distribution of α-values for two very different optimization trajectories.

Noisy observations: In order to build an approximation for the loss function in the update direction, we leverage the fourobservations of the function (and its derivative) that are available in each iteration. Due to the stochasticity of deep learningoptimization, we also take into account the noise-level of all observations by estimating them. The first two observations arethe mini-batch training losses LBt

(θt),LBt+1(θt+1) at point θt and θt+1, which are computed in every standard training loop.

The mini-batch losses are averages over individual losses,

LBt(θt) = EBt

[`(θt)] =1

|Bt|∑n∈Bt

`n(θt) ,

LBt+1(θt+1) = EBt+1

[`(θt+1)] =1

|Bt+1|∑

n∈Bt+1

`n(θt+1) ,

and using these individual losses, we can also compute the variances to estimate the noise-level of our loss observation,

VarBt[`(θt)] =

(1

Bt

∑n∈Bt

`n(θt)2

)−(

1

Bt

∑n∈Bt

`n(θt)

)2

,

VarBt+1 [`(θt+1)] =

1

|Bt+1|∑

n∈Bt+1

`n(θt+1)2

− 1

|Bt+1|∑

n∈Bt+1

`n(θt+1)

2

.

13

Similarly, we proceed with the slope in the update direction. To compute the slope of the loss function in the direction of theoptimizer’s update st, we project the current gradient along this update direction

EBt

[s>t g(θt)

‖st‖2]

=1

|Bt|∑n∈Bt

s>t gn(θt)

‖st‖2,

EBt+1

[s>t g(θt+1)

‖st‖2]

=1

|Bt+1|∑

n∈Bt+1

s>t gn(θt+1)

‖st‖2.

Just like before, we can also compute the variance of this slope, by leveraging individual gradients,

VarBt

[s>t g(θt)

‖st‖2]

=1

|Bt|∑n∈Bt

(s>t gn(θt)

‖st‖2)2

−(

1

|Bt|∑n∈Bt

s>t gn(θt)

‖st‖2

)2

,

VarBt+1

[s>t g(θt+1)

‖st‖2]

=1

|Bt+1|∑

n∈Bt+1

(s>t gn(θt+1)

‖st‖2)2

−

1

|Bt+1|∑

n∈Bt+1

s>t gn(θt+1)

‖st‖2

2

.

Quadratic fit & normalization: Using our (noisy) observations, we are now ready to build an approximation for the lossas a function of the step size, which we will denote as f(τ). We assume a quadratic function for f , which follows recentreports for the loss landscape of neural networks [43], i.e. a function f(τ) = w0 + w1τ + w2τ

2 parameterized by w ∈ R3.We further assume a Gaussian likelihood of the form

p(f |w,Φ

)= N

(f ; Φ>w,Λ

)(9)

for observations f of the loss and its slope. The observation matrix Φ and the noise matrix of the observations Λ are

Φ =

1 1 0 0τ1 τ2 1 1τ21 τ22 2τ1 2τ2

, Λ =

σf1 0 0 0

0 σf2 0 0

0 0 σf ′1

0

0 0 0 σf ′2

,

where τ denotes the position and σ denotes the noise-level estimate of the observation. The maximum likelihood solution ofEquation (9) for the parameters of our quadratic fit is given by

w =(ΦΛ−1Φ>

)ΦΛ−1f . (10)

Once we have the quadratic fit of the univariate loss function in the update direction, we normalize the scales such that theresulting α-value expresses the effective step taken by the optimizer sketched in Figure 8.

The α-quantity is related to recent line search approaches [26, 40]. However, instead of searching for an acceptable stepby repeated attempts, we instead report the effect of the current step size selection. This information could indeed be usedto automatically adapt the learning rate during the training process. But, as discussed in Section 3.3, it isn’t trivial what the“correct” decision is, as it might depend on the optimization problem, the training phase, and other factors.

B.3. CABS criterion: Coupling adaptive batch sizes with learning rates (CABS)

The CABS criterion, proposed in [4], can be used to adapt the mini-batch size during training with SGD. It relies on thegradient noise and approximately optimizes the objective’s expected gain per cost. The adaptation rule is (with learning rate η)

|B| ← ηTr(ΣP (θ))

LP (θ), (11)

and the practical implementation approximates LP (θ) ≈ LB(θ),Tr(ΣP (θ)) ≈ |B|−1|B| Tr(ΣB(θ)) (compare equations (10,22) and first paragraph of Section 4 in [4]). This yields the quantity computed in cockpit’s CABS instrument,

|B| ← η

1|B|∑D

j=1

∑n∈B [gn(θ)− gB(θ)]

2j

LB(θ). (12)

14

B.4. Early-stopping criterion for SGD (EarlyStopping)

The empirical risk LD(θ), and the mini-batch loss LB(θ) are only estimators of the target objective LP (θ). [25] motivatep(gB,D(θ) | gP (θ) = 0) as a measure for detecting noise in the finite data sets B,D due to sampling from P . They proposean evidence-based (EB) criterion for early stopping the training procedure based on mini-batch statistics, and model p(gB(θ))with a sampled diagonal variance approximation (compare Equation (8)),

p(gB(θ)) ≈D∏

j=1

N

[gP (θ)]j ;

[ΣB(θ)

]j,j

|B|

. (13)

Their SGD stopping criterion is

2

D

[log p(gB(θ))− EgB(θ)∼p(gB(θ)) [log p(gB(θ))]

]> 0 , (14a)

and translates into

1− |B|D

D∑j=1

[gB(θ)]2j[

ΣB(θ)]j,j

> 0 , (14b)

1− |B|D

D∑d=1

[gB(θ)]2d

1|B|−1

∑n∈B [gn(θ)− gB(θ)]

2d

> 0 , (14c)

1− |B|(|B| − 1)

D

D∑d=1

[gB(θ)]2d(∑

n∈B [gn(θ)]2d

)− |B| [gB(θ)]

2d

> 0 . (14d)

COCKPIT’s EarlyStopping quantity computes the left-hand side of Equation (14d).

B.5. Individual gradient element histograms (GradHist1d, GradHist2d)

For the |B| ×D individual gradient elements, COCKPIT’s GradHist1d instrument displays a histogram of

{gn(θj)}n∈B,j=1,...,D . (15)

COCKPIT’s GradHist2d instrument displays a two-dimensional histogram of the |B| ×D tuples

{(θj , gn(θj))}n∈B,j=1,...,D (16)

and the marginalized one-dimensional histograms over the parameter- and gradient axes.

B.6. Gradient tests (NormTest, InnerTest, OrthoTest)

[6, 8] propose batch size adaptation schemes based on the gradient noise. They formulate geometric constraints betweenpopulation and mini-batch gradient and accessible approximations that can be probed to decide whether the mini-batch sizeshould be increased. Because mini-batches are i.i.d. from P , it holds that

E [gB(θ)] = gP (θ), (17a)

E[gB(θ)>gP (θ)

]= ‖gP (θ)‖2. (17b)

The above works propose enforcing other weaker similarity in expectation during optimization. These geometric constraintsreduce to basic vector geometry (see Figure 9 (a) for an overview of the relevant vectors). We recall their formulation here forconsistency and derive the practical versions, which can be computed from training observables and are used in COCKPIT(consult Figure 9 (b) for the visualization).

15

(a)

gP

gBgB − gP

projgP (gB)

gB − projgP (gB)

(b)

2θnorm2θinner

2νortho

Figure 9. Conceptual sketch for gradient tests. (a) Relevant vectors to formulate the geometric constraints between population andmini-batch gradient probed by the gradient tests. (b) Gradient test visualization in COCKPIT.

B.6.1 Norm test (NormTest)

The norm test [8] constrains the residual norm ‖gB(θ)− gP (θ)‖, rescaled by ‖gP (θ)‖. This gives rise to a standardized ballof radius θnorm ∈ (0,∞) around the population gradient, where the mini-batch gradient should reside. [6] set θnorm = 0.9 intheir experiments and increase the batch size if (in the practical version, see below) the following constraint is not fulfilled

E

[‖gB(θ)− gP (θ)‖2

‖gP (θ)‖2

]≤ θ2norm . (18a)

Instead of taking the expectation over mini-batches, [8] note that the above will be satisfied if

1

|B|E[‖gn(θ)− gP (θ)‖2

‖gP (θ)‖2

]≤ θ2norm . (18b)

They propose a practical form of this test,

1

|B|(|B| − 1)

∑n∈B ‖gn(θ)− gB(θ)‖2

‖gB(θ)‖2≤ θ2norm , (19a)

which can be computed from mini-batch statistics. Rearranging

∑n∈B‖gn(θ)− gB(θ)‖2 =

(∑n∈B‖gn(θ)‖2

)− |B| ‖gB(θ)‖2 , (19b)

we arrive at

1

|B|(|B| − 1)

[∑n∈B ‖gn(θ)‖2

‖gB(θ)‖2− |B|

]≤ θ2norm (19c)

that leverages the norm of both the mini-batch and the individual gradients, which can be aggregated over parameters during abackward pass. COCKPIT’s NormTest corresponds to the maximum radius θnorm for which the above inequality holds.

B.6.2 Inner product test (InnerTest)

The inner product test [6] constrains the projection of gB(θ) onto gP (θ) (compare Figure 9 (a)),

projgP (θ) (gB(θ)) =gB(θ)>gP (θ)

‖gP (θ)‖2gP (θ) , (20)

16

rescaled by ‖gP (θ)‖. This restricts the mini-batch gradient to reside in a standardized band of relative width θinner ∈ (0,∞)around the population risk gradient. [6] use θinner = 0.9 (in the practical version, see below) to adapt the batch size if theparallel component’s variance does not satisfy the condition

Var

(gB(θ)>gP (θ)

‖gP (θ)‖2

)= E

(gB(θ)>gP (θ)

‖gP (θ)‖2− 1

)2 ≤ θ2inner (21a)

(note that by Equation (17) we have E[gB(θ)>gP (θ)

‖gP (θ)‖2

]= 1). [6] bounds Equation (21a) by the individual gradient variance,

1

|B|Var

(gn(θ)>gP (θ)

‖gP (θ)‖2

)=

1

|B|E

(gn(θ)>gP (θ)

‖gP (θ)‖2− 1

)2 ≤ θ2inner . (21b)

They then propose a practical form of Equation (21b), which uses the mini-batch sample variance,

1

|B|Var

(gn(θ)>gB(θ)

‖gB(θ)‖2

)=

1

|B|(|B| − 1)

∑n∈B

(gn(θ)>gB(θ)

‖gB(θ)‖2− 1

)2 ≤ θ2inner . (22a)

Expanding

∑n∈B

(gn(θ)>gB(θ)

‖gB(θ)‖2− 1

)2

=

∑n∈B

(gn(θ)>gB(θ)

)2‖gB(θ)‖4

− |B| (22b)

and inserting Equation (22b) into Equation (22a) yields

1

|B|(|B| − 1)

[∑n∈B

(gn(θ)>gB(θ)

)2‖gB(θ)‖4

− |B|]≤ θ2inner . (22c)

It relies on pairwise scalar products between individual gradients, which can be aggregated over layers during backpropagation.COCKPIT’s InnerTest quantity computes the maximum band width θinner that satisfies Equation (22c).

B.6.3 Orthogonality test (OrthoTest)

In contrast to the inner product test (Appendix B.6.2) which constrains the projection (Equation (20)), the orthogonality test[6] constrains the orthogonal part (see Figure 9 (a))

gB(θ)− projgP (θ) (gB(θ)) , (23)

rescaled by ‖gP (θ)‖. This restricts the mini-batch gradient to a standardized band of relative width νortho ∈ (0,∞) parallelto the population gradient. [6] uses ν = tan(80◦) ≈ 5.84 (in the practical version, see below) to adapt the batch size if thefollowing condition is violated,

E

[∥∥∥∥gB(θ)− projgP (θ) (gB(θ))

‖gP (θ)‖

∥∥∥∥2]≤ ν2ortho . (24a)

Expanding the norm, and inserting Equation (20), this simplifies to

E

[∥∥∥∥ gB(θ)

‖gP (θ)‖ −gB(θ)>gP (θ)

‖gP (θ)‖2gP (θ)

‖gP (θ)‖

∥∥∥∥2]≤ ν2ortho ,

E

[‖gB(θ)‖2‖gP (θ)‖2 −

(gB(θ)>gP (θ)

)2‖gP (θ)‖4

]≤ ν2ortho .

(24b)

17

[6] bound this inequality using individual gradients instead,

1

|B|E[∥∥∥∥ gn(θ)

‖gP (θ)‖2 −gn(θ)>gP (θ)

‖gP (θ)‖2gP (θ)

‖gP (θ)‖

∥∥∥∥2]≤ ν2ortho . (24c)

They propose the practical form

1

|B|(|B| − 1)E

[∥∥∥∥ gn(θ)

‖gB(θ)‖ −gn(θ)>gB(θ)

‖gB(θ)‖2gB(θ)

‖gB(θ)‖

∥∥∥∥2]≤ ν2ortho , (25a)

which simplifies to

1

|B|(|B| − 1)

∑n∈B

(‖gn(θ)‖2‖gB(θ)‖2 − 2

(gn(θ)>gB(θ)

)2‖gB(θ)‖4 + 1

)≤ ν2ortho . (25b)

It relies on pairwise scalar products between individual gradients which can be aggregated over layers during a backwardpass.COCKPIT’s OrthTest quantity computes the maximum band width νortho which satisfies Equation (25b).

Relation to acute angle test: Recently, a novel “acute angle test” was proposed by [3]. While the theoretical constraintbetween gB(θ) and gP (θ) differs from the orthogonality test, the practical versions coincide. Hence, we do not incorporatethe acute angle here.

B.7. Hessian maximum eigenvalue (HessMaxEV)

The Hessian’s maximum eigenvalue λmax(HB(θ)) is computed with an iterative eigensolver from Hessian-vector productsthrough PYTORCH’s automatic differentiation [30]. Like [44], we employ power iterations with similar default stoppingparameters (stop after at most 100 iterations, or if the iterate does converged with a relative and absolute tolerance of10−3, 10−6, respectively) to compute λmax(HB(θ)) through the HessMaxEV quantity in COCKPIT.

In principle, more sophisticated eigensolvers (for example Arnoldi’s method) could be applied to converge in feweriterations or compute eigenvalues other than the leading ones. [41] empirically demonstrate that the FLOP ratio betweenpower iteration and implicitly restarted Arnoldi method can reach values larger than 100. While we can use such a beneficialmethod on a CPU through scipy.sparse.linalg.eigsh we are restricted to the GPU-compatible power iteration forGPU training. We expect that extending the support of popular machine learning libraries like PYTORCH for such iterativeeigensolvers on GPUs can help to save computation time.

λmax(HB(θ)) = max‖v‖=1

‖HB(θ)v‖ = maxv∈RD

v>HB(θ)v

v>v. (26)

B.8. Hessian trace (HessTrace)

In comparison to [44], who leverage Hessian-vector products [30] to estimate the Hessian trace, we compute the exactvalue Tr(HB(θ)) with the HessTrace quantity in COCKPIT by aggregating the output of BACKPACK’s DiagHessianextension, which computes the diagonal entries ofHB(θ). Alternatively, the trace can also be estimated from the generalizedGauss-Newton matrix, or an MC-sampled approximation thereof.

B.9. Takeuchi Information Criterion (TIC) (TICDiag, TICTrace)

Recent work by [38] suggests that optimizer convergence speed and generalization is mainly influenced by curvatureand gradient noise; and hence their interaction is crucial to understanding the generalization and optimization behavior ofdeep neural networks. They reinvestigate the Takeuchi Information criterion [37], an estimator for the generalization gap inoverparameterized maximum likelihood estimation. At a local minimum θ?, the generalization gap is estimated by the TIC

1

|D| Tr(HP (θ?)−1CP (θ?)

), (27)

whereHP (θ?) is the population Hessian and CP (θ?) is the gradient’s uncentered second moment at the local minimum,

CP (θ?) =

∫∇θ`(f(θ?,x),y) (∇θ`(f(θ?,x),y))

>P (x,y) dx dy.

18

Both matrices are inaccessible in practice. In their experiments, [38] propose the approximation Tr(C)/Tr(H) forTr(H−1C). They also replace the Hessian by the Fisher as it is easier to compute. With these practical simplifications, theyinvestigate the TIC of trained neural networks where the curvature and noise matrix are evaluated on a large data set.

The TIC provided in COCKPIT differs from this setting, since by design we want to observe quantities during training,while avoiding additional model predictions. Also, BACKPACK provides access to the Hessian; hence we don’t need to usethe Fisher. We propose the following two approximations of the TIC from a mini-batch:

• TICTrace: Uses the approximation of [38] which replaces the matrix-product trace by the product of traces,

Tr (CB(θ))

Tr (HB(θ))=

1|B|∑

n∈B‖gn(θ)‖2

Tr (HB(θ)). (28)

• TICDiag: Uses a diagonal approximation of the Hessian, which is cheap to invert,

Tr(

diag (HB(θ))−1CB(θ)

)=

1

|B|D∑

j=1

[HB(θ)]−1j,j

[∑n∈B

gn(θ)�2

]j

. (29)

B.10. Gradient signal-to-noise-ratio (MeanGSNR)

The gradient signal-to-noise-ratio GSNR([θ]j) ∈ R for a single parameter [θ]j is defined as

GSNR([θ]j) =E(x,y)∼P

[[∇θ`(f(θ,x),y)]j

]2Var(x,y)∼P

[[∇θ`(f(θ,x),y)]j

] =[gP (θ)]

2j

[ΣP (θ)]j,j. (30)

[24] uses it to explain generalization properties of models in the early training phase. We apply their estimation to mini-batches,

GSNR([θ]j) ≈[gB(θ)]

2j

|B|−1|B|

[ΣB(θ)

]j,j

=[gB(θ)]

2j(∑

n∈B [gn(θ)]2j

)− |B| [gB(θ)]

2j

. (31a)

Inspired by [24], COCKPIT’s MeanGSNR computes the average GSNR over all parameters,

1

D

D∑j=1

GSNR([θ]j) . (31b)

19

C. Implementation details and additional benchmarksIn this section, we provide more details about our implementation (Appendix C.1) to access the desired quantities with

as little overhead as possible. Additionally, we present more benchmarks for individual instruments (Appendix C.2.1) andCOCKPIT configurations (Appendix C.2.2). These are similar but extended versions of the ones presented in Figures 6 and 7in the main text. Lastly, we benchmark different implementations of computing the two-dimensional gradient histogram(Appendix C.3), identifying a computational bottleneck for its current GPU implementation.

Hardware details: Throughout this Further, we are grateful to Agustinus Kristiadi, Alexandra Gessner, Christian Fröhlich,Filip de Roos, Jonathan Wenger, Julia Grosse, Lukas Tatzel, Marius Hobbhahn, and Nicholas Krämer for providing feedbackto the manuscript.we conducted benchmarks on the following setup

• CPU: Intel Core i7-8700K CPU @ 3.70 GHz × 12 (32 GB)

• GPU: NVIDIA GeForce RTX 2080 Ti (11 GB)

C.1. Hooks & Memory benchmarks

To improve memory consumption, we compact information during the backward pass by adding hooks to the neuralnetwork’s layers. These are executed after BACKPACK extensions and have access to the quantities computed therein. Theycompress information to what is requested by a quantity and free the memory occupied by BACKPACK buffers. Such savingsprimarily depend on the parameter distribution over layers, and are bigger for more balanced architectures (compare Figure 10).

Example: Say, we want to compute a histogram over the |B| × D individual gradient elements of a network. Supposethat |B| = 128 and the model is DEEPOBS’s CIFAR-10 3C3D test problem with 895, 210 parameters. Given that everyparameter is stored in single precision, the model requires 895, 210×4 Bytes ≈ 3.41 MB. Storing the individual gradients willrequire 128× 895, 210× 4 Bytes ≈ 437 MB (for larger networks this quickly exceeds the available memory as the individualgradients occupy |B| times the model size). If instead, the layer-wise individual gradients are condensed into histograms ofnegligible size and immediately freed afterwards during backpropagation, the maximum memory overhead reduces to storingthe individual gradients of the largest layer. For our example, the largest layer has 589, 824 parameters, and the associatedindividual gradients will require 128 × 589, 824 × 4 Bytes ≈ 288 MB, saving roughly 149 MB of RAM. In practice, weobserve slightly larger savings, see Figure 10 (c).

20

(a)

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 1.1 1.2 1.3 1.40

1,000

2,000

Mem

ory

[MB

]

FASHION-MNIST 2C2D

expensive: 2166± 7 MBoptimized: 2052± 1 MBbaseline: 525± 0 MB

(b)

0 5 · 10−2 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 0.55 0.60

500

1,000

Mem

ory

[MB

]

MNIST MLP

expensive: 1034± 0 MBoptimized: 765± 0 MBbaseline: 391± 1 MB

(c)

0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 20

500

1,000

Mem

ory

[MB

]

CIFAR-10 3C3D

expensive: 1247± 9 MBoptimized: 1029± 0 MBbaseline: 808± 8 MB

(d)

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 50

1,000

2,000

3,000

4,000

Time [s]

Mem

ory

[MB

]

CIFAR-100 ALL-CNN-C

expensive: 3882± 27 MBoptimized: 2618± 12 MBbaseline: 1903± 5 MB

Figure 10. Memory consumption and savings with hooks during one forward-backward step on a CPU for different DEEPOBS problems.We compare three settings; i) without COCKPIT (baseline); ii) COCKPIT with GradHist1d with BACKPACK (expensive); iii) COCKPIT

with GradHist1d with BACKPACK and additional hooks (optimized). Peak memory consumptions are highlighted by horizontal dashedbars and shown in the legend. Shaded areas, if visible, fill two standard deviations above and below the mean value, all of them result fromten independent runs. Dotted lines indicate individual runs. Our optimized approach allows to free obsolete tensors during backpropagationand thereby reduces memory consumption. From top to bottom: the effect is less pronounced for architectures that concentrate the majorityof parameters in a single layer ((a) 3, 274, 634 total, 3, 211, 264 largest layer) and increases for more balanced networks (b) 1, 336, 610total, 784, 000 largest layer, (c): 895, 210 total, 589, 824 largest layer).

21

C.2. Additional run time benchmarks

C.2.1 Individual instrument overhead

To estimate the computational overhead for individual instruments, we run COCKPIT with that instrument for 32 iterations,tracking at every step. Training proceeds with the default batch size specified by the DEEPOBS problem and uses SGD withlearning rate 10−3. We measure the time between iterations 1 and 32, and average for the overhead per step. Every suchestimate is repeated over 10 random seeds to obtain mean and error bars as reported in Figure 6.

Note that this protocol does not include initial overhead for setting up data loading and also does not include the timefor evaluating train/test loss on a larger data set, which is usually done by practitioners. Hence, we even expect the shownoverheads to be smaller in a conventional training loop which includes the above steps.

Individual overhead on GPU versus CPU: Figure 11 and Figure 12 show the individual overhead for four differentDEEPOBS problems on GPU and CPU, respectively. The left part of Figure 11 (c) corresponds to Figure 6. Right panels showthe expensive quantities, which we omitted in the main text as they were expected to be expensive due to their computationalwork (HessMaxEV) or bottlenecks in the implementation (GradHist2d, see Appendix C.3 for details). We see that they arein many cases equally or more expensive than computing all other instruments. Another expected feature of the GPU-to-CPUcomparison is that parallelism on the CPU is significantly less pronounced. Hence, we observe an increased overhead for allquantities that contain non-linear transformations and contractions of the high-dimensional individual gradients, or requireadditional backpropagations (curvature).

C.2.2 Configuration overhead

For the estimation of different COCKPIT configuration overheads, we use almost the same setting as described above, trainingfor 512 iterations and tracking only every specified interval.

Configuration overhead on GPU versus CPU: Figure 13 and Figure 14 show the configuration overhead for four differentDEEPOBS problems. The bottom left part of Figure 13 corresponds to Figure 7. In general, we observe that increasedparallelism can be exploited on a GPU, leading to smaller overheads in comparison to a CPU.

22

(a) Computational overhead for MNIST Log. Reg. (GPU)

Baseline GradientNorm

Loss Distance 1DHistogram

Norm TestOrthogonalityTest

InnerProduct Test

Trace TIC Alpha0

1

2

3

Run

Tim

eO

verh

ead

Baseline MaxEV 2DHistogram

0.0

2.5

5.0

7.5

10.0

(b) Computational overhead for MNIST MLP (GPU)

Baseline Loss GradientNorm

Distance Trace Norm Test TIC OrthogonalityTest

InnerProduct Test

1DHistogram

Alpha0

1

2

3

Run

Tim

eO

verh

ead

Baseline MaxEV 2DHistogram

0

50

100

150

(c) Computational overhead for CIFAR-10 3C3D (GPU)

Baseline Loss GradientNorm

Distance Norm Test InnerProduct Test

OrthogonalityTest

1DHistogram

Alpha Trace TIC0

1

2

3

Run

Tim

eO

verh

ead

Baseline MaxEV 2DHistogram

0

5

10

15

20

(d) Computational overhead for FASHION-MNIST 2C2D (GPU)

Baseline Loss GradientNorm

Distance Norm Test InnerProduct Test

OrthogonalityTest

Alpha Trace 1DHistogram

TIC0

1

2

3

4

Run

Tim

eO

verh

ead

Baseline MaxEV 2DHistogram

0

50

100

150

200

Figure 11. Individual overhead of COCKPIT’s instruments on GPU for four different problems. All run times are shown as multiplesof the baseline without tracking. Expensive quantities are displayed in separate panels on the right. Experimental details in the text.

23

(a) Computational overhead for MNIST Log. Reg. (CPU)

Baseline GradientNorm

Loss Distance Trace Norm Test InnerProduct Test

OrthogonalityTest

TIC Alpha 1DHistogram

0

1

2

3

Run

Tim

eO

verh

ead

Baseline MaxEV 2DHistogram

0

1

2

3

4

(b) Computational overhead for MNIST MLP (CPU)

Baseline Loss GradientNorm

Distance Trace OrthogonalityTest

InnerProduct Test

Norm Test TIC Alpha 1DHistogram

0

10

20

30

40

Run

Tim

eO

verh

ead

Baseline MaxEV 2DHistogram

0

50

100

150

(c) Computational overhead for CIFAR-10 3C3D (CPU)

Baseline GradientNorm

Loss Distance InnerProduct Test

OrthogonalityTest

Norm Test Alpha 1DHistogram

Trace TIC0

2

4

6

8

Run

Tim

eO

verh

ead

Baseline 2DHistogram

MaxEV0

20

40

60

80

(d) Computational overhead for FASHION-MNIST 2C2D (CPU)

Baseline GradientNorm

Loss DistanceOrthogonalityTest

InnerProduct Test

Norm Test Trace Alpha TIC 1DHistogram

0

5

10

15

20

Run

Tim

eO

verh

ead

Baseline 2DHistogram

MaxEV0

50

100

Figure 12. Individual overhead of COCKPIT’s instruments on CPU for four different problems. All run times are shown as multiplesof the baseline without tracking. Expensive quantities are displayed in separate panels on the right. Experimental details in the text.

24

1 4 16 64 256Track Interval

baseline

economy

business

fullCon

figur

atio

n 1 1 1 1 1

1.6 1.3 1.1 1 1

1.8 1.3 1.1 1 1

15 4.7 2 1.2 1.1

MNIST Log. Reg. (GPU)

1

2

31 4 16 64 256

Track Interval

baseline

economy

business

fullCon

figur

atio

n 1 1 1 1 1

5 2.1 1.3 1.1 1

5.8 2.3 1.3 1.1 1

1.8e+02 45 12 3.7 1.7

MNIST MLP (GPU)

1

2

3

1 4 16 64 256Track Interval

baseline

economy

business

fullCon

figur

atio

n 1 1 1 1 1

1.6 1.2 1.1 1 1

2.4 1.4 1.1 1 1

32 8.8 3 1.5 1.1

CIFAR-10 3c3d (GPU)

1

2

31 4 16 64 256

Track Interval

baseline

economy

business

fullCon

figur

atio

n 1 0.99 1 1 1

7.7 2.8 1.4 1.1 1

11 3.6 1.6 1.1 1

2.8e+02 72 19 5.3 2

F-MNIST 2c2d (GPU)

1

2

3

Figure 13. Overhead of COCKPIT configurations on GPU for four different problems with varying tracking interval.

1 4 16 64 256Track Interval

baseline

economy

business

fullCon

figur

atio

n 1 1 1 0.99 1

1.8 1.3 1.1 1 1

1.9 1.2 1 1 1

8 2.6 1.4 1.1 1

MNIST Log. Reg. (CPU)

1

2

31 4 16 64 256

Track Interval

baseline

economy

business

fullCon

figur

atio

n 1 1 1 1 1

78 21 5.9 2.3 1.3

85 23 6.8 2.4 1.3

3e+02 77 20 5.7 2.1

MNIST MLP (CPU)

1

2

3

1 4 16 64 256Track Interval

baseline

economy

business

fullCon

figur

atio

n 1 1 0.99 1 1

6.1 2.4 1.4 1.1 1

14 4.5 1.9 1.2 1.1

1.1e+02 30 8.4 2.7 1.4

CIFAR-10 3c3d (CPU)

1

2

31 4 16 64 256

Track Interval

baseline

economy

business

fullCon

figur

atio

n 1 1 1 1 1

41 12 3.7 1.7 1.2

57 16 4.7 1.9 1.2

3e+02 77 20 5.6 2.1

F-MNIST 2c2d (CPU)

1

2

3

Figure 14. Overhead of COCKPIT configurations on CPU for four different problems with varying tracking interval.

25

(a)

0 0.2 0.4 0.6 0.8 1

10−1

100

101

102

Histogram balance b

Run

time

[s]

GPU

PYTORCH (COCKPIT)PYTORCH (third party)

(b)

0 0.2 0.4 0.6 0.8 1100

100.5

Histogram balance b

Run

time

[s]

CPU

PYTORCH (COCKPIT)PYTORCH (third party)NUMPY (single thread)

Figure 15. Performance of two-dimensional histogram GPU implementations depends on the data. (a) Run time for two differentGPU implementations with histograms of different imbalance. COCKPIT’s implementation outperforms the third party solution by morethan one order of magnitude in the deep learning regime (b� 1). (b) On CPU, performance is robust to histogram balance. The run timedifference between NUMPY and PYTORCH is due to multi-threading. Data has the same size as DEEPOBS’s CIFAR-10 3C3D problem(D = 895, 210, |B| = 128). Curves represent averages over 10 independent runs. Error bars are omitted to improve legibility.

C.3. Performance of two-dimensional histograms:

Both one- and two-dimensional histograms require |B| ×D elements be accessed, and hence perform similarly. However,we observed different behavior on GPU and decided to omit the two-dimensional histogram’s run time in the main text.As explained here, this performance lack is not fundamental, but a shortcoming of the GPU implementation. PYTORCHprovides built-in functionality for computing one-dimensional histograms at the time of writing, but is not yet featuringmulti-dimensional histograms. We experimented with three implementations:

• PYTORCH (third party): A third party implementation4 under review for being integrated into PYTORCH5. It relieson torch.bincount, which uses atomicAdds that represent a bottleneck for histograms where most counts arecontained in one bin6. This occurs often for over-parameterized deep models, as most of the gradient elements are zero.

• PYTORCH (COCKPIT): Our implementation uses a suggested workaround, computes bin indices and scatters the countsinto their associated bins with torch.Tensor.put_. This circumvents atomicAdds, but has poor memory locality.

• NUMPY: The single-threaded numpy.histogram2d serves as baseline, but does not run on GPUs.

To demonstrate the strong performance dependence on the data, we generate data from a uniform distribution over[0, b]× [0, b], where b ∈ (0, 1) parametrizes the histogram’s balance, and compute two-dimensional histograms on [0, 1]× [0, 1].Figure 15 (a) shows a clear increase in run time of both GPU implementations for more imbalanced histograms. Note thateven though our implementation outperforms the third party by more than one order of magnitude in the deep neural networkregime (b� 1), it is still considerably slower than a one-dimensional histogram (see Figure 11 (c)), and even slower on GPUthan on CPU (Figure 15 (b)). As expected, the CPU implementations do not significantly depend on the data (Figure 15 (b)).The performance difference between PYTORCH and NUMPY is likely due to multi-threading versus single-threading.

Although a carefully engineered histogram GPU implementation is currently not available, we think it will reduce thecomputational overhead to that of a one-dimensional histogram in future releases.

4Permission granted by the authors of https://github.com/miranov25/RootInteractive/.../histogramdd_pytorch.py5See https://github.com/pytorch/pytorch/pull/44485.6See https://discuss.pytorch.org/t/torch-bincount-1000x-slower-on-cuda/42654

26

D. COCKPIT view of convex stochastic problems

Figure 16. Screenshot of COCKPIT’s full view for convex DEEPOBS problems. Top COCKPIT shows training on a noisy quadratic lossfunction. Bottom shows training on logistic regression on MNIST. Figure and labels are not meant to be legible. It is evident, that there is afundamental difference in the optimization process, compared to training deep networks, i.e. Figure 2. This is, for example, visible whencomparing the gradient norms, which converge to zero for convex problems but not for deep learning.

27