![Page 1: Trace Debugging in Academics – the Future of Multicore ... · 2 Technische Universität München Philipp Wagner Institute for Integrated Systems Well-known roads h t t p: / / e](https://reader035.vdokument.com/reader035/viewer/2022062607/604b933271ce8d23a75cb9a9/html5/thumbnails/1.jpg)
Technische Universität München
Institute for Integrated SystemsProf. Dr. Andreas Herkersdorf
Arcisstraße 2180333 MünchenGermany
http://www.lis.ei.tum.de
Trace Debugging in Academics – the Future of Multicore Debugging?
Philipp Wagner
November 14, 2013
![Page 2: Trace Debugging in Academics – the Future of Multicore ... · 2 Technische Universität München Philipp Wagner Institute for Integrated Systems Well-known roads h t t p: / / e](https://reader035.vdokument.com/reader035/viewer/2022062607/604b933271ce8d23a75cb9a9/html5/thumbnails/2.jpg)
2
Technische Universität München
Philipp WagnerInstitute for Integrated Systems
Well-known roads
http
://e
n.w
ikip
edia
.org
/wik
i/File
:Hum
e_Fr
eew
ay_C
raig
ieb
urn_
Byp
ass.
jpg
Wik
iped
ia u
ser
Mel
bur
nian
, CC
-BY
-SA
![Page 3: Trace Debugging in Academics – the Future of Multicore ... · 2 Technische Universität München Philipp Wagner Institute for Integrated Systems Well-known roads h t t p: / / e](https://reader035.vdokument.com/reader035/viewer/2022062607/604b933271ce8d23a75cb9a9/html5/thumbnails/3.jpg)
3
Technische Universität München
Philipp WagnerInstitute for Integrated Systems
Road Signs: Where are we going?
http
://w
ww
.flic
kr.c
om/p
hoto
s/sc
hosc
hie/
1258
6476
76fli
ckr
user
“sc
hosc
hie”
, CC
-BY
![Page 4: Trace Debugging in Academics – the Future of Multicore ... · 2 Technische Universität München Philipp Wagner Institute for Integrated Systems Well-known roads h t t p: / / e](https://reader035.vdokument.com/reader035/viewer/2022062607/604b933271ce8d23a75cb9a9/html5/thumbnails/4.jpg)
4
Technische Universität München
Philipp WagnerInstitute for Integrated Systems
And a Glimpse Beyond?
http
://w
ww
.flic
kr.c
om/p
hoto
s/d
ougt
one/
6172
0504
48fli
ckr
user
“d
ougt
one”
, CC
-BY
-SA
(ed
ited
)
![Page 5: Trace Debugging in Academics – the Future of Multicore ... · 2 Technische Universität München Philipp Wagner Institute for Integrated Systems Well-known roads h t t p: / / e](https://reader035.vdokument.com/reader035/viewer/2022062607/604b933271ce8d23a75cb9a9/html5/thumbnails/5.jpg)
5
Technische Universität München
Philipp WagnerInstitute for Integrated Systems
The Road to Multicore Debug – Overview
• How we're debugging now• Tracing – the new way of Multicore Debugging
– How we're doing it now– How we might do it in the future
![Page 6: Trace Debugging in Academics – the Future of Multicore ... · 2 Technische Universität München Philipp Wagner Institute for Integrated Systems Well-known roads h t t p: / / e](https://reader035.vdokument.com/reader035/viewer/2022062607/604b933271ce8d23a75cb9a9/html5/thumbnails/6.jpg)
6
Technische Universität München
Philipp WagnerInstitute for Integrated Systems
Proven and Tested:
The way we debug today.
![Page 7: Trace Debugging in Academics – the Future of Multicore ... · 2 Technische Universität München Philipp Wagner Institute for Integrated Systems Well-known roads h t t p: / / e](https://reader035.vdokument.com/reader035/viewer/2022062607/604b933271ce8d23a75cb9a9/html5/thumbnails/7.jpg)
7
Technische Universität München
Philipp WagnerInstitute for Integrated Systems
Run-Control Debug
• It's easy• It's well-known• It works great
– for programs that have no/little concurrent parts
• Today that's the majority of software!
![Page 8: Trace Debugging in Academics – the Future of Multicore ... · 2 Technische Universität München Philipp Wagner Institute for Integrated Systems Well-known roads h t t p: / / e](https://reader035.vdokument.com/reader035/viewer/2022062607/604b933271ce8d23a75cb9a9/html5/thumbnails/8.jpg)
8
Technische Universität München
Philipp WagnerInstitute for Integrated Systems
Run-Control Debug
• What to do with– complex, heterogeneous
systems?– concurrency problems?
![Page 9: Trace Debugging in Academics – the Future of Multicore ... · 2 Technische Universität München Philipp Wagner Institute for Integrated Systems Well-known roads h t t p: / / e](https://reader035.vdokument.com/reader035/viewer/2022062607/604b933271ce8d23a75cb9a9/html5/thumbnails/9.jpg)
9
Technische Universität München
Philipp WagnerInstitute for Integrated Systems
Run-Control Debug
• What to do with– complex, heterogeneous
systems?– concurrency problems?
• Different clock domains
![Page 10: Trace Debugging in Academics – the Future of Multicore ... · 2 Technische Universität München Philipp Wagner Institute for Integrated Systems Well-known roads h t t p: / / e](https://reader035.vdokument.com/reader035/viewer/2022062607/604b933271ce8d23a75cb9a9/html5/thumbnails/10.jpg)
10
Technische Universität München
Philipp WagnerInstitute for Integrated Systems
Run-Control Debug
• What to do with– complex, heterogeneous
systems?– concurrency problems?
• Different clock domains• Real-time?
Motor:http://commons.wikimedia.org/wiki/File:Ac-elektromotor-robuster-asynchronmotor.jpgUser “Egzon123” on Wikipedia, CC-BY-SA
![Page 11: Trace Debugging in Academics – the Future of Multicore ... · 2 Technische Universität München Philipp Wagner Institute for Integrated Systems Well-known roads h t t p: / / e](https://reader035.vdokument.com/reader035/viewer/2022062607/604b933271ce8d23a75cb9a9/html5/thumbnails/11.jpg)
11
Technische Universität München
Philipp WagnerInstitute for Integrated Systems
Run-Control Debug
• What to do with– complex, heterogeneous
systems?– concurrency problems?
• Different clock domains• Real-time?• Unattended debug?
Wind turbine: http://images.cdn.fotopedia.com/flickr-185488411-original.jpgUser “phault” on Flickr, CC-BY
![Page 12: Trace Debugging in Academics – the Future of Multicore ... · 2 Technische Universität München Philipp Wagner Institute for Integrated Systems Well-known roads h t t p: / / e](https://reader035.vdokument.com/reader035/viewer/2022062607/604b933271ce8d23a75cb9a9/html5/thumbnails/12.jpg)
12
Technische Universität München
Philipp WagnerInstitute for Integrated Systems
A Road Sign: Tracing
• Can find concurrency bugs• Can work in heterogeneous
environments• Can be non-intrusive• Can be unattended
http://www.flickr.com/photos/countylemonade/5321535677User “Garrett” on flickr, CC-BY (edited)
Tracing can solve the problems of run-control debugging!
![Page 13: Trace Debugging in Academics – the Future of Multicore ... · 2 Technische Universität München Philipp Wagner Institute for Integrated Systems Well-known roads h t t p: / / e](https://reader035.vdokument.com/reader035/viewer/2022062607/604b933271ce8d23a75cb9a9/html5/thumbnails/13.jpg)
13
Technische Universität München
Philipp WagnerInstitute for Integrated Systems
Road Bumps
Data Trace for a DDR3 Memory [1]
DDR3-1333 Peak Data Rate: ~ 10 GByte/s
Compression for an average memory trace: ~ 50 % reduction
NEXUS 5001 Trace Port [2]
Freescale MPC5777M
4 lanes of 1 GBit/s Xilinx Aurora trace port
[2] Only on the emulation device, 4 lanes w/ 1.25 GBit/s data rate and 8b10b encoding. Ignoring all NEXUS 5001 message overhead.
[1] Data compression rate: Christian Morgenstern, “Collection and Compression of Memory Traces for Manycore System-On-Chip”, Bachelor Thesis, 2013
40 GBit/s data generation 4 GBit/s off-chip
![Page 14: Trace Debugging in Academics – the Future of Multicore ... · 2 Technische Universität München Philipp Wagner Institute for Integrated Systems Well-known roads h t t p: / / e](https://reader035.vdokument.com/reader035/viewer/2022062607/604b933271ce8d23a75cb9a9/html5/thumbnails/14.jpg)
14
Technische Universität München
Philipp WagnerInstitute for Integrated Systems
Think! On Chip.
DIKW Chain (Ackoff, 1989). This representation is inspired by Gene Bellinger, “Knowledge Management—Emerging Perspectives”. http://www.systems-thinking.org/kmgmt/kmgmt.htm. 2004
![Page 15: Trace Debugging in Academics – the Future of Multicore ... · 2 Technische Universität München Philipp Wagner Institute for Integrated Systems Well-known roads h t t p: / / e](https://reader035.vdokument.com/reader035/viewer/2022062607/604b933271ce8d23a75cb9a9/html5/thumbnails/15.jpg)
15
Technische Universität München
Philipp WagnerInstitute for Integrated Systems
Think! On Chip.
![Page 16: Trace Debugging in Academics – the Future of Multicore ... · 2 Technische Universität München Philipp Wagner Institute for Integrated Systems Well-known roads h t t p: / / e](https://reader035.vdokument.com/reader035/viewer/2022062607/604b933271ce8d23a75cb9a9/html5/thumbnails/16.jpg)
16
Technische Universität München
Philipp WagnerInstitute for Integrated Systems
Think! On Chip.
Attach meaning to data on chipReduce amount of data to send off chip
![Page 17: Trace Debugging in Academics – the Future of Multicore ... · 2 Technische Universität München Philipp Wagner Institute for Integrated Systems Well-known roads h t t p: / / e](https://reader035.vdokument.com/reader035/viewer/2022062607/604b933271ce8d23a75cb9a9/html5/thumbnails/17.jpg)
17
Technische Universität München
Philipp WagnerInstitute for Integrated Systems
From Data to Information: First Road Signs
• Information from the software developer– ARM CoreSight System Trace Macrocell (STM),
NEXUS 5001-2008 Data Acquisition Messaging (DQM)– like printf()/kprintf() with hardware support
• Configurable trace collection– Infineon MCDS
• Recognizing known bugs– On-chip Data Race Detection [Wen et. al, 2012]
C.-N. Wen, S.-H. Chou, C.-C. Chen, and T.-F. Chen, “NUDA: A Non-Uniform Debugging Architecture and Nonintrusive Race Detection for Many-Core Systems,” IEEE Trans. Comput., vol. 61, no. 2, pp. 199 – 212, Feb. 2012.
![Page 18: Trace Debugging in Academics – the Future of Multicore ... · 2 Technische Universität München Philipp Wagner Institute for Integrated Systems Well-known roads h t t p: / / e](https://reader035.vdokument.com/reader035/viewer/2022062607/604b933271ce8d23a75cb9a9/html5/thumbnails/18.jpg)
18
Technische Universität München
Philipp WagnerInstitute for Integrated Systems
From Data to Information: Take it a Step Further
1. Give the developer a way to express the meaning of the data 2. Add on-chip hardware to actually extract information out of
data
![Page 19: Trace Debugging in Academics – the Future of Multicore ... · 2 Technische Universität München Philipp Wagner Institute for Integrated Systems Well-known roads h t t p: / / e](https://reader035.vdokument.com/reader035/viewer/2022062607/604b933271ce8d23a75cb9a9/html5/thumbnails/19.jpg)
19
Technische Universität München
Philipp WagnerInstitute for Integrated Systems
Express the Meaning of Data# increment a drop counter for every
# location we drop a packet at
probe kernel.trace("kfree_skb") {
locations[$location] <<< 1
}
# Every 5 seconds report our drop locations
probe timer.sec(5) {
printf("\n")
foreach (l in locations-) {
printf("%d packets dropped at %s\n",
@count(locations[l]), symname(l))
}
delete locations
}
Data point: Linux kernel function kfree_skb() is called
#> stap dropwatch.stp3 packets dropped at 0xffffffff81495cfb
9 packets dropped at 0xffffffff81495cfb1 packets dropped at 0xffffffff8154da4c
4 packets dropped at 0xffffffff81495cfb2 packets dropped at 0xffffffff814f2100
Information: dropped TCP packets
SystemTap example from https://sourceware.org/systemtap/SystemTap_Beginners_Guide/useful-systemtap-scripts.html
![Page 20: Trace Debugging in Academics – the Future of Multicore ... · 2 Technische Universität München Philipp Wagner Institute for Integrated Systems Well-known roads h t t p: / / e](https://reader035.vdokument.com/reader035/viewer/2022062607/604b933271ce8d23a75cb9a9/html5/thumbnails/20.jpg)
20
Technische Universität München
Philipp WagnerInstitute for Integrated Systems
Now put it in hardware
![Page 21: Trace Debugging in Academics – the Future of Multicore ... · 2 Technische Universität München Philipp Wagner Institute for Integrated Systems Well-known roads h t t p: / / e](https://reader035.vdokument.com/reader035/viewer/2022062607/604b933271ce8d23a75cb9a9/html5/thumbnails/21.jpg)
21
Technische Universität München
Philipp WagnerInstitute for Integrated Systems
Now put it in hardware
![Page 22: Trace Debugging in Academics – the Future of Multicore ... · 2 Technische Universität München Philipp Wagner Institute for Integrated Systems Well-known roads h t t p: / / e](https://reader035.vdokument.com/reader035/viewer/2022062607/604b933271ce8d23a75cb9a9/html5/thumbnails/22.jpg)
22
Technische Universität München
Philipp WagnerInstitute for Integrated Systems
Think! On Chip.
![Page 23: Trace Debugging in Academics – the Future of Multicore ... · 2 Technische Universität München Philipp Wagner Institute for Integrated Systems Well-known roads h t t p: / / e](https://reader035.vdokument.com/reader035/viewer/2022062607/604b933271ce8d23a75cb9a9/html5/thumbnails/23.jpg)
23
Technische Universität München
Philipp WagnerInstitute for Integrated Systems
Think! On Chip.
Find patterns.From information to knowledge?
![Page 24: Trace Debugging in Academics – the Future of Multicore ... · 2 Technische Universität München Philipp Wagner Institute for Integrated Systems Well-known roads h t t p: / / e](https://reader035.vdokument.com/reader035/viewer/2022062607/604b933271ce8d23a75cb9a9/html5/thumbnails/24.jpg)
24
Technische Universität München
Philipp WagnerInstitute for Integrated Systems
From Information to Knowledge
event1(INFO) → event2(INFO) → event3(FATAL)
don't care don't care oops! too late.
this sequence repeats for a couple times
![Page 25: Trace Debugging in Academics – the Future of Multicore ... · 2 Technische Universität München Philipp Wagner Institute for Integrated Systems Well-known roads h t t p: / / e](https://reader035.vdokument.com/reader035/viewer/2022062607/604b933271ce8d23a75cb9a9/html5/thumbnails/25.jpg)
25
Technische Universität München
Philipp WagnerInstitute for Integrated Systems
From Information to Knowledge
Be careful. A fatal error might be coming up. I saw it coming!
event1(INFO) → event2(INFO) → event3(FATAL)
![Page 26: Trace Debugging in Academics – the Future of Multicore ... · 2 Technische Universität München Philipp Wagner Institute for Integrated Systems Well-known roads h t t p: / / e](https://reader035.vdokument.com/reader035/viewer/2022062607/604b933271ce8d23a75cb9a9/html5/thumbnails/26.jpg)
26
Technische Universität München
Philipp WagnerInstitute for Integrated Systems
Pawlow's Dog: Reinforcement Learning
Can we bring machine learning into the chip?
http
://w
ww
.flic
kr.c
om/p
hoto
s/le
arto
on/3
4089
5280
5Fl
ickr
use
r “N
ikol
a B
”, C
C-B
Y
![Page 27: Trace Debugging in Academics – the Future of Multicore ... · 2 Technische Universität München Philipp Wagner Institute for Integrated Systems Well-known roads h t t p: / / e](https://reader035.vdokument.com/reader035/viewer/2022062607/604b933271ce8d23a75cb9a9/html5/thumbnails/27.jpg)
27
Technische Universität München
Philipp WagnerInstitute for Integrated Systems
Summary: The Road to Multicore Debug
• Less run-control debug, less full system tracing in the future• Put debug intelligence inside the chip!• Give meaning to data: “Debug Coprocessors”• Find recurring patterns: Machine learning
![Page 28: Trace Debugging in Academics – the Future of Multicore ... · 2 Technische Universität München Philipp Wagner Institute for Integrated Systems Well-known roads h t t p: / / e](https://reader035.vdokument.com/reader035/viewer/2022062607/604b933271ce8d23a75cb9a9/html5/thumbnails/28.jpg)
28
Technische Universität München
Philipp WagnerInstitute for Integrated Systems
Thank you for your attention.
Your Thoughts?
Philipp WagnerInstitute for Integrated Systems, TUM