devday 2016: artur speth - devops - microsoft developer divisions weg ins nächste agile zeitalter

4,307

467

Spread out across 35 feature teams

ProductionDevelopment

Backlog

Requirements

Visual Studio& TFS

Update 1

Visual Studio& TFS

Update 2

Visual Studio& TFS

Update n

VS Team Services

Code Test & Stabilize Code Test & Stabilize

Beta RTM

2 years

Planning

Customer feedback – we should

change the way a feature works. We

didn’t get it quite right…

… but we’re booked solid already. 2 years

S1 S2 S3 S4 S5 Stabilization S6

A

B

S7 S8

2 years

3 weeks

https://flic.kr/p/arXUyP

Alignment

Autonomy

“Let’s try to give our teams three things…. Autonomy, Mastery, Purpose”

Scenarios

Features

Stories

Tasks

Sprint3 week

3

Plan3 sprint

Season6 month

Scenario18 month

3 6

SpringFallSpring Fall

Aspirational

60%

Sprint3 week

Plan3 sprint

3

Season6 month

Scenario18 month

3 6

SpringFallSpring Fall

Hopeful

80%

What Epics are we lighting up

Sprint3 week

3

Plan3 sprint

Season6 month

Scenario18 month

3 6

SpringFallSpring Fall

Thoughtful

90%

What features are planned?

Sprint3 week

3

Plan3 sprint

Scenario18 month

3 6

SpringFallSpring Fall

Confident

95%

What stories are we complete? What features are shipping?

Season6 month

Week 1 Week 2 Week 3

Week 1 Week 2 Week 3Week 2 Week 3

Sprint 98

Sprint 97 Sprint 99

The sprint plan What we accomplished

• Updates were large

• Months apart

• Lots of problems!

4/1/2010 4/23/2012

5/3/2010

TFS 2010 RTM

4/23/2011

Service Deployment

8/5/2011

Service Update

9/26/2011

//BUILD 2011

12/7/2011

Service Update

1/30/2012

Service Update

2/20/2012

Service Update

3/12/2012

Service Update

4/2/2012

Service Update

Program Management Development Testing

Operations

Program Management Engineering

Operations

Engineering

Program Management Engineering

Week 1 Week 2 Week 3

Week 1 Week 2 Week 3Week 2 Week 3

Sprint 98

Sprint 97 Sprint 99

DeploymentSprint Planning

Done

Week 1 Week 2 Week 3

Week 1 Week 2 Week 3

Week 1 Week 2 Week 3

Week 1 Week 2 Week 3

ONE

Code Test & Stabilize Code Test & Stabilize

Beta RTM

Planning

Code

Complete

ON

OFF

ON

OFF

ON

OFF

ON

OFF

ON

OFF

ON

OFF

VSO SU1

Chicago

VSO SU0

San Antonio

VSO SU4

Amsterdam

Shared Platform Services

San Antonio

Existing experience Baseline:

36% conversion to project

50% to 100% customers

conversion to project (+18%)

There’s no place like production!

Telemetry everywhere

Customer IntelligenceBusiness IntelligenceOperational Intelligence

Dashboard DevOps Debug Experiments

Getting the availability model right

0,8

0,82

0,84

0,86

0,88

0,9

0,92

0,94

0,96

0,98

1

-200

0

200

400

600

800

1000

1200

1400

1600

9.25.13 2:24 PM 9.25.13 3:36 PM 9.25.13 4:48 PM 9.25.13 6:00 PM 9.25.13 7:12 PM 9.25.13 8:24 PM 9.25.13 9:36 PM 9.25.13 10:48 PM

Sept 25th 2013 LSI

FailedExecutionCount SlowExecutionCount Start End Availability (ID4 - Activity Only) Availability (Current)

Alerting is key to fast detection

Every alert must be actionable and represent a real issue with the system.

Alerts should create a sense of urgency –false alerts dilutes that

Redundant alerts for same the issue

Needed to set right thresholds and tune often

Stateless alerts contributed to further noise

Health model in action

• 3 errors for memory

and performance

• All 3 related to same

code defect

• APM component mapped to feature team

• Auto-dialer engaged Global DRI

Eliminated alert noise

~928 alerts per week to

~22 and reduced DRI

escalations by ~56%

Live Site Issues (LSIs)

Time to MitigateTime to Detect

% o

f In

cid

en

ts

DRAFT

DRAFT

Microsoft Confidential 52

Service Availability & Health Metrics

DRAFT DRAFTDRAFT

Inci

den

t C

ou

nt

Inci

den

t C

ou

nt

DRAFT

DRAFTDRAFT

% o

f In

cid

en

ts

Use

r M

inu

tes

DRAFT

DRAFTDRAFT

Error By SourceIncidents by SeverityUser Impact Minutes During Incidents [TFS

Only]

3

2

1

4

1. TFS Availability is on an improving trend. No Sev0/Sev1 LSIs for July.

2. App Insights switched from synthetic availability to real-user experience in Ibiza portal. A high

volume of SEV-2 LSIs (72) contributed to customer impact in addition to intermittent UX errors.

(UX fixes applied on 8/11 that improves availability)

3. App Insights was impacted by 3 long running LSIs related to ES maintenance, Ibiza updates and

an Azure Storage outage.

4. TFS Service attainment (SLO) improved significantly MoM with focus on minimizing failed/slow

commands and reviewing in weekly LiveSite reviews

Service status

© 2015 Microsoft Corporation. All rights reserved.