How the Visual Studio Team Services Team Uses its Own Product

One of the sessions that I attended at Build 2016 was titled “DevOps at Scale: A True Story”. It was a really interesting presentation from Microsoft’s Principal Group Program Manager for Visual Studio Team Services (VSTS) about how they use their own product, VSTS, to build VSTS! Other Microsoft divisions also “dog food” VSTS quite a bit. As of the time of the presentation, they had 48,629 internal Microsoft VSTS users!
The VSTS team is made up of 436 people, divided into 35 different feature teams spread out around the globe. The goal of the VSTS team is to deliver value to customers more quickly, increase cycle time, and to get their work into their customer’s hands a lot faster. Although it used to take them 2 years to deliver product updates, they have found a way, by improving their processes and tooling (i.e. VSTS), to now deliver updates every 4-5 months by way of 3-week sprints.

The Challenges of a Large Global Team
With such a large global team, it was really hard to keep all of the individual teams aligned. They eventually focused on the following guiding principles: Organization, Roles, Teams, Cadence, Taxonomy, Plan, and Practices. They were able to use some of the ideas from the book “Drive” by Daniel Pink to give their teams drive, autonomy and purpose. If one team liked scrum, then they used scrum. If another one wanted to do extreme programming, then they did that. The project management team focused on taxonomy and staying aligned – as long as every team and business tracked scenarios and features consistently then they were able to roll things up in VSTS and see where they were. Some of the alignment practices they adopted were:

• Feature chats – bring in the teams explain what they are doing in their 3-week sprints…face to face is important here.
• Sprint mail – the team sends an email to their PM at the end of their 3-week sprint with a summary of features completed. They also make videos of their end of sprint demos that are embedded in the email!
• Kanban – used a lot for showing the backlog. They also use color-coded tags.
• Each feature is linked directly to the specifications (storyboard).
• Every team customizes their own Kanban boards and dashboards.
• They track what has been deployed in VSTS…each time something is deployed the PM has to write the release notes for the feature.

Revolutionize Your Business With Generative AI

From product design and software development to virtual agents, content creation, and reporting, GenAI is transforming business. Our AI experts help you unlock GenAI’s full potential and drive growth.

Let’s Get Started

Deployments
In the old days of the VSTS application lifecycle, deployments were really difficult and painful. They always had issues and they took a long time, sometimes up to 2 years! They had a really difficult time merging their various teams’ code branches together. After implementing agile practices and doing 3-week sprints, they were able to shorten the deployment cycle to 4-5 months. Finally, once they made the decision to deploy at the end of every sprint, it forced them to face the painful deployment issue head-on. At first it was difficult, but then they started to build their deployment “muscle”.
Some of the things that they changed to shorten their deployment cycles were:

• They started using Git – they put everyone on the master branch. They realized that when dev teams created their own branches it was really hard to merge their changes back in. They found that Git was really good for lightweight topic branching, and tiny, continuous merging. They could create a tiny branch, and then do a pull request to get the changes into the master. They also setup a pull request policy with required reviewers. Once the request was approved, then it would get merged back into the master branch. They found that this was great for hot-fixes
• Accountability – who’s responsible for what’s in PROD? They merged their DEV and Test teams, and brought in Operations also.
• Team rooms – moved from offices into a team room…the room only fits people on one team.
• Feature flags – this is for code that is not ready to get deployed to production. They can keep deploying incomplete features, and the team can stay in the master branch. Think of them as windows registry entries. Launch Darkly is who does the add-on for feature-flagging.
• Release Management – they use VSTS Release Management to deploy VSTS!
• They use heath dashboards to see the current state of things
• They want to keep deploying to avoid technical debt, and they don’t want to be risk averse. They adopted the mantra that everything must go to production.

The Bottom Line
Everyone is happier when the code they write gets into the hands of customers at a rapid pace!