TFS 2017 – Sometimes we must look back before we can move forward! – Part 1
This is the first of a series of posts is provided by Senior Application Development Manager (ADM), Chuck Goodspeed, focusing on the challenges encountered in migrating/evolving TFS 2005/2008/2010 installations a full decade forward into TFS 2017 and/or VSTS. Experiences and situations encountered at multiple customer installations provide real-life NOT “out-of-the-box experiences”, [! OOB} which can be built upon to avoid common pitfalls.
Background
My primary goal as an ADM is to provide value to my customers and help them succeed with our technologies. I continually seek new ways to meet more development managers, teams and individual developers. Each new person or group becomes my logical foundation for growth when I can prove that Microsoft can help them meet their goals and accommodate their need for diversity.
My current customers are large financial institutions with huge development footprints. The software developed includes C++ unmanaged code projects, various versions of .NET applications, Maven JAVA projects, ANT JAVA projects, Python projects, Database Development projects (SQL Server, Oracle, Sybase, DB2, Teradata and open source databases.
The developers are on-shore, off-shore, employees, consultants and /or contractors. Each project HAD a team and owner but not all projects HAVE a team and owner. Yes, many large organizations have “code artifacts” that are in production but the team or developer who either authored the code or maintained it are long since gone! Sometimes the code “running” in productions does NOT even match the last/only copy of the source!
Development and Operations managers are under pressure to bring these environments under control. Their efforts are often complicated when financial pressures move support and subject matter expertise offshore. They are also besieged by new independent internal development groups who want to throw out the old and bring in the new! They often advocate open source alternatives including alternate version control systems, alternate build tools and alternate deploy tools.
It is always easier to move forward when one does not have to accommodate the past and/or preserve historical artifacts. Migrating older application to newer versions can be relatively simple and straight forward provided that the older versions “colored between the lines” and stayed “within the box”. Migration guides are plentiful to guide the lucky ones forward. Unfortunately, “life is what happens when are planning something else”! Some clever soul in the past heavily customized the installation and them complicated that further by taking employment elsewhere. This makes it more difficult to go forward and unfortunately leaves the doors wide open to those who advocate throwing it all out!
Goal
I have worked and managed in large development organizations. I have worked in non-Microsoft primarily open source institutions, managed C++ organizations and just about every type of Microsoft .NET and before development organizations. I have used a variety of Version Control Systems including SCCS, RCS, Subversion (SVN), Rational Clearcase, Perforce Helix, PVCS, Vault, Visual SourceSafe, TFS and VSTS. It is easy to claim that each product is completely different. That is simply not true! Each product must ultimately deal with a common set of principles even though they solve them in different ways. If we abstract the concepts we can identify each product and/or product module as a virtual component.
The virtual components can be “plugged” into the abstract circuit board that defines the flow of our DevOps strategy. This flow expressed in a swim lane transitions from Version Control to Development to Build and finally to Deploy. I could use TFVC or Git for versions control. I could use Visual Studio, VS Code, Eclipse or other tools as Dev Tools. I can use MS Build, Visual Studio Build, Maven, ANT, Gradle, TeamCity, Cruise Control and a myriad of others as Build Tools. For CI/CD I might use Jenkins, Bamboo, TeamCity or TFS. I can use Release Management, Puppet, Chef, Ansible, SALT or others to release and/or deploy my applications. I should be able to allow internal development teams the opportunity to select which tool they want to use for their own projects but still remain in control of the process.
Managing multiple tools and products continues to be a challenge for most organizations. I believe that TFS/VSTS can be used efficiently and effectively to perform, manage and/or facilitate the development, build and deployment of nearly all types of applications. This includes integrating the use of other tools into the flow of a TFS DevOps environment. There may be some customization required.
Microsoft’s and my own credibility are enhanced when I acknowledge the value of other products without continually insisting on using a comparable Microsoft product instead. When we become competent in integrating other tools into our own and acknowledging the value of those tools we are not excluded from strategy sessions but instead are more frequently invited. I often spend more time investigating the integration off other tools than working with Microsoft products.
My goal in this series of posts is to create a VSTS or TFS on-premises or hybrid environment where we are facilitating the DevOps process by integrating properly with the customer’s choice of tools. The more familiar we become with those other tools the more value we can bring to our customers. The less insistent we become avoiding integration with other tools the less often we will get the request to do so!
Challenges
TFS 2017 and VSTS provide excellent platforms to begin the development of a fully integrated and comprehensive DevOps platform that can be readily integrated with other products. The first and sometime the most difficult challenge in the entire process is bridging the gulf between TFS 2005 or TFS 2008 and the modern TFS 2017 environment.
Some of my current TFS 2017 environments are still embroiled in transition from the old XAML build system to VNext builds provided through the Web Portal. Many existing TFS installations are not on XAML builds yet. They are using the older Team Build System which relies on tfsbuild. proj files for each Team Project stored in a folder call TeamBuildTypes. These tfsbuild. proj files import custom MS Build .targets files that in themselves use tasks and properties.
Both TFS 2005 and TFS 2008 must be first migrated to TFS 2010 prior to proceeding forward in the migration steps. TFS2010 must be migrated to either TFS 2013 or TFS2015 prior to migrating to TFS 2017.
The simplest migration is from an installation that did not use Reporting Services, Analysis Services, SharePoint Integration, automated Builds, Gated Check-ins, TFS Event Subscriptions, Custom Process Control Templates, Custom Work Items Types, automated deployment and so on. These types of installations can easily be migrated following the excellent documentation currently available.
Customer migration issues no matter how complex the customization can be dealt with provided that you know each issue that is in play.
Ask the following questions prior to the migration planning:
General
- What version is being migrated? (2005, 2008, 2010, 2012, 2015)
- How many collections are being migrated?
- How large are each collection database?
- Is this an in-place upgrade?
- Is the installation being migrated to new hardware at the same time?
- Is the installation being moved to a new Active Directory Domain?
- Is the installation being move to a new Domain on new hardware?
- Is the topography being changed?
- One or more app servers?
- Additional build servers?
- Test automation servers?
- Release Management Servers?
- InRelease Servers
- Separate database server?
- Is this an SSL based installation?
- Will the new installation be based on SSL?
Reporting
- Is Reporting Services being migrated?
- Is Analysis Service being migrated?
- How many custom reports are being migrated?
- Are there custom reports using the OLAP cube?
- How large is the TfsWarehouse database?
- Will you rebuild TFS Warehouse after the migration?
- Will you rebuild TFS Analysis after the migration
- Is Reporting Services current installed on same server as TFS Databases?
- If so will it be moved to a new server?
- Is Analysis Services current installed on same server as TFS Databases?
- If so will it be moved to a new server?
SharePoint
- Is SharePoint integrated with TFS?
- How many content databases are in SharePoint installation?
- Is SharePoint Report Services integrated?
- Is Excel Services defined in SharePoint?
- Is SharePoint currently installed on the new server?
- If so, will it be moved to a separate server?
- Is Search setup on SharePoint server?
Event Subscriptions
- Does the installation support TFS Events subscriptions?
- Registered through bissubscribe?
- Added programmatically
- Does the installation support email handling for subscribed events?
- Is the SMTP server on the TFS server?
- Does the application support build completion event subscriptions?
- Does the application support work item update events?
- Is there a web service defined to handle TFS events?
TFS 2005
- How many build definitions are defined in TFS per team project?
- Does each build definition have its own tfsbuild.proj defined?
- Has the customer modified any Microsoft .targets files?
- Has the customer created any custom .targets files?
- Has the customer created any custom tasks?
- Has the customer created any custom properties?
TFS 2008
- How many build definitions are defined in TFS per team project?
- Does each build definition have its own tfsbuild.proj defined?
- Has the customer modified any Microsoft .targets files?
- Has the customer created any custom .targets files?
- Has the customer created any custom tasks?
- Has the customer created any custom properties?
TFS 2010
- How many build definitions are defined using TFS upgrade template?
- Does each build definition have its own tfsbuild.proj defined?
- Has the customer modified any Microsoft .targets files?
- Has the customer created any custom .targets files?
- Has the customer created any custom tasks?
- Has the customer created any custom properties?
- Does the installation support Maven builds?
- Has the customer modified the Maven .targets files?
- Does the installation perform ANT builds?
Process Control Templates and Work items
- Does the installation use work items?
- For requirements?
- For bugs?
- Has the customer customized the process control templates?
- Has the customer modified or added custom work flows?
- Has the customer add new column types?
- Do “same type” work item types in different Team Projects vary in content?
- Do same name columns have different column type?
- Are there custom date type columns?
- Are there custom “user” (ApprovedBy, VerifiedBy) type columns?
There are a number of other questions that can affect the timeline of the migration. Many of the items noted above may not affect the migration but delay the validation of the new environment. The worst case is not to know? Sometimes this cannot be avoided because the customer themselves do not know as well. Their environments have been setup be contractors or long gone employees. The rationale behind some of the more esoteric decisions is no longer clear.
Unfortunately, we may only discover these issues by falling into them. I have always believed that:
- Good judgement comes from experience
- Experience comes from bad judgement
As humans, we are doomed to learn from our mistakes. That said, through this series of posts, I will detail my mistakes or bad judgement and note the experience points I earned. I will trade in those experience points along the way for good judgement and demonstrate the results.
I am a person who learns from an examples. I generally prefer a more complex example as opposed to a simple one. I have found that simple examples only address simple situations. So far it has been my luck in life not to be dealt too many simple solutions to deal with. I will provide more robust examples that the reader should be able to distill down rather that simple examples that do not contain enough substance to scale up.
Looking forward
The succeeding blog posts in this series will address the following scenario:
Hypothetical customer 1:
- TFS 2010 Installation with a 500+ GB collection
- 200 + Team Projects
- All projects use TFS 2010 upgrade template
- Every process control template was duped and modified
- Multiple custom work item types
- Same work item type can be customized in different Team Projects.
- Multiple custom column types
- Integrated with SharePoint 2010 on TFS App Server
- Two content databases
- 3 TFS Build Servers
- Full SSRS and SSAS installations
- 45 % Team Project in .NET
- 50% Maven Projects
- 5 % various other project types
- MS Build .targets files modified
- Maven target files more heavily modified
- 8 additional extensive custom target files
- 15 custom MS Build Tasks developed in C#
- 300 GB of custom tooling and maven repositories on each build machine
- Multiple versions of WebSphere supported
- All deployments done by a custom open source deployment tool
- Tool is 5 years out of support
- All deployments are managed by a custom ASP.NET application that uses work items to advance through the approval and deploy phases.
- 3 TFS Events are current subscribed to.
- A custom web service catches the events and over a 1000 lines of code drives the deployment process
- New environment
- Move to new servers
- Change versions of SQL Server (2016 for TFS 2017, 2014 for SharePoint 2013)
- Separate data servers for TFS, SharePoint, SSRS and SSAS
- Additional App Server
- Additional Build Servers
- New Linux Deploy Server
- Integrate Jenkins
- Integrate Git
- Integrate Subversion
- Setup plan to move all builds from XAML to VNext staggered over 1 year
- Educate offshore support over same length of time.
- Migrate deployment to Ansible and/or Octopus Deploy both called by Release Management.
Next article in series, TFS 2017 – Sometimes we must look back before we can move forward! – Part 2 – Setting up the test environment
Premier Support for Developers provides strategic technology guidance, critical support coverage, and a range of essential services to help teams optimize development lifecycles and improve software quality. Contact your Application Development Manager (ADM) or email us to learn more about what we can do for you.
Light
Dark
0 comments