TD Bank has prioritized convenience and customer service for its clients for more than 150 years. The bank opened its doors in 1852, as Portland Savings Bank. Over the next century and a half, the company grew through a series of mergers and acquisitions. By the early 2000s, the bank was known as BankNorth, and it caught the eye of TD Bank Group of Toronto, Canada. The Canadian multinational quickly became the major shareholder of BankNorth, and then finished the acquisition by 2007.
In terms of its software development methodology, TD Bank's application development organization followed a traditional waterfall approach until 2014. It was around this time that they recognized a need to modernize, both in terms of approach and tooling. So, they embarked on an agile transformation.
Bill McArthur, development engineering lead, and Sladjana Jovanovic, VP of enterprise payments technology, were both key participants in the large team effort that pioneered this transformation. For part of this modernization, they turned to Sonatype’s Nexus Platform, including Nexus Lifecycle and Nexus Repository. Jason Hills, who joined as Head of Application Security, helped them further advance their usage of this tooling.
As any veteran of such an initiative will attest, an agile transformation is, in and of itself, a serious undertaking. All too often, the need for change goes much deeper than following a different project management methodology and having different team meetings. Organizations need to change their philosophies, technical practices, and tooling.
This need for change beyond agile adoption presented TD Bank with its first major challenge: adopting tooling and automation that would allow an agile delivery cadence. As McArthur put it, to work in an agile way, they needed modern tools.
Recognizing the need for automation was only part of the story. The next step was to look at specific shortcomings, pick the right tools to address those shortcomings, and then build a business case for the tools. Enterprise transformations rarely happen monolithically, and TD Bank was no exception. As early advocates for the transformation, McArthur and Jovanovic had to address these issues in proof-of-concept situations and then help spread the story to other areas of the business—to make these solutions enterprise-standard.
For example, one specific challenge was that they had a build and deployment process that had a cumbersome, manual process for nonrepudiation. Without a timely nonrepudiation of build components, or actual software bill of materials, it takes a lot longer to trace the software in production back to its original source. This is an untenable situation for teams that are deploying, maintaining and managing software while meeting increased continuous delivery demands.
Another challenge they faced was governance and approval for open-source component adoption. This is how McArthur described the situation:
"There was a centralized group responsible for ensuring that for all open source, they understood who was using it, why they were using it, where they were using it, and which versions are allowed. You had to get approval to use anything."
This process had a long cycle time and was labor-intensive. An entire group of people would have to do manual research while the different application development groups simply waited on their findings. This created an unfortunate incentive, as described by McArthur: the process to gain approval for the use of any open source components was painful. So naturally, the organization observed a limited adoption rate.
Despite these challenges during the early days of the agile transformation, McArthur, Jovanovic, and TD Bank were able to build business cases for their solutions and make a great deal of progress. But with progress comes new challenges.
When Hills joined TD Bank in 2018, several years later, he encountered a pervasive application security challenge. While TD Bank had more data than ever before about release security and vulnerabilities, that data was not calibrated to business risk. SAST (Static Application Security Testing) tools reliably identify security defects, however, the tools lack the context to assign meaningful severity scores. More risk-based intelligence is required to create meaningful KRIs and remediation plans. "We continuously tune and customize our automated tools to improve accuracy and then feed those results into a severity customization factory," Hills said.
“Nexus Platform doesn't presume how you want to use it. It provides you with information. It provides you with data and then it gives you the tools to take that information, customize it, and do what you want with it.”
As TD Bank began their agile transformation, McArthur and Jovanovic quickly understood that they would need serious tooling, practice, and philosophy changes. One of the core principles of agile methodologies is delivering working software early and often. For an enterprise with a long history of traditional waterfall development, like TD Bank, this required more than just setting different release milestones. It required a comprehensive tooling overhaul.
TD Bank embraced this, with McArthur and Jovanovic as early adopters. Here is how McArthur described the tool adoption that accompanied their agile transformation:
"In order to work in an agile way, we needed to modernize the tools that we used to be successful. So, we took that same opportunity to do it in the development world, with all of our CI tooling. That's when we adopted an Atlassian stack and took on the Nexus Platform. We started pushing ourselves forward, from a CI point of view."
This tooling was important to solving the challenge of agile—the demand for a faster release cadence. It also gave them the ability to solve other challenges. For instance, this faster delivery cadence created additional challenges for a bank concerning timely nonrepudiation and accounting for production components. By adopting Nexus Repository, TD Bank was able to quickly and definitively trace, as well as account for, all of the components they were delivering to production.
The benefits from improved tooling did not stop there, either. TD Bank was able to shift its governance cycle to the left and become significantly more efficient. By using Nexus Lifecycle, TD alleviated two major pain points: a long, labor-intensive component evaluation process and organizational avoidance of using open source technologies. This turned a manual process that took countless staff hours into an automated one, allowing engineers to self-serve in minutes.
Building on wins like this, McArthur and Jovanovic were able to drive the agile transformation and changes to critical mass. Jovanovic pointed out that, when it comes to organizational change, "if it's happening in the pockets, it just doesn't happen." They built on their momentum and implemented these changes on a high profile organizational initiative. "It took this big initiative to actually propel it forward, structure it, and formalize it," Jovanovic said.
Agile transformation, better tooling, and organizational adoption had reached the tipping point. Even so, however, neither the challenges nor the innovation stopped. Upon joining TD, Hills was appreciative of the tooling and reporting that was in place and immediately identified areas for improvement. TD’s consolidated reporting did not distinguish between code developed internally and code sourced from third-party libraries. While SAST tools do not make this distinction, Nexus Lifecycle does. "Lifecycle is highly customizable, which allowed us to not only identify third-party components, but to also catalogue our internal, proprietary components. Having this level of detail in our firm-wide bill-of-materials is key to understanding our risk posture across the portfolio. If I were starting an Application Security program from scratch, building that bill-of-materials is the first thing I would want to do," Hills said.
As TD Bank continues to improve and conquer new challenges, the Nexus Platform continues to provide the tools that enable them to do so.
TD Bank was able to achieve something impressive in the world of agile transformations. They quickly recognized that switching to a new methodology would demand significant changes to their tooling and infrastructure, and they embraced that from the start.
Because of that approach, DevOps culture followed naturally from their agile transformation. Jovanovic mentioned that they have now formally adopted DevOps practices, and McArthur points out that their early holistic strategy made it so the team was taking a DevOps approach almost from the outset.
With good tooling, solid organizational discipline, and valuable business cases, it seems inevitable in that these changes would gain organization-wide adoption. The fact that it seems obvious in retrospect makes it no less impressive.
The organizational changes are profound. All parties can appreciate it, especially when remembering the way things used to be. Thinking back on the manual component evaluation process that preceded the Nexus Platform, McArthur says, "I can't even imagine what it would be like these days to go back to a manual process. I can't even think of how long that would take because back in the days when we were doing it manually we purposefully used less open source software because it was so painful."
There is perhaps no better sign of a successful transformation than the old ways of doing things becoming unthinkable, thanks to the efficiency of the current state.