I just returned home from three weeks of hard travel covering nine cities in seven countries in a little over 14 days. Along with an acute case of jet lag, I find myself enjoying the unique perspective that one can only gain from such a trip. As a semi-reluctant blogger (I thoroughly enjoy writing these once I am finished but usually dread them up to that point), I figured this would be an ideal time to share some thoughts while the perspectives and the jetlag are at their peak.
Overwhelming would be a good description for the broad range of viewpoints and strategies that I came across meeting dozens of organizations at various stages of implementing software security initiatives – thankfully most all of the companies I met are evaluating our products in the process. (Due to the great work of our international sales teams and many global partners our customer count overseas will rival that of the US by year end, although the implementations here at home tend to be much larger and generally deeper into the development organizations). In spite of the many different approaches, I am starting to clearly see the great divide that separates the successful programs from the train wrecks and in a word it is “accountability” - lets explore this a bit.
On the 14-hour plane flight from Sydney to SF I was able to catch up on quite a bit of reading, including my favorite trade magazines and my subscription to the Economist, which can stack up mercilessly. During the flight, I came across an excellent feature article written by Jennifer deJong from SD Times published on August 15th “Slipping in The Side Door With App Security Message”. The article is spot on, describing the challenges many organizations face in making application security a real priority in development in an attempt to put it on par with the classic “big three” development drivers - features, dollars, and dates. Jennifer has written as much on the topic of application security as anyone and has been instrumental at shedding light on this topic in the development community over the last several years. Her story was complimented recently by a survey of 400 US based software developers released by Symantec today (URL) that shares some troubling data in regards to just how far we have to go. According to the Symantec survey:
“93 percent (of developers) felt that secure application development was more of a priority now than three years ago. Also 70 percent indicated that their employers emphasize the importance of application security, 74 percent indicated that security was a high priority in their development process..”
Sounds great, right? Not so fast, it is goes on to state:
“only 29 percent stated that security was always part of the development process.”
That basically sucks. So, just about everyone knows its important, at nearly 3/4’s of all companies the folks paying the bills want us to do something, but we are fulfilling those wishes once in a while (29%) and thus dropping the ball 71% of the time – that’s a failing grade. So what gives? Some might throw in the famous quote relating the management of programmers to the herding of cats and chalk it up to that, but there has to be something more fundamental at work here. What are a small subset of companies doing right that a much larger group are getting wrong? I believe that a handful of conversations during my trip shed a most illustrative light on this question. They also give me reason for concern that the information technology advantage we enjoy here in the United States might be threatened by the very same factors that have traditionally given us a competitive advantage. In spite of the popular rumors of the demise of the US software industry due to lower cost offshore competitors, US companies account for nearly 80% of worldwide software product revenues. I believe that security issues can have a much greater affect on the industry than cheaper labor in foreign markets. We can, and do, leverage the latter to our benefit, but if US development organizations are endemically challenged when it comes to creating secure code, well that opens up a significant competitive problem for the US software industry and a serious risk for the many industries that depend on it.
The Wild West
One insightful conversation towards the end of my trip (which took me to the east coast of the United States) was with a gentleman who runs security at a US financial services infrastructure firm employing several hundred developers. He told me that he had a “controversial” outlook on application security. He explained that in his organization they considered the problem to be an organizational one of culture, awareness and knowledge in the development organization not a technology problem – “tools are not the answer” he exclaimed. I think it startled him when I (a maker of tools) agreed, “tools are not the answer”. I shared with him one of my favorite quotes about tools and their relative place in the overall scheme of things:
“The world’s finest hammer does not build a house, it does give you the incentive or motivation to build a house, it does not teach you how or why we build houses. However, if you do have the knowledge, desire, motivation and you are going to be held accountable for building a house well and efficiently, then you are **really** going to want one. Good tools help you get things done better, faster, cheaper – if you have no intention on doing the work then they are not going to help nor will you see value in them.”
While tools are not the answer, they are a part of the equation – in my opinion the last part of a three-part puzzle. I think this gentleman was 1/3 of the way there. Indeed a culture, awareness, and knowledge are all good things. But this is software development and we have the “big three” (features, dates, dollars) to contend with. Culture, awareness and knowledge are all a part of the big three too. In fact most of us that manage software projects have graduate degrees and years of apprenticeship that delve into these concepts with great precision. But the big three also have another thing going for them and that is accountability. As a development manager the execution of any project against features, dates, and dollars is not going to go un-noticed by those paying the bills. Those pesky users tend to notice when we don’t deliver the functionality that they asked for (even the implicit features they didn’t ask for). Miss a date by a significant margin or go way over budget and someone is definitely going to notice that too. Consequences? I am a development manager, that is my profession and that profession is all about trade-off’s. Make the right ones and you are a hero, make the wrong ones and your looking for a new job. Do that too many times and you are not ever going to work in the “A” league. Accountability for the big three basically boils down to the people that make a fortune in this industry and those that wash out. Only someone watching from the sidelines could see the big three as absolutes. This is an industry where smart guys and gals that could not carry a tune, throw a touchdown, or deliver a punch line make as much as any rock star if they make those trade-offs exceptionally well. The accountability could not be more extreme. So what about security? Where is the accountability and who is going to place it? If you train me and “build the right culture” but in the end you don’t check the results for security guess which of the “big four” (features, dates, dollars, and security) is going to get traded out every time? Welcome to the 71% club. So what are the other 29% up to? They combine the education and cultural aspects with some controls to check security (which unlike the other three is the “hidden property” of any piece of software –it is not as obvious as the other three unless of course you are a world class hacker).
Let me contrast that discussion with what I encountered over and over again in Asia. Categorically companies there were a bit behind in the general awareness of software security, but not nearly as much as one might expect. Typically we expect IT technology moving from the US to foreign markets in a rather leisurely pace with a lag time that can be as much as five years. A friend of mine in Japan explained it this way. “In the long run it is better for us to wait and see which of the new technologies pan out in the US before we commit to any of them over here. That allows us to derive the productivity gains when they have most of the problems worked out and we also figure out which companies and solutions are going to be around for the long term.” Smart strategy, but the hackers aren’t allowing us to play that game with software security. Instead of waiting five years to wreak havoc on systems outside the US, they are busy as can be driving the application security market in parallel to the market here. Software risk reduction is important in just about every country in the world today (one can only imagine how frustrating it must be to keep a decent server up and running in North Korea right now). There is a bit of lag time on the “ah-hah” that the root cause of today’s security problems is software and the prevailing strategies of patching it or solving it with network security solutions are fundamentally flawed. That message is not a tough one to digest and people are figuring it out quickly. Nearly every firm I met with had gotten past this initial awareness and assembled detailed plans complete with process diagrams depicting control points at various stages of the software development lifecycle (what we call security assurance gates at Fortify) ensuring that secure development activities were being followed and that software was verified to be free of vulnerabilities. In one of the “Four Dragons”, I met with a banking regulator that was not only extremely well versed in software security but had clear intentions of adding requirements for verifiable proof of security results for a wide range of supporting software as soon as 2007 – (I refrain from mentioning the specific country in respect of the privacy of the individuals I met with). The bottom line is that first thing nearly all of these companies sought to address was the accountability issue, then the awareness and training, followed by tools. Should that be a surprise? There is obviously some fundamental differences in the US software community that are not found in other places hence the exceptional performance. Oddly enough I believe those differences are cultural. However these differences that give an advantage when it comes to bringing innovative ideas into the marketplace at break-neck speed could actually be our undoing in a world where security of the software becomes a major requirement. Where we assume the artists will be inspired by knowledge and compelled by the desire to do the right thing, the people I met in Asia felt compelled to design a process that ensured the right thing would be done in a repeatable, dependable manner. Of course I think their results will be all the more fruitful with proper training, awareness, and of course high quality tools!
