1 Raindrop

Who Manages App Gateways? Who Indeed? Yo La Tengo - Call in Security DevOps

Mark O'Neill asked a tough question - "Who Manages Application Gateways?", the heart of the matter is -

So, breaking it down, there are two things going on here: (1) moving integration and security tasks onto network infrastructure, and (2) managing the Gateway as a piece of network infrastructure. It follows logically that there are two distinct roles involved here: (1) The person configuring services and policies on the Gateway, and (2) the person operationally managing the Gateway. Person (1) knows about APIs, apps, and policies. Person (2) does not need to know anything about APIs (much less know anything about REST or XML) but knows all about SNMP, Splunk, load-balancers, and NAT.

I've long described AppSec as a Texas Leaguer problem. For people who are not into baseball - a Texas leaguer is a weak fly ball that two fielders could catch but each thinks the other will get it and so neither runs to it and the catchable ball drops in for a hit. (I try to ask international audiences if there is a cricket equivalent but I have not uncovered one yet).

AppSec is the exact same problem. Developers don't know enough about security and security people don't know about software. So you have this massive gap that allows solveable problem after solveable problem to remain unsolved.

Ops adds another dimension to this problem and perhaps in looking for developers and security people to solve the problems we've been ignoring a leading candidate. The problem that Mark O'Neill describes is a classic DevOps problem and a Security DevOps role would be able to fill a number of these gaps.

The most famous Texas Leaguer happened to the 1962 New York Mets one of the worst teams of all time.

New York Mets center fielder Richie Ashburn and Venezuelan shortstop Elio Chacón found themselves colliding in the outfield. When Ashburn went for a catch, he would scream, "I got it! I got it!" only to run into Chacón, who spoke only Spanish. Ashburn learned to yell, "¡Yo la tengo! ¡Yo la tengo!" instead. In a later game, Ashburn happily saw Chacón backing off. He relaxed, positioned himself to catch the ball, and was instead run over by left fielder Frank Thomas, who understood no Spanish and had missed a team meeting that proposed using the words "¡Yo la tengo!" as a way to avoid outfield collisions. After getting up, Thomas asked Ashburn, "What the heck is a Yellow Tango?"

Security can feel like playing for the New York Mets - there's a lot of ground to cover and many domains that need to work together. AppSec has this problem in spades - deep app knowledge, operational/infrastructure management, assurance, and security protocol expertise to develop the technical policies, all must work together to avoid collisions.

This mix of software, ops, assurance testing, and security protocols is quite important in a Security role more so than almost any IT function because security is a system level property but the mechanisms only work at a functional level. So each mechanism must make sense in a wider context and a project by project incremental view won't give you the security you need. I mentioned this as a core issue in a chat with Marcus Ranum:

Marcus Ranum: Gunnar, your blog (1 Raindrop) is one of my favorite security forums, since you seem to be as comfortable with “the big picture” strategic problems as well as the practicalities, and you do it fluently and coherently -- do you realize how rare that is?

Gunnar Peterson: Thanks for the kind words on the blog. In terms of bouncing between big picture and practical issues, I think this is a must in security. We’re vulnerable to poor design and implementation. Getting the level of abstraction calibrated correctly is one of the enduring challenges in infosec. How many times have we seen a big picture policy or architecture document essentially filled with low-level configuration settings that offer no strategic guidance? Conversely, we often see low-level implementations where the assumptions inherent in the implementation cascade back up through the big picture and ripple through the whole security architecture: “Well of course for this little widget to run you have to open XYZ firewall ports, disable the sandbox, and send everything in the clear.”

So the challenge in security architecture is clear separation between the views and artifacts for big picture versus practical views, and then working toward making the decisions implicit then consistent across both.

The person who knows fuzzing is likely not the same one who knows Splunk and NAT and not the same person who knows SAML and Kerberos, and making sure they all work cohesively is fundamental to success. A working model is needed. Like the architecture example, the Security DevOps role exists informally in many organizations - its the people who cut across the siloes to get the hard stuff done, but it would not hurt to see a formal Security DevOps role emerge.

Dreaming a Little Appsec Dream

As much as developer education in secure coding needs to improve, we cannot leave the invoice for all the issues on the developer's doorstep. For one thing, there is basically no support for security in programming languages. 

Allow me to dream a little dream, most people involved in appsec have a set of pet ideas they would like to see implemented to make major improvements. Being in the appsec world for a long time, I am not sure there will be sweeping changes that will create big benefits any time soon, but I do have my own pet list of smaller things that could make a big difference.

Remember prorgamming in the 90s? Specifically do you remember network programming in the 90s? Buffer overflows, memory leaks, and connection leaks were a leading cause for outages and bugs in general. For all its faults, Java swept a lot of this away.

Anybody programming in that time remembers seeing systems getting hung up on connection leaks, and this problem has been reduced by a simple language construct - Java's Try...Catch..Finally. Its possible to still write egregious code inside Try..Catch, do a poor job configuring, and still shoot yourself in the foot many ways with Java, but this simple construct enforced by the compiler has seen near elimination of a once prevalent and powerfully destructive bug. The compiler ensures the check is made and the connection will be cleaned up when necessary.

I am afraid we won't see big sweeping changes in appsec this year, but at the top of my list is a Try..Catch..Finally "you can't compile without it" declaration in programming languages. This would do more to change developer's output in the short run than anything else I can think of.

The next question is - what check(s) should be enforced at that level? Three things stand out (but if any language developers are reading this I would be happy to get even one):

1. Access Control container: the first two lines in your program should define "who are you" and "what can you do"? This mostly not the case today. But a language could enforce that every app border (remote i/o) must define the authN/authZ providers.

2. I/O Filtration: A good sanity check for languages is - how easy is it to intertwine the control plane and the data plane? It should be hard -  Languages should requre Input validators, output encoding, data escaping. Again similar to JDBC connections it would still be possible for a developer to make a hash of this but at the same time it would remove one part of the sin ommssion problem.

3. App Logging: the app must be configured to report out all access events to a log for monitoring purposes - define log provider, audit record formats and event types

Number 1 & 3 would allow the language to facilitate a reference monitor that is always invoked, that an auditor or admin can clearly identify and tune policy for, and report out on its health. #2 is effectively the biggest lesson learned in web app building for the last decade and the source of many of the worst issues.

I don't think languages are in a position to fully "solve" the core issues here, just like Java didn't solve performance issues, but at the same time these three mandatory compiler checks would nudge developers in the direction of secure coding. The biggest influencer on your security is the person coding your app.

Bottom line - there are developers writing code right now with none of these checks right now and adding them would be a big win.

Update: Jeff Williams adds his vote for what would be valuable - "how about an environment that automatically generates evidence that controls are used where they should be" What would yo ulike to see in a Security Try..Catch..Finally... can't compile without it block? Its ok dream (a little)

Making a *Gasp* Tech Dividend Growth Fund

Well it took til 2012, but Apple is paying a dividend (not quite actually they had paid one previously all the way back in 1995). But in honor of this latest entrant in the dividend paying companies lets look at the possibility of build a dividend index out of canyoubelieveit a set of tech companies. Ten years ago tech companies were mocked as bogus and lumped under the heading of pets,com style flameouts and the *real* companies, Financial services, were all the rage as "adult" investments for the sober investor. What a difference a decade makes.

Who was really the risky bet in 2002? The likes of Apple and Microsoft or the likes of Lehman and company? As we sit here the large tech companies are sitting on pristine balance sheets with tens of billions in cash, and the financials still standing are happy to be, well, still standing, and high fiving that they passed the stress tests after some governmental CPR.

What do companies do when they have a warchest of cash? Well they can reinvest it in the business, but many tech companies operate on a pretty asset light model. The only really large expenses come when they buy other companies and this often works out poorly. The second thing they can do is buy back shares, which may provide a boost to share price. And the third thing they can do is to pay a dividend to shareholders. 

It used to be the case that the chances of a tech company paying a dividend were slim and none. Dividend paying companies are a pretty dowdy lot, and so dividend investors needed to look at Utilities and the like as options for income. But as tech companies grew, and the options for how to deploy cash were exhausted paying dividends is now trendy.

The S&P Index yields around 2% and its possible now to pick a basket of tech stocks that pay a better current dividend yield with arguably more attractive growth possibilities. 

The yield is in part a function of price. IBM shares have been on a tear and even though the company has raised its dividend from $0.59/share in 2002 to $2.90/share (almost 5x growth over the decade) the yield works out to only 1.5%. Other companies like SAP (0.9%) and Oracle (0.8%) pay only a token yield. Cisco recently joined other big techs and now pays a 1.6% dividend.

So with a goal of beating the S&P's ~2% yield with tech companies here is a basket of four companies to do just that

• Apple                      1.8% dividend yield
• Applied Materials,    2.9%
• Intel                       3.0%
• Microsoft                 2.5%

There are three questions to ask when examining dividends. The first and easiest to answer is current yield. An equally weighted basket in the above companies equates to current yield over 2.5% which beats the S&P by around 25%.

The next question is how safe is the dividend? Each of these four companies has billions of cash on its Balance sheet, AMAT has the least at $6 Billion. Probably enough to weather a storm or two.

At this point you might ask, besides the fact that its a hot topic this week - why include Apple on the list? After all, they pay less than the market average dividend yield. This brings us to the third topic in dividend investing - dividend growth. Remember the IBM example of 5x dividend growth over the past ten years? Who know if Apple can pull this off or not, but if so this would mean a vurry, vurry tasty 9% dividend yield in 10 years. Microsoft began paying a dividend in 2003 when it paid $0.08/share, today its dividend has grown 8x to $0.68/share. Intel paid $0.08/share dividend in 2002, and today doles out $0.78/share, AMAT initiated a $0.06/share dividend in 2005 which its grown to $0.31/share today.

	AMAT	INTC	MSFT
2002		0.08
2003		0.08	0.08
2004		0.16	0.16
2005	0.06	0.32	0.32
2006	0.16	0.40	0.34
2007	0.22	0.45	0.39
2008	0.24	0.55	0.43
2009	0.24	0.56	0.50
2010	0.26	0.63	0.52
2011	0.30	0.78	0.61

 

2.5% by itself isn't that much to get excited about, but at the same time its way better than what your cash pays you (nothing). Bonds may have a little better current yield but as Buffett says today "bonds should come with a warning label" meaning they will not fare well in an inflationary environment. Plus bonds can't grow their yield, and certainly not like the level of growth Intel, Microsoft and Applied Material managed. 

Without getting into the specific business dynamics, the four companies mentioned all exhibit desirable characteristics for a dividend investor - above average current yield, safety ensured by robust cash flows and strong balance sheets, and with a decent chance to grow their yield over time.

As an historical note, one of the things that has worked best in investing is to find firms with higher than average yields with low payout ratios. Google is the last behemoth tech to not participate in the tech dividend revolution, let's hope they get on board soon.

As always this is not investment advice, just ideas, you need to do your own research. What I find interesting its that its possible to find companies, tech companies, where the stocks offer both higher returns and a safer return than cash and bonds.

Infosec Mindset

From Stephen Northcutt's remembrance of Hal Tipton:

I was asked to work with NASA as part of the getback into space after the Challenger disaster. The project culminated with a series of briefings to senior management and I did one onsecurity. In the evening there was a mixer. Hal came up to me, pushed his finger into my chest and said, "You have no idea what you are talking about!" OK, I thought to myself and waited. Hal continued, "Your job is just like the loss prevention officer at Kmart. You can't protect your organization from attack, the best you can hope for is to keep shoplifting to a low enough level that they do not close the Kmart."  At the time I was a bit offended, but as the years have gone by, I havecome to see the wisdom of his point of view.

This mindset gives the right business context on how to look at security issues, especially in terms of survivability. I would further add one clarifying quote from Roger Needham (talking about systems back in the 1970s):

Going all the way back to early time-sharing systems, we systems people regarded the users, and any code they wrote, as the mortal enemies of us and each other. We were like the police force in a violent slum.

The world changes, but in Infosec not so much, eh? So combining these two thoughts the right mindset combination is a Loss Prevention Officer at a Kmart located in a violent slum.

Security Failure Scenarios

Richard Bejtlich can pack more depth into a tweet than most people can in a 2,000 word article:

Q: What happens when you try to prevent an attack by professionals?

A: You lose. So, fast detection & response is best refuge.

Security is a loser's game where you aren't trying to win per se but rather trying to avoid losses; however losses still happen. Good engineering practice means designing for failure and security mechanisms are no different.

In any SDLC security will at some point review the design and implementation against some target state, threat model and/or risk assessment. The output of this is a set of security countermeasures that the security team recommends.

But there is a critical step to iterate on here - the security mechanisms did not come down on stone tablets from Moses or something,  they'e just code and in many cases built by the exact same team, and they are just as susceptible to failure as anything else in the system. So once the design and implementation is updated with the countermeasures, the target state review, threat model and risks assessment must loop through another iteration to determine any new issues.

Did I say the security countermeasures are just as susceptible to failure as anything else in the system? Actually, not quite so, I would say that security countermeasures are more susceptible to failure than other parts of the system. The reason is simple: complexity - security people rail against complexity from dawn to dusk, but in any system you will have to look long and hard to find anything more complex than Kerberos, X.509, crypto implementation, Key management, and other security countermeasures. 

Security countermeasures are complex in design, implementation and testing. So the design must factor in the failures of the secuirty countermeasures themselves. This brings us back to Richard Bejtlich's detection and response guidance. Let's take an example of an access control system, much time will be spent on architecture and design of the authentication protocol, whether SSO is offered, what identity token(s) in use, how does it map to authorization, what level of granularity is authorization granted on, where is the session managed, and a myriad of other issues. However, a fundamental underlying assumption in these discussions is that the access control system can not be bypassed, but designing for failure means assuming that it can be.

Access control design tries to answer the question - who has access to what? Implicit in any access control design should also be attempting to answer the question - who accessed what? The different assumptions in these two questions is the difference between night and day. Answering who has access to what presumes the access controller can manage this relationship. Answering who accessed what is about recording events in context. And this is where detection services come in - its not solely an identity or appsec question to answer, but identity and appsec have roles to play here. There are ways to accomplish this either within and/or outside the access control system that give improved visibility into the system's use. The app, identity, and access control layers all contain context that is not possible to get solely from network monitoring (which is state of the art for most companies today). 

Its surprisingly often that people building access control systems either

a) focus on the runtime but do not factor in the back end monitoring services

b) assume that it won't be bypassed

Neither of these will meet the determined opponent test. Locating log event sensors App and access control layers (and/or related subjects/objects) offers concrete improvements for what gets reporting to the log, some initial effort to design the log events messages can simplify the approach of for example rehydrating the transaction history to see the chain of events that preceded a failure of some sort.

About 400 years back, Yamamoto Tsunetomo had a great post on this:

“There is something to be learned from a rainstorm. When meeting with a sudden shower, you try not to get wet and run quickly along the road. But doing such things as passing under the eaves of houses, you still get wet. When you are resolved from the beginning, you will not be perplexed, though you will still get the same soaking. This understanding extends to everything.” 

So Protection and Detection are not necessarily separate things.  When designing a countermeasure such as access control the mindset should be that this protective countermeasure (and/or the subjects/objects it protects) must itself be backstopped with a margin of safety to report out any failures through logging events, and any protection mechanism should be armed with the same sort of detection services.

Trust

In God we trust, all others go through the access control matrix.

Market Low Anniversary

Its pretty clear that we humans have a tenuous at best grasp on risk. One reason I enjoy looking at market examples for risk-related behavior is that stock market provides a transparency that's lacking in infosec but the decisions and behavior are dervied from the same biases.

Today we are three years removed from generational stock market downturn where in March 2009, the S&P Index touched 666. Take yoruself back to that time, collapseniks were the belles of the ball. At the head of the list was Nouriel Roubini who when asked how the market would perform  "Lower ... much lower," was his answer (remember he is saying this as the S&P is at 666). "Expect ... new lows reached in the next months and the year ahead." What we got instead was perhaps the best Bull Market this generation will ever see. The market is up 123% since.

Unfortunately, most people missed it. Instead, driven by fear, people were pulling money out of the market at the exact time it should have been moving in. And of course now people are more interested in investing with the market recovering. As Buffett says - you pay a high price for a cheery consensus.

Risk analysis needs to be back not by "wisdom" of the crowds or hysterical doom and gloomers but by data. When the S&P was hitting its 2009 lows its was back to the same price levels of 1996, for your reference in 1966 a house cost $14,200 and gasoline was 32 cents a gallon. Were you taking crazy risk to buy stocks at that level, as Roubini asserted, or were you getting the bargain of a generation?

Doing your own analysis based on data pays. In the same way, we should not judge the risk of technologies or efficacy of controls because convetional wisdom obliges us to look at one deployment scenario "safe" because its "inside the DMZ" and another control effective at stopping range of attacks simply because it translates network addresses. 

Being a perma-Bull or a perma-Bear is plain silly. In 2009 everyone was celebrating the permaBears but in 2007 it was the opposite. Instead understand the scenarios, get the data and be objective. The crowd agreeing with you - whether its stock picks or making network firewalls your #1 priority - is irrelevant.

Federal Cloud Security Challenges and Solutions

There is a lot happening in the Federal space related to security standards. I wrote a paper which you can get on Intel's blog on some of the more interesting standards - Federal ICAM Roadmap, NSTIC, FedRAMP, and HSPD-12. 

• Federal ICAM Roadmap goal: “Increased security, decreased identity theft, data breaches, and trust violations”
• NSTIC ‘Why We Need It:’
  • 1. Passwords are inconvenient and insecure
  • 2. Individuals are unable to prove their true identity online for significant transactions
• FedRAMP: “The decision to embrace cloud computing technology is a risk-based decision, not a technology-based decision.”
• HSPD-12 Requirement: “Secure and reliable forms of identification”. NIST Guidelines on Security in Cloud Computing: Critical pieces of technology, such as a solution for federated trust, are not yet fully realized, impinging on successful cloud computing deployments.

"The themes that each of these have in common is a recognition of rising cybersecurity threats and acceptance that the security baseline must be raised to counter these threats.

The Federal government has produced a number of timely publications and standards that offer guidance for building a strong cybersecurity posture to handle the changes the Cloud brings. Taken together, this work represents a shift in how the Federal government engages with technology and it’s consumers. What they all have in common is a recognition of the current trends, and the types of security technology required – stronger identity, identity federation, use case centric architecture, continuous monitoring, and the importance of information security."

The full paper is available here.

Gillian Tett on Risk

 

Gillian Tett: "So we Super Seniored the entire financial system and just ignored it."

This is still going on by the way, but what I think it shows and what I think is instructive to technical risk such as infosec is the role of assurance and challenging assumptions.

There are huge pools of risk that get missed or ignored due to "inside the firewall/outside the firewall" type grouping. The interworking is poorly understood and layering on assumptions what controls can realistically achieve is no different than Super Senioring your enterprise "inside the firewall."

Richard Bejtlich tweeted yesterday

Q: What happens when you try to prevent an attack by professionals?

A: You lose. So, fast detection & response is best refuge.

We need controls and protection services, but without Monitoring and Assurance, the Super Senioring the whole system mindset only amount to piling up more risk. 

Pragmatically, what I recommend is that for each control and for each asset, the architecture, development and oeprational model must include not just Controls but also Monitoring and Assurance. And to avoid the Super Senior mindset these should be done with he mindset that the controls don't work.

I like to pick on firewalls because they are a chief progenitor of the Super Senior/inside the firewall way of thinking, but the same exercise works for any control. In a design review, I ask the team - how would your system security work without a firewall? At this point there is generally a long pause. Because really from an attacker point of view the firewall is only stopping a small percentage of attacks.

The mindset that assumes controls fail leads the design in a direction of avoiding those failures to the extent possible but also in the direction of assurance and monitoring.

Heartland 2011

Heartland's stock fell precipitously when the breach was announced from around $17/share to under $5. But anyone who bought the stock then has done quite well. A little more than three years on, the shares closed today at $28.43 good for a 568% improvement (who cares about Facebook IPO? Too bad you didn't buy Heartland the day of the breach!).

Of course, its easy to find winners after the fact - just like I can give you a great weather report for yesterday -  and as evidenced by HPY's price lots of people did not have confidence they would make it. What is quite interesting about Heartland through the breach is that their shares are not just surviving, they are thriving compared the S&P 500 index. The chart below shows Heartland outperforming the index by almost 20%.

Stock price is one thing - business metrics are another. In January 2011, I asked - has the bleeding stopped at Heartland? And using a very conservative set of value metrics on the data available through that time it appeared they had. Now that we have full 2011 data, that thesis has been confirmed. Let's dig into the numbers

This is the fifth in a series of posts looking at Heartland's stock and business performance. As Ben Graham said in the short run the market is a voting machine (what people intuit will happen), but in the long run its a weighing machine (the returns).

Just to review our valuation metrics, we are going to take Heartland to the value investor's Mayo Clinic and examine the following data:

• Operating & Net Profit Margins, Free Cash Fow: how much the company earned after taxes
• Balance Sheet: Current Ratio & Long term debt/equity
• Return on Equity: as a proxy for how well the company is run - are they good stewards of capital?

Before we dive into that though, let's take a look at top line Revenue (figures in millions). 

	2011	2010	2009	2008	2007
Revenue	$1,997	1,864	1,652	1,545	1,314

 

So in terms of weighting top line Revenue returns, we can see from 2008-2011 a 29% increase in Revenue. There are many, many businesses that would be happy with that performance over the financial crisis time period.

Of course, businesses need profit, top line Revenue is of little importance if the profits don't flow. So let's look at this area and Net Profit Margins.

	2011	2010	2009	2008	2007
Net profit	2.2%	1.9	-3.1	2.7	2.7

Net profit margins haven't completely recovered from the breach, but they are still steadily improving. 2009 margins showed the negative effect (and why the market "votes" were short term correct), but 2010 showed a return most of the way back to pre-breach profit margins that improved in 2011. 

If you were running a candy store Free Cash Flow is the metric you would use to show much money came into your cash register and how much was left.

	2011	2010	2009	2008
Free Cash Flow	$71M	-46	20	52

And here we see in Free Cash Flow the distinguising characteristic of a business returning to health and probably the main reason why the stock price has moved up sharply. As I wrote last year of the negative $46M FCF - "This is one area where see the breach effect lingering into 2010, likely the result of paying the fines out of cash flow, and its one to watch to see if the recovery is complete." The 2011 FCF shows the effects in the rear view mirror.

The Current Ratio shows the liquidity in total current assets divided by the total current liabilities. The higher the number the better because this calculation divides Total Current Assets by Total Current Liabilities. As such its an important defensive calculation to check if the business is playing Balance Sheet games to achieve revenues, margins and so on. 2011 shows a slightly smaller but still healthy ratio.

	2011	2010	2009	2008
Current Ratio	1.12	1.19	0.80	0.94

Long term Debt to equity ratio looks at the company's solvency, here is another area where we see an effect. The Long term debt jumped from $8M in 2009 to $85M in 2010. The effect is moderating in 2011, where Heartland paid down $15M debt to $70M

	2011	2010	2009	2008
Long term debt/equity	32%	48	6.5	9.5

Many investors consider Return on equity as the leading indicator of a well run business, it shows how well the company does at deploying capital.

	2011	2010	2009	2008
Return on Equity	22%	22	-33	24

This metric shows a return to pre Breach levels of ROE. Note that 22% Return on Equity places Heartland in the company of elite companies like Coca Cola which had a 27% ROE in 2011. Companies that deliver ROE around 20% generally do quite well over the long haul.

In sum, we can see with three full years of data since the breach was reported, there is a hole in the balance sheet courtesy of fines, but those are in the rear view mirror. The overall stock and business metrics show a thriving business.