Web services security metrics

Here is a list of measurements we built today in my SOA, Web Services Security class

Metric: Number of Web services vulns
Source: scanner
Expression: H/M/L

Metric: Application issues - exposure issues
Source: *.WAF logs
Expression: Security event vs number of request

Metric: Anomalous behavior
Source: application logs
Expression: known good vs anomalies

Metric: Policy Compliance
Source: SOAP Sniffer, Encryption Scheme, Keys
Expression: number of policy events - success/fail

Metric: AuthN strength
Source: SOAP Analyzer
Expression: Policy compliance for service request authN

Metric: AuthZ
Source: Design or logs or access Policy Enforcement Point
Expression: Success/fail based on policy

Metric: Unsuccessful authN
Source: Log files, *.AccessMgmt
Expression: Pct of failed requests

Metric: Input validation errors - XSD validation
Source: JAXB schema validation, *.XSG
Expression: Number of fields vs. validation failures

Metric: XDoS
Source: Synthetic transaction monitor
Expression: Availability, and uptime

Metric: R/R
Source: App logs or gateway
Expression:Number of inbound requests vs number of responses

Metric: Usage patterns
Source: App logs or gateway
Expression: Value metric based on usage

Metric: XSS, SQL Injection, XML Injection
Source: IDS, IPS
Expression: Time of attacks, before or after business hours

Metric: authN vs. un-authN attacks
Source: IDS, IPS
Expression: reverse engineer based on resource - success/fail

Metrics: access logs, servlets - learn about app
Source: proxy, app logs
Expression: known and unknown resource deltas

Metric: success/unsuccessful cross domain authN
Source: access control container
Expression: measure by services

Comments

[Alex]

Hi Gunnar,

I'd be interested to find out how you differentiate between "metric" and "statistic".

[Alex]

That didn't come out right. Not enough coffee and too much jetlag

- that should read:

"I'd be interested to find out how you differentiate between -the value of the different- "metrics" and -their- "statistics" to an organization."

That is to say, clearly some metrics are worth more than others. How would you arrive at how much more valuable, say, your AuthZ metric is vs. your Policy Compliance metrics?

[Gunnar]

Hi Alex,

I think it is a fair point that many of these are stats not metrics. This list came out of a rapid fire exercise in the middle of a web services security training I did last week. The point of the exercise (beyond me banging the drum for metrics) is to try to get the people in the class to reason about the security problems in a new way - instead of relying on axioms like firewalls, let's try to come up with objective measures. So stats may be used to build up metrics, where I would define a metric as stat(s) plus context. This list was compiled from me just typing in what the class came up with on the fly, so context needs to be built up.

I like to publish these lists, because there are frequently senior practitioners in the class that come up with some creative ways at looking at these problems