Metrics and Reporting

This page is devoted to our Metrics and Reporting Survey and associated member responses and content. 

Our Metrics and Reporting Survey continues to attract respondents, and has spurring some members to share additional information.  In this, the Year of the Member, we are happy to feature the comments of CISO Executive Network members.  Kevin Hamel of COCC Inc., has been kind enough to share his experiences with metrics and reporting.  Here’s what Kevin has to say:

“Your survey about metrics struck a “passion point” for me.  It’s something that I’ve been working on here for a while with some success, but still with plenty of room for growth.  Since this is something that is near and dear to my heart, I thought I’d share some of my thoughts and experiences.  

I came across an article that I really like in one of the ISSA magazines (it can be found below). I took some of these suggestions and created metrics for our CTO for each infrastructure piece (each major OS and routers – I plan to add DB’s)

·           Configuration Compliance

·      % overall compliance (per machine, per setting)

·      % of machines that are 100% compliant with standard config

·           % of machines with AV installed and running

·           Patch status

·      % of machines fully patched or within patch deployment time frames

·      % not patched

·           Vulnerability status - % of machines that have zero vulnerabilities

·           # of open security issues per department (a security issue is security “hole” that needs to be resolved)

When we first started reporting these, the numbers themselves were, of course, valuable.  Now that we’ve had them in place for a while, we can trend them over time.  So now we can also see whether we are getting better or worse in any of these areas.  And, naturally, our technical teams can drill into these numbers to see exactly where things are out of compliance. 

But where I’ve had the most challenge is in coming up with metrics or measures for Board reporting.  I present semi-annually to our Risk Committee and over the years they have become less “security” focused and more “risk” focused.  So, my current challenge is how do I take those stats mentioned above and translate that into “risk” for our Board?  They need (and want) to understand, what does this data mean to us as Board members?  Should we be concerned about the fact that x% of our devices are not patched?  And how concerned should we be? 

So, what I am trying now (which seems to be working pretty well), is a quadrant chart.  I try to convey where we are with certain aspects of our security program by plotting them on a quadrant chart.  It’s sort of like a Gartner Magic Quadrant, only reversed.  We DON’T want things in the upper right hand quadrant.  Our y axis is Exposure and I have defined criteria to assess overall exposure to the company.  Our x axis is a Security Rating.  That, too, uses a defined rating scale that encompasses the weaknesses found, balanced by any mitigating factors.  From that we communicate our risk mitigation strategy for any items that are beyond our risk appetite. 

I still include general metrics like packet dropped at the firewall, etc., in the Board report.  But they are included solely as supplemental material and are no longer covered in discussion. 

It’s working pretty well so far but it’s still pretty new, so I suspect we’ll continue to tweak it.”

Thanks, Kevin, for the insight into your techniques.

If you have similar information you’d like to share, please let us know.  We are committed to sharing member insights.