MSSP Blues and the Theory of Agency


I like the approach of listening to a good podcast and then using it to expand on a particular idea. This time, I listened to Brakeing Down Security’s fantastic episode where they discussed the fallout from a very rocky response to a security incident by an unnamed Managed Security Services Provider (MSSP). Bryan Brake talked to Nick Selby and Kevin Johnson, based on Nick’s original blog post. Please read the original post and listen to the podcast, but here is the summary:
  • Nick helped an unnamed customer respond to a security incident.
  • This customer had a long-standing contract with an MSSP for monitoring their network, which included having dedicated gear on-site.
  • When Nick & customer got the MSSP involved, they had a number of nasty surprises:
    • The monitoring gear on-site was not working as expected, and had actually not worked for a long time.
    • The customer-facing employees at the MSSP were not only not helpful but almost evasive. Bailing out on phone calls, not giving straight answers, …
    • The actual value the customer was getting from the MSSP was far less than what they imagined, and was not useful during the incident response.

In short, a series of horrible news and interactions. Bryan, Nick, and Kevin make a TON of excellent points on the podcast. Worth the listen.

This whole incident reminded me of a topic I’d been meaning to write about…


“Agents” have “Principals”, but do they have “Principles”?

How do you feel about hiring someone to do something for you? Maybe it’s an employee you bring in to your company, maybe it’s a mechanic you hire to look at your car, maybe it’s a lawyer you call on to help you with a contract negotiation.

This is a very common economic transaction. When looking at it, we often use specific terminology: those doing the hiring are ‘principals’ while those being hired are ‘agents’.

In an ideal scenario, the person/company you hire (the ‘agent’) is having their interests met with the compensation they’re receiving, and will perform their tasks in a way that meets your interests (you’re the ‘principal’). In all those cases – and pretty much any relationship like it – there’s always a potentially thorny issue: despite being compensated for their efforts, are those ‘agents’ acting on a way that is aligned with the ‘principal’s’ interests? What happens when interests don’t align? This happens all the time:
  • Is a mechanic over-estimating the effort to fix a car?
  • Is the lawyer extending the negotiation because they bill by the hour?

Say hello to the “Principal-Agent problem“, a well-known problem in economics (and political science). It is also known by other terms, such as “theory of agency” or the “agency dilemma”. Fundamentally, it is the study of the dynamics between principals and agents with distinct self-interests in a scenario where there is significant information asymmetry.

Information asymmetry, you may recall, is the situation when one of the parties in an economic transaction has much more material knowledge about it than the other.  There are further nuances on whether the information asymmetry develops before a contract is established – the agent has superior information to the principal from the get-go – or that asymmetry develops post-contract – as the agent begins to work, they realize the discrepancy. These lead to slightly different solutions.

Principal agent

 (source: wikipedia)

Another common example of Principal-Agent problems is the conflict between a company’s shareholders – who have limited information about how it is run – and the company management. Depending on how that management team is compensated, they may make decisions that are not in the shareholders interest: maybe boost stock price by playing accounting tricks, for example.

Both economics and politics have identified a series of mechanisms to help address Principal-Agent issues, but they fundamentally come down to a combination of:
  • Contract design – how compensation is dispensed (deferred), fixed versus variable, profit sharing, etc…
  • Performance evaluation – both objective and subjective
  • Reducing the information asymmetry – having more information to make informed decisions


Back to the MSSP debacle…
Now that we have this notion of Principal-Agent fresh in our minds, looking into the unfortunate MSSP incident we see the clear issues caused by the agency dilemma: there’s indication that the MSSP did not perform their tasks with the interests of the customer in mind. That is very unfortunate, and well deserving of the criticism they got …

Still, let’s look a bit deeper into the whole thing. As we do, we see there’s plenty of potential blame to go around (again, I suggest reading Nick’s blog for deeper background):
  • First of all, did the original security team at the customer that chose the MSSP do so with the organization’s best interest in mind? Were they trying to actually implement a proper monitoring solution or were they just trying to check off a ‘have you contracted with a managed security vendor for monitoring?’ item from some compliance checklist?
  • There was plenty of blame for the MSSP not following up a poorly deployed solution, but what about on the customer side? Why was there no oversight?
  • When the new security team started at the customer, what level of diligence was done on taking on a new infrastructure?
  • Did the management team at the MSSP care that a particular customer was not deployed properly? Did the team/individuals that created the on-boarding run-books for new customers care? Was the implementation team at the MSSP side properly measured on how to do on-boardings?
  • During the initial calls, were the employees of the MSSP acting on their own self-interest of “just get this customer off my back”? Were they empowered to do something but chose not to?
  • Back to MSSP management: did they structure internal operations to empower their employees to handle the exceptions and urgent requests?
One minor point I differ from Bryan, Nick, and Kevin on their well-deserving roasting of the MSSP is that they seem to assume that the individuals at the MSSP had lots of freedom to deviate from the established procedures. I’m not so sure: it’s one thing for senior, knowledgeable professionals to do so, but it may be radically different for others. Again, what did the MSSP empower their team to do?

I’m being overtly picky here to drive a point that there’s potential for agency issues at multiple levels of the event chain, both within each organization (customer and MSSP) and between them. There can be agency issues between employees and employers, as well as between separate commercial entities.


The broader impact

The point for this post is broader than the MSSP debacle. By the very nature of our industry, it is extremely easy for Principal-Agent issues to appear:
  • There is tremendous information asymmetry in InfoSec to begin with: There are too many details to go wrong, things change too fast, too many moving parts, etc… Those who hire us are often not aware of what we do.
  • We have tendencies to compartmentalize information about security itself (“sorry, we can’t talk about this”). This leads to further information asymmetry.
  • With “security” being a latent construct – it is difficult/expensive to observe/measure – our principals have a hard time measuring the effectiveness of security efforts.
  • With the difficulty & cost in hiring for security – be it employees, contractors, or businesses – there is less flexibility and interest in exploring details of contract design.
How do we – as an industry – get better? How do we deal with this? I think it comes down to:
  • First, we need to be aware of the issue and recognize it for what it is: a well-defined economic problem for which there are broad classes of solutions.
  • Then, we should recognize our roles within the transaction:
    • Sometimes as a buyer – hiring outsourcers, buying security solutions.
    • Sometimes as a seller – employee/contractor providing security services/expertise to someone, or selling a security solution/service.
  • Finally, within our roles, we should expand beyond the technical nuance – networks, encryption, appsec, etc… – and delve into:
    • clearly define and deliver reporting
    • pay more attention to contract design, service level definitions
    • perform periodic evaluation of the services
    • anticipate where principal-agent issues might arise and address early on. Maybe it is creating a better report, maybe it is having a lunch&learn on the solution, etc…
  • Lastly, we should continue to grow as community by sharing information – blogs, podcasts, conferences, … All that helps to reduce the underlying information asymmetry.
On that final point, I salute Bryan, Nick, and Kevin for their excellent podcast episode, and all the other community participants from whom we all learn so much…

If I had to summarize things:
  • Know what you’re buying. Educate yourself as needed.
  • Know what you’re selling and help your customer understand it as well.
As with so many other things, it’s not only an InfoSec issue , it’s an economic one…

On the “shortage” of InfoSec professionals…

It was interesting that two podcasts I listen to – PVCSec and Down the Security RabbitHole – both covered the ‘shortage in InfoSec’ topic. Both discussed the nuances and uncertainty around it. This is my contribution to that debate, just looking at things from a different perspective.

Getting things out of the way: yes, I think there is a shortage, and the sooner we acknowledge it, the sooner we can work on addressing it. I don’t have the hard data I wished to have to back up this claim, unfortunately, and will have to make semantic and anecdotal arguments. Yes, “plural of anectode is not data“.

Semantically, the very definition of shortage (from Oxford Dictionary) is:

{noun} A state or situation in which something needed cannot be obtained in sufficient amounts:

shortage of hard cash

food shortages


Yes, I admit there might be confirmation bias in my perspective.

I really enjoyed @catalyst‘s position here (and Rafal had a brilliant description of why we’re failing at training Tier1-type resources in the podcast), but I wanted to approach the problem from a different angle.

Let’s drill down into this a bit more and explore what this ‘shortage’ really means. To me, when people refer to ‘InfoSec professional shortage’, they really mean that:

“the current hiring process is not finding a large enough supply of professionals with a particular skillset and/or experience, for these roles within these teams at these companies, at these levels of compensation.”
So, as we analyze each of these sections, we can inquire/debate things such as:
  • current hiring process – is it broken? is the flow between HR and hiring managers appropriate? Is the screening process contributing to this? Are responsibilities and incentives for each party in this process properly allocated?
  • not finding – are they looking in the right place? Just waiting for resumes to arrive? Actively engaging with communities?
  • large enough supply – is the number of people being required a true necessity, or a reflection of inefficiencies somewhere else in the environment?
  • with these skills – are the skills being required actually relevant? How much of asking for a particular skill is not ‘playing it safe’, as opposed to understanding that some skills are easily transferable (especially products in same space: FWs, IDS, …)?
  • this type of experience – same as with skills, are the experience requirements really required, or are they a byproduct of inefficiency somewhere else?
  • for these roles – are we looking at the right roles for these people? Is it something that should be done within InfoSec or another team? Internal or outsourced? Human-driven or automated?
  • these teams at these companies -is it a matter of leadership? are the teams structured in a way that encourages professionals to apply? is the reputation of the team, manager, company such that would attract qualified candidates?
  • at these levels of compensation – finally, “rubber meets the road”. Is the overall compensation acceptable? Is the package of benefits attractive to the professionals looking into these roles?

I think @catalyst is right in that any one of these areas present an opportunity for enlightened leadership. All of them can be ‘fixed’:

  • adjust the hiring process to not “throw baby out with bathwater” – collaborate with HR to screen candidates adequately at all stages, look for candidates in different ways.
  • take a good look at the skills and experience being required.
  • reconsider the role and looks for alternatives, but considering the ‘whole’ picture and not just the immediate need to fill a seat.
  • take a good look in the mirror and check to see if there’s anything in the structure, culture, or leadership approach that might be driving candidates away.
  • finally, understand that there may be a case of ‘surge pricing’ and adjust expectations on compensation.

In economics, the labour market is a perfect example of information asymmetry at play. George Akerlof, Michael Spence and Joseph Stiglitz were awarded the Noble prize for their work on this. They have incorporated two mechanisms of reducing that asymmetry – signalling & screening. Learn about them and consider how your process currently implements them (even if unconsciously).

Finally, there may be perverse incentives at play: is the hiring manager (or HR) evaluated on the short term of ‘stopping the pain’ (just get someone!) or are there broader considerations about the health of the business (Raf’s point is poignant here).

So many opportunities here for leadership, like @catalyst said… but still, IMHO, there is a shortage of information security professionals.

As I look at the breakdown of what the shortage really means, I’m reminded of that adage: “Good, Fast, and Cheap. Pick Two.