Rating Cyber Security Practitioners

1330 words, 5 minutes.

Jean-Léon Gérôme, The Gladiators (Bronze, c.1878).

Do you like hyper-masculine, gladiatorial confrontations where losers suffer not just the ignominy of defeat but significant even permanent loss of status in a rigid hierarchical system? You do, do you? Well then you’ll like this.

Or do you think that sounds detestable? If so you’ll want to defeat this barbarous idea before it takes hold. Keep reading.

The World As It Is

The number of security qualifications has exploded. There are over 40 certifying bodies alone¹. Hundreds of University courses. Dozens of community-based competitions and tournaments. It’s no wonder students can’t figure out the relative value of each of them.

Employers are lost in a sea of acronyms too. How does one compare CREST to SANS to CISSP? It’s a simple answer. You don’t. Some of these courses are excellent and the role of the teacher is a noble one, but the qualifications don’t tell you much about an individual’s relative ability. There are too many of them and they are too widely held to be much of a discriminator. Even for candidates, there’s little reason to be selective. For commercial certificates, if you don’t pass you’ve got nothing to lose but the retest fee.

There’s something about all this that’s disconnected from reality. There’s a degree to which even Capture The Flag falls short. It’s not the material. It’s the lack of an individual opponent with real skin in the game. It’s the consequence-free outcome. It’s the artificial insertion of a certification body between the challenge and the challenged.

For Cyber Security professionals, reality is adversarial. It has a binary outcome. It’s often solitary. It’s organic. By which I mean developing naturally over time from that which is prior. The prior bit being your previously unsecured systems, the stuff you inherited from the last guy.

Not infosec as you might wish it to be, but as it is.

Adversarial

As a scientist, I understood the importance of remaining detached from an experiment. But I’d never solve the problem until I got involved; until I worried about the cancer patients who might be injured by this guy; until I became angry that this hacker was directly threatening all of us. - Cliff Stoll, Cuckoo’s Egg (US: Doubleday, 1989), p. 201.

Outside serving in the military or the police, Cyber Security may be the one area of technology that’s truly adversarial. Even those who “fight” diseases do so only metaphorically. Those of us building software are “fighting” complexity to a degree, but in infosec the struggle is direct. Even personal. Most of the time you won’t get to find out the name of the human on the other end of the connection, writing the malware, demanding the ransom, or controlling the botnet, but adversaries do have names, faces, and personalities, even if you never get to read the indictments.

Binary

At the micro-level, a breach is the result of a whole series of binary outcomes.

• A vulnerability is present or not.
• The attempted exploitation of that vulnerability is successful or not.
• That attempt is detected or not.
• Analysts make the correct determination and course of action, or not.
• The adversary gets to deploy the next exploit in the chain or not.
• We go around this cycle again until the objective is achieved.

At the macro-level, the effort cost for the adversary is either raised sufficiently to deter or repel them until next time, or it’s not. In the case of Advanced Persistent Threats, which most companies will thankfully never encounter, the adversary is forced to consider whether their gain is sufficiently great to expend treasure or not. Arsenals are not infinitely broad and deep. Most ordinary adversaries simply use a mix of open-source tools. In any case, there’s a winner and a loser for every engagement and the outcome is clear.

Solitary

If the first truth is that most will never encounter an APT, then the second is that most companies are well below the infosec poverty line². Many are lucky to have even 1 dedicated staff member let alone a department for IT security. Outside of large corporates and tech companies, it’s often a solitary post. A person with responsibility but little power, seeking to influence and improve the behaviour of a mass of developers, engineers, architects, and operations staff, and of course the business people. We’ve all seen the job ads for “CISO” (the ones listing 1st line SOC engineer skills and an on-call schedule). Rarely is there mention of a need to manage a well-staffed and established team, or a full-time deputy. Malware authors too are often solitary. There are gangs, but some authors market their wares to them. Look at the prosecutions.

Organic

In short, the map is not the territory. While the best exams and labs are comprehensive, they are by nature static. CTFs do a great job of channelling the energy that comes from problem-solving against the clock, but ultimately they’re synthetic. There’s no real consequence for failure. There’s no real risk. It’s all constructed. Nobody loses anything.

Meanwhile, unfiltered Internet traffic has been a firehose of bad things since the late 90s. The average time to compromise for an unpatched Windows host is measured in minutes. CISOs are single-use items, like a distress flare. They’re let go after taking the fall for a breach or they bail voluntarily when the CEO makes statements they know to be unsupportable. That’s reality. That’s the world as it is.

Something Worth Having

What can we add to the sterile, contrived, and uninspiring catalogue of security certifications?

• Can we combine the energy of CTF with the spectacle of eSports³ and the reality of the work?
• Can we do this in a way that produces a clear outcome?
• Can we express that outcome in a way that layman can use to discriminate equitably between people?
• Can we impose a price for failure that doesn’t unduly dissuade new starters?

The Elo Rating System

Arpad Elo was a Hungarian physicist, chess master, and inventor of the Elo rating system⁴. His system is used in chess and mainstream sports like American Football, Basketball, and Baseball. It’s also used in a range of eSports.

• It’s designed specifically for two-sided, zero-sum competitions.
• It takes account of the predicted outcome between winner and loser.
• Differences between player ratings decide the points awarded from a game.
• The loser suffers more heavily if they were expected to win.
• This makes the system self-correcting.

In the Elo system, a new player is allocated an initial rating. Every time one player defeats another, the winner takes points from the loser. The quantity of points transferred (known as the K-value) depends upon whether or not the result was expected or was an upset. A very low-rated player beating a very high-rated one would take more of the high-rated players' points than they would have lost had the result gone the other way.

What if we used Elo to rank Cyber Security professionals? What if this ranking was decided by having professionals compete directly just as in chess? What if there was a red (attack) and blue (defence) rating? When hiring employees or consultants you’d immediately know their relative skill level.

We’d need a concept of an environment or “board”. Each player would have standard tools or “pieces”. Some of these might have defensive utility and some offensive. Finally, we’d need a clear victory condition. None of this is difficult and most of the groundwork has been done already by CTF organisers.

It would be a system for sorting and rating people that better-reflected reality. Adversarial. Binary. Solitary. Organic. The individual rated not as they might wish to be, but as they are.

Perhaps I’m unduly harsh on the certification bodies. There’s a part for them to play in this too. If the idea of Elo for Cyber Security professionals caught on, there would be a market for courses, exercises, mock tournaments, books, and coaching. Both the training providers and the students could point to a concrete outcome and justify their investment of time and money.

The crowd is on their feet. So what’s it to be, citizen?

👍 Using ratings to discriminate between Cyber Security pros is glorious.

👎 Using ratings to discriminate between Cyber Security pros is an outrage.