Detecting anomalies: The what, the why and the how?

Anomaly is what escapes not only the law but the rule. What is outside the game, "offside," no longer in a position to play.
- Jean Baudrillard in the book The Perfect Crime

This book was published in 1994 and has nothing to do with cybersecurity, but it does justice to the definition of the term anomaly. An anomaly can be identified only by the players in the game, as only the players know the game's rules and can identify who or what is not playing by the rules. To identify anomalies in cybersecurity, we need both data scientists, who check for deviations from the norm, and security experts, who know the rules of the game.

In this three-part series on anomalies, we will learn what anomalies are, why we need anomaly detection, and how to use the available techniques of anomaly detection in our organizations. Here's the first part which will take you through the different types of anomalies.

What is an anomaly?

Let's play a game to understand what anomalies are.

Look at the three images below and see if you can pick the odd one out in each. Obviously one at a time, silly. Do pat yourself on the back if you find the odd one in every image.

Great, if you were able to identify the exceptions in every image!

How did your brain identify the odd ones? Your brain analyzed the patterns in each image and first registered the similarities. For example, it made a "silent note" that a "normal cat" has both eyes looking straight ahead. When one of the cats exhibited a behavior that deviated from what is considered normal, it was registered as an anomaly in your brain.

The same procedure was followed by your brain to identify the anomalies in the other two images.

An anomaly is a deviation from an established normal pattern

What is normal needs to be established to identify the abnormal. But what is normal? Normal is a subjective perceptive and differs from person to person, department to department, and organization to organization.

What could be the anomaly in the image above to a farmer in Australia? Most of us would think that the correct answer is the white rabbit. But the correct answer according to the farmer would be the brown squirrel, because there are no native squirrels in Australia.

Therefore, contextual information is also very important in identifying anomalies.

Anomaly detection in organizations

Whenever we notice abnormalities in our day to day life, we may try to find solutions to bring the situation back to normal. Detecting anomalies early and rectifying them helps you believe you've returned the situation to a more comfortable, normal state.

Where do we see anomaly detection in real life?

Anomaly detection in images: Hospitals, laboratories, manufacturing companies, etc. could use anomaly detection in images to serve their customers more effectively. For example, this could include detecting cancer in an MRI image, sorting produce in agriculture, or removing defective parts in an assembly line.

Anomaly detection in time series data: A series of data collected and collated over a particular period of time could indicate normal and abnormal behavior. For example, the check-in data of employees collected during regular intervals of time can be considered time series data, and it can be used to detect abnormal check-in times.

Now that we have an idea about where we could use anomaly detection, we can now discuss the different methods of anomaly detection. Before that, let's understand the different types of anomalies.

Different types of Anomalies

Point anomalies (global outliers)

This is the simplest and most widely observed anomaly category. If a single object stands out completely from the rest of the observed objects, then it is a point anomaly.

In the graph above, Point 11 is an anomaly. It can be observed that there is a gradual but steady increase in the data set, but at point 11 the increase is abnormal. The data dips down to the regular range from point 12.

Example of a point anomaly in cybersecurity: A single server crashing at a particular time could be due to multitude of reasons. But if numerous servers in an organization crash within a short span of time, that is definitely an anomaly and could be a security threat. In this case, the IT security team should definitely be cautious and be on red alert.

Contextual Anomalies (conditional outliers)

These kinds of anomalies occur at a particular time or during specific circumstances.

In the graph above, a dip normally happens at an interval of 5. Let us assume there is a legitimate cause or reason behind this dip. But if we take a closer look at the graph, there is no dip at Point 15 as it must be. This is a contextual anomaly. You can identify an anomaly of this kind only if you know the context that the graph is set upon.

Example of a contextual anomaly in cybersecurity: Your human resources manager logs in every day around 10am, and logs out before 7pm. On a random day, you notice her account has been logged in at 3am. This could be the result of a mistake or an emergency. But a security analyst needs to investigate this anomaly, particularly if the user is already on a watchlist. If not for her previous log in and log out information, this particular incident will not sound an alarm. A contextual anomaly is anomalous in a specific context but not outside of this context.

Collective Anomalies:

A collective anomaly occurs when a subset of data points significantly deviates from the rest of the data, but the individual data points on their own are not anomalous.

In the graph above, we see the number of active privileged users on each server during a typical working day. Server 4 seems to have an abnormal number of privileged users. This would not have been considered abnormal if we did not have the data from the other three servers. This is an anomalous activity that falls under collective anomaly.

Example of a collective anomaly in cybersecurity: Let us assume that there is an increase in the number of privileged users for a particular server. An IT security analyst might not have noticed the unusual activity if not for the entirety of the data from other three servers. Collecting and comparing data is a major part of an analyst's role in keeping the network secure as it helps detect anomalies at the right time.

We now know what an anomaly is, and the types of anomalies in general and in cybersecurity. In the next blog in this series, we will learn why organizations need anomaly detection and how to detect anomalies in the network.