Security has three areas that are very closely linked, that all bring value to any product, and are very rarely valued unless you are working in finance, government, or any industry that needs FDA or similar clearance. It’s not sexy, but understand if you don’t have it you’ll never realise how much you need it until its too late.
Who are these security heroes, that will save your bacon when you are thrown in the fire? They are Logging, Auditing, and the less appreciated but clearly critical sibling Monitoring.
What is the difference between Logging and Auditing?
You could be forgiven in assuming these terms were interchangeable.
The difference is more specifically around how the information is used, both terms can store the same base information (date/time, actor, action) but their purpose is different, and the extent of the data representation is different.
- An actor added a new record in system 1
- Addition triggers a request to system 2
- System 2 responded with success
- Success response delivered to the actor
This could be bundled as a transaction in an audit, but a log would have these recorded as and when they happened, multiple different transactions could be recorded together, especially in the case of asynchronous requests.
Logging is “What happened?”, typically this records implementation level events that happen as the application is running (methods calls, objects creation, modification, deletion, etc).
Auditing, however, is “Who did what?” and possibly why, it is about recording domain-level events: a transaction is created, a user is performing an action, etc. In certain application settings such as finance and medical, there are legal obligations to record the context of such events.
Monitoring, would check “Did what I expect to happen, happen?”, this interprets what is going on and can then allow the system to take action on events.
Such as raising a flag that a user has called, System 2, without a record being added in system 1, or if a high-risk action occurred what action should follow it (An admin account is created or modified, then the system administrator must be notified)
is user 1 logging into the system from NYC and Dubai within 5 minutes of each other (this could be acceptable if the user was located in Dubai and had been using the company VPN, or this could be an attack).
Monitoring gives the system administrators to detect abnormal behaviour as it is happening, this could be intentional or unintentional misuse, which would allow them to protect customers, by identifying defects in the field or locking down the system in the event of an attack.
Is this really a vulnerability of a system to not have an audit trail?
Or more specifically, a question, how do you know you are under attack if the attacker is not being noisy? Or what about if there is a small data manipulation error, such as a session synchronisation error.
Taking the previous example, it would only become apparent if you had multiple transactions occurring simultaneously, these errors can be quite difficult to detect during the verification and validation testing, and even in production may only occur very rarely, so detection could take a very long time before a customer reported it. Once reported replication in a testing environment may be difficult, this could even have begun so far in the past that standard backups (if it is not beyond the retention schedule for the backups entirely) might not be able to restore the object’s state to a stable state.
This was actually a feature of one of the first excel macro viruses, the Laroux virus back in 1995 which would randomly modify a cell, increasing or decreasing it by 0.01% once per day.
This is where audit trails can be invaluable, as it is possible to roll back time to the exact moment the first corruption occurred and see the event that caused it and allow a restoration of the system from that point.
The OWASP (Open Web Application Security Project) Top Ten, calls out Insufficient Logging & Monitoring as its’ 10th vulnerability, which must be accounted for when developing and testing a system.
OWASP state that Insufficient logging and monitoring
is the bedrock of nearly every major incident. Attackers rely on the lack of monitoring and timely response to achieve their goals without being detected.
The first stage of any security engagement is reconnaissance. When performing a forensic investigation following an attack this initial reconnaissance stage can be quite revealing. It only takes one lapse of operational security on the part of the attacker to get caught, that one time they visited the target without a VPN, proxy, or anonymity network could be during this phase, and the logs could be that critical piece of evidence for use against them.
But, do I need it?
It is not sexy, it doesn’t grab headlines and doesn’t sell the product to customers (until GDPR came into force in 2018) or upper management, it can be a difficult sell to get comprehensive logging and an audit trail implemented within an application. Unless, as previously mentioned, the business you are in, is Finance or Healthcare where auditing is common practice, and even then,
They will ask, “Do we really need it?” and you will answer
Of course, as a responsible team, you would be following the principles of security-by-design and have implemented this from the very birth of the application… but realising the reality of a world where security is often thought of late in the lifecycle rather than earlier, you will be able to inform those that question its’ importance that, Yes, it is necessary.
A defense-in-depth strategy typically involves a combination of preventive, detective, and corrective security measures.
Striving for the highest standards of quality and security for customers and any organisation is what we do. To support this, there are many information and technical standards that organisations can call upon, and if your organisation wants to be respected on an international stage or claim compliance or adherence to any of these, then audit trails, monitoring and logs need to be a critical technology in your organisation.
GOV.uk – Minimum Cyber Security Standard
In section 8 – Detection, while doesn’t specifically mention logs, it does call out that cybersecurity attacks should not be able to occur undetected. Also, post-incident recovery determining if personal data was accessed and what data was compromised, and how the attack occurred would be very difficult without these controls. Bear in mind these are minimum controls for any system, the regulations get far stricter if dealing with sensitive information.
ISO/ IEC 27001
The ISO/IEC 27001 standard is very useful measurable standard for organisational security, and it covers a great number of protective controls, some are relevant at organisational or process level, while others must be implemented at an application level, to record and respond to incidents, the following two sections which cover logging, auditing and monitoring
A.12 Operations security
A.13 Communications security
National Institute of Standards and Technology – NIST
NIST has several standards for protecting data and even specific guides to assist organizations in understanding the need for sound computer security log management. These guides are very useful in defining the need for, challenges of, and example configurations for log management solutions.
Payment Card Industry (PCI)
Having security and application, logs and audit trails are critical in the financial industry which is why the PCI DSS standards are invaluable for defining a security logging practice within an application and its requirements are well defined. Even if the industry you are working in is not Finance, these requirements, which are designed to protect the
organisations which look after your money, are still very relevant.
The OWASP Application Security Verification Standard (ASVS) Project is designed to normalize the range in the coverage and level of rigour available in the market when it comes to performing application security verification, the standard can be used to establish a level of confidence in the security of applications. Specifically, there are several sections referring to log security which can be directly translated into functional requirements for design and development teams.
As well as the entire section, V7: Error Handling and Logging Verification Requirements.
I couldn’t recommend highly enough that if you are looking to implement a logging strategy, investigate the OWASP ASVS.
Then there is the reason you really should not forget, especially if you are in/with EU or working with EU customers, that is GDPR
General Data Protection Regulation (GDPR)
GDPR doesn’t explicitly talk about logs, just like many other regulations, it is an industry-independent regulation for protecting citizens privacy, it also never states any technical implementation of their rules. It does however in the eyes of many data protection authorities (such as ICO) consider logs to be a good way of demonstrating compliance and “demonstrating compliance” is a key point of GDPR. Keeping logs of instances of processing activities is a best practice and can (and should) be done.
Some of the GDPR principles cover:
Tracking access to data, under the Accountability principle, especially when determining who accessed what and when. If access to data goes through a unified interface (UI and/or API), you can track all access to data and thus manifest that only the authorised personnel have read the data.
The GDPR principle of Integrity and Accuracy covers Tracking data modifications, you have to keep the data correct, so any modification should be logged. That way, you can reconstruct an old state or prove the modifications that happened for a reason.
Recording when the data subject invokes their rights. Each request can be securely logged so that you can prove to authorities the exact sequence of events relating to the particular data subject. This is a clear expectation of having an audit trail.
Logging consent and withdrawal and the accompanying circumstances, the history of the consent of the data subject must be available and you must be able to prove to regulators when you had and when you didn’t have consent for processing.
What about Brexit, if the UK is not a part of Europe, do they have to do this?
The GDPR is an EU Regulation and, in principle, it will no longer apply to the UK from the end of the transition period. However, if you operate inside the UK, you will need to comply with UK data protection law. The government intends to incorporate the GDPR into UK data protection law from the end of the transition period – so in practice, there will be little change to the core data protection principles, rights and obligations found in the GDPR.
The EU version of the GDPR may also still apply directly to you if you operate in Europe, offer goods or services to individuals in Europe, or monitor the behaviour of individuals in Europe.
The default position is the same as for a no-deal Brexit: the GDPR will be brought into UK law as the ‘UK GDPR’, but there may be time for further developments about how we deal with particular issues such as UK-EU transfers.
Is there anything else?
Yes, there are many other standards and guidance for various industries, which either explicitly or implicitly reference a need for logging, auditing, and monitoring. In conclusion, there are many reasons, beyond the legal requirement, how you can understand how your software is being used and misused, this allows you to build higher quality products because you can deliver products that your customers will actually use, and protect them
when someone misuses it.
- HIPAA – Security Rule
- SANS Information Logging Standard
- California Consumer Privacy Act (CCPA)
- Laroux – DEFCON 19: The History and the Evolution of Computer Viruses
- OWASP – A10-Insufficient Logging & Monitoring
- Logicworks – Security By Design
- GOV.uk – Minimum Cyber Security Guidance
- ISO/IEC 27001 – INFORMATION SECURITY MANAGEMENT
- NIST 800-92 – Guide to Computer Security Log Management
- PCI – Effective Daily Log Monitoring
- OWASP Application Security Verification Standard
- General Data Protection Regulation
- ICO GDPR – Accountability Principle
- ICO GDPR – Integrity and Confidentiality Principle
- ICO GDPR – Accuracy Principle
- ICO GDPR – Individual Rights
- ICO GDPR Consent
- HIPAA – Security Rule
- SANS – Information Logging Standard
- California Consumer Privacy Act (CCPA)