Log monitoring is difficult for many reasons. For one thing there are not many events that unquestionably indicate an intrusion or malicious activity. If it were that easy the system would just prevent the attack in the first place. One way to improve log monitoring is to name implement naming conventions that imbed information about objects like user accounts, groups and computers such as type or sensitivity. This makes it easy for relatively simple log analysis rules to recognize important objects or improper combinations of information that would be impossible otherwise.

However asking for special naming convention changes for the sake of log monitoring may be difficult to pull off. It’s common to treat log monitoring as strictly one-way activity in relation to the production environment. By that I mean that security analysts are expected to monitor logs and detect intrusions with no interaction or involvement with the administrators of the systems being monitored other than for facilitating log collection.

I realize that such a situation may not be easy to change but if security analysts can have some input in the standards and procedures followed upstream from log collection they can greatly increase the detectability of suspicious or questionable security events. Here’s a few examples.

There are at least 3 kinds of user accounts that every organization uses:
• End-user accounts
• Privileged accounts for administrators
• Service/application accounts

Each of these 3 accounts are used in different ways and should be subject to certain best practice controls. For instance no person should ever logon to an interactive logon session (local console or remote desktop) with a service or application account. But of course a malicious insider or external threat actor is more than happy to exploit such accounts since they often have privileged authority and are frequently insecure because of difficulties in managing these accounts. Conversely, end-user and admin accounts assigned to people should not be used to run services and applications. Doing so will cause all kinds of problems. For instance, if Service A is running as User B and that user leaves the company, Service A will fail the next time it is started after User B is disabled. In audits I’ve seen highly privileged admin accounts of long departed employees still active because staff knew that there were different applications and services running with these credentials. This of course creates all kinds of security holes including residual access for the terminated employee.

Event ID 4624 makes it easy to distinguish between different logon session types with the Logon Type field. See the table below. But of course Windows can’t tell you what type of account just logged on. Windows doesn’t know the difference between end user, admin or service accounts. But if your naming convention embeds that information you can easily compare account type and logon type and alert on inappropriate combinations. Let’s say that your naming convention specifies that service accounts all begin with “s-“. Now all you need to do is set up a rule to alert you whenever it sees Event ID 4624 where Logon Type is 2 or 10 and account name is like “s-*”.

This is just one example of why it is so valuable to implement naming conventions that embed key information about objects. If you name groups with prefixes or something else that tags privileged groups as such, it now becomes very easy to detect whenever a member is added to privileged group. Perhaps you follow certain procedures to protect privileged accounts from pass-the-hash attacks such as limiting admins to only logging on to certain jump boxes. If privileged accounts and jump box systems are recognizable as such by their name then you can easily alert when a privileged account attempts logon from a non jump box system.

This of course requires upfront cooperation from administrators who may resistant to changing their naming styles just for the sake of logs. And you need to get to know the procedures and controls used to keep your network secure so that you can configure your SIEM to recognize when intruders or malicious insiders bypass these controls. But both challenges are worth the effort to face.

Unstructured data access governance is a big compliance concern.  Unstructured data is difficult to secure because there’s so much of it, it’s growing so fast and it is user created so it doesn’t automatically get categorized and controlled like structured data in databases.  Moreover unstructured data is usually a treasure trove of sensitive and confidential information in a format that bad guys can consume and understand without reverse engineering the relationship of tables in a relational database.

Most of this unstructured data is still found on file shares throughout the network, and file system permissions are the main control over this information.  Therefore knowing when permissions change unstructured is critical to governance and control. File permissions should normally be fairly static but end-users are (by default) the owner of files and subfolders they create and can therefore change permissions on those files. And of course, administrators can change permissions on any object.  Either way you need to know when this happens. Here’s how to do it with the Windows Security Log.

First we need to enable the File System audit subcategory.  You’ll find this in any group policy object under Computer ConfigurationWindows SettingsSecurity SettingsAdvanced Audit Policy ConfigurationSystem Audit PoliciesObject Access.  Enable File System for success.  (By the way, make sure you also enable Computer ConfigurationWindows SettingsSecurity SettingsLocal PoliciesSecurity OptionsAudit: Force audit policy subcategory settings to override audit policy category settings to make sure your audit policy takes effect.) Now you need to enable object level auditing on the root folders containing your unstructured data.  For example, if you have a shared folder called c:files, go to that folder in Windows Explorer, open the security tab of the folders properties, click Advanced and select the Auditing tab.  Now add an entry for Everyone that enables successful use of the Change permissions as shown below.

At this point Windows will begin generating two events each time you change permissions on this folder or any of its subfolders or files.  One event is the standard event ID 4663, “An attempt was made to access an object”, which is logged for any kind of audited file access like read, write, delete, etc.  That event will show WRITE_DAC under the Access Request Information but it doesn’t tell you what the actual permission change was.  So instead, use event ID 4670, “Permissions on an object were changed”, which provides the before and after permissions of the object under Permissions Change as shown in the example below.

“What does D:AI(A;ID;FA;;;AU)(A;ID;FA;;;WD)(A;ID;FA;;;BA)(A;ID;FA;;;SY)(A;ID;0x1200a9;;;BU) mean?” This is the original access control list of asdf.txt but in the very cryptic Security Descriptor Definition Language (SDDL).  SDDL definitely isn’t something you want to manually parse and translate on a regular basis, but you can when necessary.

Look for the “D:” which is close to the beginning of the string or even the very beginning in this case.  “D:” means Discretionary Access Control List (DACL) which are the actual permissions on the object as opposed to other things that show up in a security descriptor – like owner, primary group and the audit policy (aka SACL).  Until you hit another letter-colon combination like “S:” you are looking at the object’s permissions.  An ACL is made up of Access Control Entries which correspond to each item in the list you see in the Permissions tab of an object’s properties dialog.  But in SDDL before listing the ACEs comprising the ACL you will see any flags that affect the entire ACL as a whole.  In the example above you see AI as the first element after D:.  AI stands for SDDL_AUTO_INHERITED which means permissions on parent objects are allowed to propagate down to this object.

Now come the ACEs.  In SDDL, each ACE is surrounded by parenthesis and the fields within it delimited by semicolons.  The first ACE in the event above is (A;ID;FA;;;AU).  The first field tells you what type of ACE it is – either A for allow or D for deny.  The next field lists any ACE flags that specify whether this ACE is an inherited ACE propagated down from a parent object and if and how this ACE should propagate down to child objects.  The only flag in this ACE is ID which means the ACE is in fact inherited.  The next field lists the permissions this ACE allows or denies.  In this example FA stands for all file access rights.  The next 2 fields, Object Type and Inherited Object Type,  are always blank on file system permissions (hence the 3 semicolons in a row); they are only used places like Active Directory where there are different types of objects (user, group, computer, etc) that you can define permissions for.  Finally, the last field is Trustee and identifies the user, group or special principal begin allowed or denied access.  Here you will either see the SID of the user or group if the ACE applies to a so-called “well-known” SID you’ll the corresponding acronym.  In this example AU stands for Authenticated Users.

Event ID 4670 does a great job of alerting you when permissions change on an object and telling you which object was affected and who did it.  To go further and understand what permissions where actually changed you have to dive into SDDL.  I recommend Ned Pyle’s 2-part TechNet blog, The Security Descriptor Definition Language of Love for more information on SDDL.

By Randy Franklin Smith

Experimenting with Windows Security

Interest continues to build around pass-the-hash and related credential artifact attacks, like those made easy by Mimikatz. The main focus surrounding this subject has been hardening Windows against credential attacks, cleaning up artifacts left behind, or at least detecting PtH and related attacks when they occur. 

All of this is important – especially because end-users must logon to end-user workstations, which are the most vulnerable systems on the network.

Privileged admin accounts are another story. Even if you eliminated pass-the-hash, golden ticket, and other credential artifact attacks, you would remain vulnerable whenever admin accounts logon to insecure endpoints.  Keystroke logging, or simply starting a process under the current user’s credentials, are viable methods for stealing or hijacking the credentials of a locally logged-on user.

So, the big lessons learned with Mimikatz and privileged accounts are to avoid using privileged credentials on lower security systems, such as any system in which web browsing or email occurs, or any type of file or content is downloaded from the internet. That’s really what ESAE (aka Red Forest) is all about. But privileged accounts aren’t limited to just the domain admin accounts contemplated by the Red Forest. There’s many other privileged accounts for member servers, applications, databases, devices, and so on.

Privileged accounts should only be used from dedicated administrative workstations maintained at the same level of security as the resources being administered.

How do you implement controls that really enforce this kind of written policy? And how do you detect attempts to circumvent?

When it comes to Windows, you have a few options:

• Logon rights defined at the local system
• Workstation restrictions defined on the domain account
• Authentication silos

I’ll briefly explain each one and show how you can monitor attempts to violate the policies.

Logon Rights

There’s five logon types and corresponding “allow and deny rights” for each, with “deny” overriding “allow”, of course. You define these in group policy and they are enforced by the local systems in which the group policy objects are applied. For instance, if you have an OU for end-user Workstations and you assign “deny logon locally” to an AD admin group, those members won’t be able to logon at the console of workstations regardless of their authority.

If someone tries to violate a “deny logon” right you can catch this by looking for <a data-cke-saved-href=" www.ultimatewindowssecurity.com="" />event ID 4625 – an account failed to logon with status or sub-status code 0xc000015b. But be aware that these events are logged via the local workstation – not on the domain controller. This is another reason to use native Windows Event Collection to get events from your workstations.

Workstation Restrictions

This is something you’d have to specify on individual user accounts as shown below in Active Directory User and Computers. This control only applies to interactive logons.

In this example, I’ve allowed Tamas to logon only at SAW1 (secure admin workstation 1). Depending on how many SAWs and admins you have, this could be tedious. If Tamas tried to logon at a different workstation, that computer would log <a data-cke-saved-href=" www.ultimatewindowssecurity.com="" />event ID 4625 – an account failed to logon with status or sub-status code 0xC0000070. The domain controller would log event ID 4769 with failure code 0xC.

Authentication Silos

This is a new feature of AD that allows you to carve out groups of computers and users, and limit those users to those computers – centrally from AD Authentication policy silos, which are containers you can assign user accounts, computer accounts, and service accounts to. You can then assign authentication policies for this container to limit where privileged accounts can be used in the domain. When accounts are in the Protected Users security group, additional controls are applied, such as the exclusive use of the Kerberos protocol. With these capabilities, you can limit high-value account usage to high-value hosts. Learn more about silos in Implementing Win 2012 R2 Authentication Silos and the Protected Users Group to Protect Privileged Accounts from Modern Attacks.

When a user tries to logon outside the silo of permitted computers, the domain controller will log event ID 4820: A Kerberos Ticket-granting-ticket (TGT) was denied because the device does not meet the access control restrictions.

Bad guys have more methods and shrink-wrapped tools than ever to steal credentials, so it’s especially important to lock down privileged accounts and prevent artifacts of their credentials from being littered throughout your network where the bad guys can find them. Windows gives you controls for enforcing such policies and provides an audit trail when someone attempts to violate them. Remember that besides just non-compliant or forgetful admins, these events may signal a bad guy who’s successfully stolen privileged credentials but is unaware of the controls you’ve put in place.  So, take these events seriously.

If attackers can deploy a remote administration tool (RAT) on your network, it makes it so much easier for them. RATs make it luxurious for bad guys; it’s like being right there on your network. RATs can log keystrokes, capture screens, provide RDP-like remote control, steal password hashes, scan networks, scan for files and upload them back to home. So if you can deny attackers the use of RATs, you’ve just made life a lot harder for them.

We are getting better at catching so-called advanced persistent threats by detecting the malware they deploy on compromised systems. We can say this because experts are seeing more attackers “living off the land.” Living off the land means an attacker goes malware-free and instead relies on the utilities, scripting engines, command shells and other native resources available on systems where they gain an entry point.

By living off the land, they keep a much lower profile. They aren’t stopped as much by application control and whitelisting controls. There’s no malware for antivirus to detect.

And Windows provides plenty of native resources for this kind of attacker. (Linux and UNIX do too, but I’m focusing on Windows since client endpoints initially targeted by today’s attackers mostly run Windows.) You might be surprised how much you can do with just simple batch files, let alone PowerShell. And then there’s WMI. Both PowerShell and WMI provide a crazy amount of functionality. You can access remote systems and basically interface with any API of the operating system. You can open up network connections for “phoning home” to command and control servers, and more. This is all stuff that in years past required an EXE or DLL. Now you can basically do anything that a custom built EXE can do but without touching the file system which so much of our current security technology is based on.

How do you prevent attacks like this? PowerShell has optional security restrictions you can implement for preventing API access and limiting script execution to signed script files. With WMI it’s not as clear. Obviously, all the normal endpoint security technologies have a part to play.

But let’s focus on detection. It’s impossible to prevent everything and mitigate every vulnerability. So we can’t neglect detection. The challenge with detecting attackers that are living off the land is twofold. The activities you need to monitor:

1. Aren’t found in logs
2. Are happening on client endpoints

Both of these create big challenges. Let’s talk about #1 first. A.N. Ananth and I describe the types of activities that are clues to possible attacker living off the land in 5 Indicators of Evil on Windows Hosts using Endpoint Threat Detection and Response and I encourage you to watch that session which is full of good technical tips. But the point is that what you need to watch for isn’t in the Windows security log or other logs. Instead, detection requires a combination of file scanning, configuration checks, querying of running processes and so on — all stuff that requires code running on the local system or very powerful and complex remote access. If we were only talking about servers, we could consider deploying an agent. But to catch today’s threats, you need to be monitoring where they begin, which is on client endpoints — the desktops and laptops of your employees. And there’s no way to remotely reach into that many systems in real time, even if you overcame the technical hurdles of that kind of remote access. So that leaves agents, which always cause a degree of pushback.

But it’s time to stop calling them agents. Today what we need on endpoints are sensors. It’s a subtle but important shift in mindset. In the physical world, everyone understand the need for sensors, and that sensors have to be deployed where the condition is being monitored. If you want to know when someone enters your building at night, you need a sensor on every door. Likewise, if you want the earliest possible warning that your organizations have been compromised, you need a sensor on every endpoint.

So I encourage you to start thinking and speaking in terms of leveraging your endpoints as a sensor rather than yet another system that requires an agent. And look for security vendors that get this. EventTracker has done a great job of evolving their agent into a powerful and irreplaceable endpoint security agent that “sees” things that are just impossible to see any other way.

An area of audit logging that is often confusing is the difference between two categories in the Windows security log: Account Logon events and Logon/Logoff events.  These two categories are related but distinct, and the similarity in the naming convention contributes to the confusion. That being said, what is the difference between authentication and logon?  In Windows, when you access the computer in front of you or any other Windows computer on the network, you must first authenticate and obtain a logon session for that computer. A logon session has a beginning and end. An Account Logon event  is simply an authentication event, and is a point in time event.  Are authentication events a duplicate of logon events?  No: the reason is because authentication may take place on a different computer than the one into which you are logging.

Workstation Logons

Let’s start with the simplest case.  You are logging onto at the console (aka “interactive logon”) of a standalone workstation (meaning it is not a member of any domain).  The only type of account you can logon with in this case is a local user account defined in Computer Management Local Users and Groups.  You don’t hear the term much anymore but local accounts and SAM accounts are the same thing.  In this case both the authentication and logon occur on the very same computer because you logged on to the local computer using a local account.  Therefore you will see both an Account Logon event (680/4776) and a Logon/Logoff (528/4624) event in its security log.

If the workstation is a member of a domain, at this point it’s possible to authenticate to this computer using a local account or a domain account – or  a domain account from any domain that this domain trusts. When the user logs on with a domain account, since the user specifies a domain account, the local workstation can’t perform the authentication because the account and its password hash aren’t stored locally.  So the workstation must request authentication from a domain controller via Kerberos.  An authentication event (672/4768) is logged on which ever domain controller handles the authentication request from the workstation.  Once the domain controller tells the workstation that the user is authenticated, the workstation proceeds with creating the logon session and a records a logon event (528/4624) in its security log.

What if we logon to the workstation with an account from a trusted domain?  In that case one of the domain controllers in the trusted domain will handle the authentication and log 672/4768 there, with the workstation logging 528/4624 the same as above.

In all such “interactive logons”, during logoff, the workstation will record a “logoff initiated” event (551/4647) followed by the actual logoff event (538/4634).  You can correlate logon and logoff events by Logon ID which is a hexadecimal code that identifies that particular logon session.

Accessing Member Servers

After logging on to a workstation you can typically re-connect to shared folders on a file server.  What gets logged in this case?  Remember, whenever you access a Windows computer you must obtain a logon session – in this case a “network logon” session.  You might assume that the logon session begins when you connect to the share and then ends when you disconnect from it – usually when logging off your local workstation.  Unfortunately this is not the case: Windows servers only keep network logon sessions alive for as long as you have a file open on the server.  This accounts repeated logon/logoff events on Windows file servers by the same user throughout the course of the day.  With network logons, Windows 2003 logs 540 instead of 528 while Windows 2008 logs 4624 for all types of logons.

When you logon at the console of the server the events logged are the same as those with interactive logons at the workstation as described above.  More often though, you logon to a member server via Remote Desktop.  In this case the same 528/4624 event is logged but the logon type indicates a “remote interactive” (aka Remote Desktop) logon.  I’ll explain logon types next.

When looking at logon events we need to consider what type of logon are we dealing with: is this an interactive logon at the console of the sever indicating the user was physically present, or is it a remote desktop logon?  For that matter the logon could be associated with a service starting or a scheduled task kicking off.  In all such cases you will need to look at the Logon Type specified in the logon event 528/540/4624.  A full list of Logon Types is provided at the provided links for those events but in short:

Events at the Domain Controller

When you logon to your workstation or access a shared folder on a file server, you are not “logging onto the domain”. Each Windows computer is responsible for maintaining its own set of active logon sessions and there is no central entity aware of everyone who is logged on somewhere in the domain.  After servicing an authentication request, the domain controller doesn’t maintain information about how you were logging (console, remote desktop, network, etc) or when you logged off.

On domain controllers you often see one or more logon/logoff pairs immediately following authentication events for the same user.  But these logon/logoff events are generated by the group policy client on the local computer retrieving the applicable group policy objects from the domain controller so that policy can be applied for that user.  Then approximately every 90 minutes, Windows refreshes group policy and you see a network logon and logoff on the domain controller again.  These network logon/logoff events are little more than noise.  In forensic situations, they provide an estimate of how long the user was logged on (as long as the user remains logged on group policy will refresh about every 90 minutes), and can help to infer that the preceding authentication events for the same user were in conjunction with an interactive or remote desktop logon as opposed to a service or scheduled task logon.

What about the other service ticket related events seen on the domain controller? Basically, after your initial authentication to the domain controller which logs log 672/4768 you also obtain a service ticket (673, 4769) for every computer you logon to including your workstation, the domain controller itself for the purpose of group policy and any member servers such as in connection with shared folder access.  Then as computers remain up and running and users remain logged on, tickets expire and have to be renewed which all generate further Account Logon events on the domain controller.

The Facts: Good, Bad and Ugly

Both the Account Logon and Logon/Logoff categories provide needed information and are not fungible:  both are distinct and necessary.  Here are some important facts to understand, and accept about authentication and logon/logoff events.

1. To determine definitely how a user logged on you have find the logon event on the computer where the account logged on.  You can only make some tenuous inferences about logon type by looking at the domain controller and that requires analyzing multiple events.
2. To determine when a user logged off you have to go to the workstation and find the “user initiated logoff” event (551/4647).
3. To correlate authentication events on a domain controller with the corresponding logon events on a workstation or member server there is no “hard’ correlation code shared between the events.  Folks at Microsoft have suggested the Logon GUID field in these events would provide that but my research and experiments indicate that unfortunately the GUIDs are either not supplied or do not match.  So to make that correlation you basically have to dead reckon based on time, computer names and user account names.
4. Account Logon events on domain controllers are great because they allow you to see all authentication activity (successful or failed) for all domain accounts.  Remember that you need to analyze the security logs of all your domain controllers – security logs are not replicated between DCs.
5. Account Logon events on workstations and member servers are great because they allow you to easily pick out use of or attacks against local accounts on those computers.  You should be interested in that because using local accounts is bad practice and bad guys know they tend to be more vulnerable than domain accounts.  But, you don’t have to use Account Logon to detect logon attempts on local accounts; you can use Logon/Logoff events if you know what you are doing.  When viewing a Logon/Logoff event compare the domain name in the event details to the computer name that generated the event; if they match you are looking at a local account logon attempt – otherwise the domain name field with reflect some domain.  So can you survive with only enabling Logon/Logoff events on member servers and workstations?  I suppose so.
6. Logon/Logoff events are a huge source of noise on domain controllers because every computer and every user must frequently refresh group policy.  If you disable this category on domain controllers what will you lose?  You will lose some visibility into logons at the domain controller itself such as when an admin logs on at the console, via remote desktop or a service or scheduled task starts up.  In all cases Account Logon events will still be logged but see points 1 and 2 above.
7. Successful network logon and logoff events are little more than “noise “on domain controllers and member servers because of the amount of information logged and tracked.  Unfortunately you can’t just disable successful network logon/logoff events without also losing other logon/logoff events for interactive, remote desktop, etc.  Noise can’t be configured out of the Windows security log; that’s the job of your log management / SIEM solution.

Account Logon (i.e. authentication) and Logon/Logoff events.  All things considered, I’d like to see both categories enabled on all computers ideally.  I haven’t seen these events create a noticeable impact on the server but the amount of log data might exceed your log management / SIEM solution’s current capacity.  If you can’t afford to collect workstation logs, I still suggest enabling these 2 categories on workstations and letting the log automatically wrap after reaching 100MB or so.  Chances are the data will be there if you need it for forensic purposes.

Intrusion detection and compliance are the focus of log management, SIEM and security logging.  But security logs, when managed correctly are also the only control over rogue admins.  Once root or admin authority has been given to, or acquired by, a user, there is little they cannot do:  with admin authority, they can circumvent access or authorization controls by changing settings or using tools to leverage their root access to tamper with the internals of the operating system.

Audit logs, when properly managed, can serve as a control and deterrent against the privileged super user’s authority. Simply enabling auditing and deploying a log management solution may not suffice; to really be a deterrent, the audit log must be protected from deletion or tampering by rogue admins.

First and foremost, log data must be moved as frequently as possible from the system where it is generated to separate secure log repository.  Today’s enterprise log management solutions do a great job of frequent log collection and long term archiving.  However, who has privileged access to the log management solution and the systems on which it runs?

A log management process is not an effective control if administrators have privileged access to the log management components.  Though administrators should not be denied access to run reports, configure alerts or research logs, privileged access to the log management solution that allows someone to disable, erase or otherwise compromise the integrity of the log collection and archival process should be carefully managed.

A log management solution cannot serve as a deterrent over administrators who have privileged access at the application level or any of the infrastructure components on which it runs.  This includes:

• the operating system of the log management solution
• any database servers it uses
• the Active Directory forest in which the log manage server resides if a Windows server
• the NAS or SAN where it stores log data

And if the log management application or any of the above components run inside a virtual machine this also includes:

• the virtualization host, such as VMWare ESX(i), if it runs inside a virtual machine
• the virtualization manager, such as VMWare vCenter
• any of the components listed earlier which are used by or host the virtualization manager

Physical access to any of these components could potentially allow administrators to compromise the integrity of the audit trail.  To the extent possible, the log management solution should run on a completely separate infrastructure.

Remember such separation is a protection against not just internal rogue admins but outsiders who succeed in obtaining privileged access.  Typically the larger the organization, the more important and practical it is to achieve maximum separation between the log management solution and the environment it monitors.

Beyond hardware and software separation, the log management application, database servers, storage, OS and other components also need careful management. Larger organizations generally have dedicated information security teams, and usually within that group is someone responsible for the audit log management process.  For full accountability and separation of duty, that team should have no privileged access to production business systems monitored by the log management process.  Ideally that group would provide the oversight necessary for all components in the log management solution and supervise any action that touches the audit log to insure its integrity and prevent the introduction of backdoors into the system.

There are a host of reasons why even “supervised access” can be compromised:  staff in smaller IT shops aren’t always able to specialize so the possibility for separation of skills and duties may not exist. When an in-house log management system can’t be physically and logically separated, log management as a service may be an alternative to consider.  With cloud-based log management, the entire system is controlled by a professional service team at a separate site. Services can be set up with role-based access control so the ability to erase audit logs is controlled.  If organizations can overcome the frequent pushback to sending audit logs to the cloud, full isolation and integrity of log data can be achieved without building a separate log management system, and without the requirement of expertise for audit log management.

Whether an organization goes with an in-house audit log management or turns to the cloud-based service, it should carefully assess its choices in architecture and administrative responsibility. When the worst happens, audit logs may be the only deterrent and detective control over rogue admins.  Are they secure?

I think one of the most underutilized features of Windows Auditing and the Security Log are Process Tracking events.

In Windows 2003/XP you get these events by simply enabling the Process Tracking audit policy.  In Windows 7/2008+ you need to enable the Audit Process Creation and, optionally, the Audit Process Termination subcategories which you’ll find under Advanced Audit Policy Configuration in group policy objects.

These events are incredibly valuable because they give a comprehensive audit trail of every time any executable on the system is started as a process.  You can even determine how long the process ran by linking the process creation event to the process termination event using the Process ID found in both events.  Examples of both events are shown below.

Trying to determine what a user did after logging on to Windows can be difficult to piece together.  These events are valuable on workstations because often, they are the most granular trail of activity left by end-users: for example, you can tell that Bob opened Outlook, then Word, then Excel and closed Word.

The process start event tells you the name of the program and when it started.  It also tells you who ran the program and the ID of their logon session with which you can correlate backwards to the logon event. This allows you to determine the kind of logon session in which the program was run and where the user (if remote) was on the network using the IP address and/or workstation name provided in the logon event.

Process start events also document the process that started them using Creator Process ID which can be correlated backwards to the process start event for the parent process.  This can be invaluable when trying to figure out how a suspect process was started.  If the Creator Process ID points to Explorer.exe, after tracking down the process start event, then it’s likely that the user simply started the process from the start menu.

These same events, when logged on servers, also provide a degree of auditing over privileged users but be aware that many Windows administrative functions will all show up as process starts for mmc.exe since all Microsoft Management Console apps run within mmc.exe.

But beyond privileged and end-user monitoring, process tracking events help you track possible change control issues and to trap advanced persistent threats.  When new software is executed for the first time on a given system it’s important to know that, since it implies a significant change to the system or it could alert you to a new unauthorized and even malicious program running for the first time.

The key to this seeing this kind of activity is to compare the executable name in a recent event 592/4688 to executable names in a whitelist – and thereby recognizing new executables.

Of course, this method isn’t foolproof because someone could replace an existing executable (on your whitelist) with a new program but with the same name and path as the old.  Such a change would “fly under the radar” with process tracking.  But my experience with unauthorized changes that bypass change control and APTs indicates that while certainly possible, the methods described here-in will catch their share of offenders and attackers.

To do this kind of correlation you need to enable process tracking on applicable systems (all systems if possible, including workstations) and then you need a SIEM solution that can compare the executable name in the current event to a “whitelist” of executables.

How you build that whitelist is important because it determines if your criteria for a new executable is unique to “that” system, or if it is based on a “golden” system, or your entire environment.  The more unique your whitelist is to each system or type of system, the better.  You can build the whitelist by either scanning for all the EXE files on a given system or by analyzing the 592/4688 events over some period of time.  I prefer the latter because there are many EXE files on Windows computers that are never actually executed and I’d like to know the first time any new EXE is run – whether it came with Windows and installed applications out of the box or whether it is a new EXE recently dropped onto the system.  On the other hand if you only want to detect when EXEs run which were not present on system at the time the whitelist was created, then a list built from simply running “dir *.exe /s” will suffice.

If you opt to analyze a period of system activity make sure that the period is long enough cover the full usage profile and business process profile for that system – usually a month will do it. Take some time to experiment with Process Tracking events and I think you’ll find that they are valuable for knowing what running on your system and who’s running it.

Over the years, security admins have repeatedly asked me how to audit file shares in Windows.  Until Windows Server 2008, there were no specific events for file shares.  The best we could do was to enable auditing of the registry key where shares are defined.  But in Windows Server 2008 and later, there are two new subcategories for share related events:

• File Share
• Detailed File Share

File Share Events

This subcategory allows you to track the creation, modification and deletion of shared folders (see table below).  You have a different event ID for each of those three operations.  The events indicate who made the change in the Subject fields, and provides the name the share users see when browsing the network and the patch to the file system folder made available by the share.  See the example of event ID 5142 below.

The bad news is that the subcategory also produces event ID 5140 every time a user connects to a share.  The data logged, including who accessed it, and their client IP address is nice, but the event is logged much too frequently.  Since Windows doesn’t keep network logon sessions active if no files are held open, you will tend to see this event frequently if you enable the “File Share” audit subcategory.  There is no way to configure Windows to produce just the share change events and not this access event as well.  Of course, that’s the point of a SIEM and log management platform which is at the heart of Netsurion Open XDR, which filters out the noise.

Detailed File Share Events

Event ID 5140, as discussed above, is intended to document each connection to a network share, and as such it does not log the names of the files accessed through that share connection.  The “Detailed File Share” audit subcategory provides this lower level of information with just one event ID – 5145 – which is shown below.

This event tells identifies the user (Subject fields), the user’s IP address (Network Information), the share, and the actual file accessed via the share (Share Information) and then provides the permissions requested and the results of the access request.  This event actually logs the access attempt and allows you to see failure versions of the event as well as success events.

Be careful about enabling this audit subcategory because you will get an event for every file accessed through network shares each time the application opens the file.  This can be more frequent than imagined for some applications like Microsoft Office.  Conversely, remember that this category won’t catch access attempts on the same files if a locally executing application accesses the file via the local patch (e.g. c:docsfile.txt) instead of via a patch.

You might also want to consider enabling auditing on individual folders containing critical files and using the File System subcategory.  This method allows you to be much more selective about who, which files and what types of access are audited.

For most organizations, enable the File Share subcategory if it’s important to you to know when new folders are shared. You will probably want to filter out the 5140 occurrences.  Then, if you have file level audit needs, turn on the File Access subcategory, identify the exact folders containing the relevant files and enable auditing on those folders for the specific operations (e.g. Read, Write, Delete) needed to meet your audit requirements.  Don’t enable the Detailed File Share audit subcategory unless you really want events for every access to every file via network shares.

We hear a lot about tracking privileged access today because privileged users like Domain Admins can do a lot of damage. But more importantly, if their accounts are compromised the attacker gets full control of your environment.

In line with this concern, many security standards and compliance documents recommend tracking changes to privileged groups like Administrators, Domain Admins and Enterprise Admins in Windows, and related groups and roles in other applications and platforms.

But in some systems you can also granularly delegate privileged access, ultimately giving someone the same level of authority as a Domain Admin, but “underneath the radar.” This is especially true in AD. This capability is a double-edged sword. It’s necessary if you are going to implement least privilege, but it also creates a way for privileged access to be granted inadvertently, or even maliciously in such a way that will go unnoticed unless you are specifically looking for it. Here’s how:

First you need to enable “Audit Directory Service Changes” on your domain controllers — probably using the Default Domain Controllers Policy GPO.

Then open Active Directory Users and Computers and enable Advanced Features under View. Next select the root of the domain and open Properties. Navigate the Audit tab of the domain’s Advanced Security Settings dialog shown below.

advanced_security_settings

Add an entry for “Everyone” that audits “Modify permissions” on all objects like the entry highlighted above. At this point domain controllers will record Event ID 5136 whenever someone delegates authority of any object in the domain — whether an entire OU or a single-user account. Here’s an example event:

A directory service object was modified.

directory_service_object

This event tells you that a MTGpad-rsmith (that’s me) modified the permissions on the Scratch organizational unit in the MTG.local domain. nTSecurityDescriptor and “Value Added” tell us it was a permissions change. The Class field tells the type of object and DN gives us the distinguished name of the object whose permissions were changed. Subject tells us who made the change. I removed the lengthy text for Attribute Value because it’s too long to display and it’s in SDDL format which isn’t really human readable without a significant amount of effort. Technically, it does provide you with the full content of the OU’s new access control list (aka Security Descriptor) but it’s just not practical to try to decode it. It’s probably going to be faster to actually find the object in Active Directory Users and Computers and view its security settings dialog via the GUI.

So the Security Log isn’t perfect, but this method does give you a comprehensive audit trail of all permission changes and delegation within Active Directory. If you combine this with group membership auditing you’ll have a full picture of all changes that could impact privileged access in AD which is a key part of security and compliance.

HIPAA Logging HOWTO, Part 2

The Health Insurance Portability and Accountability Act of 1996 (HIPAA) outlines relevant security and privacy standards for health information – both electronic and physical. The main mission of the law is “to improve portability and continuity of health insurance coverage in the group and individual markets, to combat waste, fraud, and abuse in health insurance and health care delivery” (HIPAA Act of 1996 http://www.hhs.gov/ocr/privacy/).   A recent enhancement to HIPAA is called Health Information Technology for Economic and Clinical Health Act or HITECH Act. The act  seeks to “promote the adoption and meaningful use of health information technology” and “ addresses the privacy and security concerns associated with the electronic transmission of health information.“(HITECH Act of 2009 )

As we mentioned before, HIPAA itself does not descend to the level of security controls and technologies to implement.  This requires the organizations affected by HIPAA – also known as “covered entities” –to try to follow the spirit of the regulation as opposed to its letter.  What is also interesting to note is that insurance companies and many hospitals that accept payment cards are subject to both HIPAA and PCI DSS (covered in our previous newsletters). Understandably, the scope of their applicability across the organization might be different since payment processing systems should not store patient health information and vice versa.  Still, considering the same technical and administrative controls for both regulations is prudent and will save money in both the short term and long term.

The previous newsletter focused on general HIPAA logging and log review processes and platform logging. This newsletter installment covers application logging issues specific to medical applications.

While platform level logging is useful for protecting sensitive health information, and it is a fact that a majority of health information is stored in databases and processed by healthcare specific applications and.  Such applications are either procured from specialty vendors or developed internally – war via outsourced developers.

HIPAA of audit controls, mentioned in Section 164.312(b), apply to application logging as much or more than to platform logging. This means that custom applications need to be engineered to have adequate logging.  Existing application logging needs to be assessed having adequate logging – it should be noted that many legacy applications will often not record sufficient details for events and might even skip logging events altogether. Thus, before embarking on this project, it makes sense to determine which applications within your organization contain Protected Health Information (PHI) and what their existing levels and methods of logging are.

Let’s define some of the guidance for what to log to satisfy the spirit and letter of HIPAA Security Requirement as well as NIST 800-66 HIPAA clarifications.

Application Logging Guidance

Before we can define good HIPAA logging,  let’s consider typical security use for log data.

From high-level, best audit logs tell you exactly what happened – when, where and how- as well as who was involved. Such logs are suitable for manual, semi-automated and automated analysis. Ideally, the can be analyzed without having the application that produced them at hand – and definitely without having the application developer on call.  In case of healthcare applications, such developer might not be an available at all and the security team will have to proceed on their own. From the log management point of view, the logs can be centralized for analysis and retention.  Finally, they should not slow the system down and can be proven reliable, if used as forensic evidence.

Two primary things need to be defined.

• First, there are types of activities or events that always needs to be recorded.  For example, authentication decisions, health information access and system changes should always appear in logs.
• Second, for each type of a recorded event there a particulate details that a mandatory for its interpretation, whether by a human or by an automated system.  For example, every log should have a reliable time stamp and every log related to user activity should contain the username of that user.

It should also be noted that certain details should never be logged.  The example is obvious: application or system passwords should never appear in logs (this, sadly, still happens for web applications sometimes).  Just as obviously, the health information itself should be kept out of logs.

What events to log?

What is the overall theme for selecting which events to log?

Clearly, we need to know who, when and why accesses any of health information.  We also need to know who adds, changes or deletes it.  But this is not all – we also need to note who tries but fails to read, change or delete information.  If we are unable to record access to each piece of data, we need to carefully record all access to the application itself.

Next, we need to know who performs other actions on systems that process health information as such activities might affect future access to healthcare data.  For example, we need to record if somebody turns logging off or adds a new component to the system which might enable unfettered access to data.  In addition, we need to record other critical events a caring on health information systems, as such events might present circumstantial evidence for unauthorized access.

The following list presents a structured view of the above criteria:

• Authentication, Authorization, Access
  • Authentication/authorization decisions, successful and failed (see “status” below) – and especially privileged authentication
  • Recording user logoffs is also important for knowing when user no longer had access to the application
  • Switching from one user account to another
  • System access, data access, application component access
  • Network access to the application, including remote access from one application component to another in a distribute environment
• Changes
  • System/application changes (especially privilege changes)
  • Data change (creation and destruction are changes too)
  • Application and component installation and updates as well as removals
  • Sensitive data changes, additions and deletions
• Availability Issues
  • Startups and shutdowns of systems, applications and application modules/components
  • Faults and errors, especially those errors that affect the availability of the application
  • Backups successes and failures (affect availability)
•  “Badness” / Threats
  • Invalid inputs other likely application abuses
  • Malicious software detection events
  • Attempts – successful and failed-  to disrupt or disable security controls or logging
  • Logging termination events and possibly attempts to modify or delete the logs
  • Other security issues that are known to affect the application

While creating a comprehensive ”what to log” list for every healthcare application in existence is probably impossible, the above list should give you a useful starting point for your relevant applications.  It can be converted into your application logging policy without much extra work.  Please refer to previous newsletter for setting up log monitoring and review process.

What details to log?

Next, what data should you log for each event, and at what level of detail should you log it?  The overall theme we use here is the following:

Who was involved?

What happened?

Where did it happen?

When did it happen?

Why did it happen?

How did it happen?

The list below gives you a starting point based on that theme:

Timestamp + time zone : this helps to answer “when” question, time zone is essential for distributed applications

System, application or component: this helps to answer “where” question and needs to provide relevant application context as well

Source:  for messages related to network connectivity or distributed application  operation, logs also need to answer “where from” question by providing a network source.

Username: this helps to answer “who” question – for those events that are relevant to user or administrator activities

Action: this helps to answer “what” question by providing the nature of the event that is recorded in the log

Object: this also helps to answer “what” question by helping to know which system component or other object (such as user account) has been affected

Status:  this also helps to answer “what” question by explaining whether the action aimed at the object succeeded or failed (other types of status are also possible, such as “deferred”)

Priority: last but not least, every logged event should have an indication of how important it is.  While creating a uniform scale for renting events by importance is impossible since different organizations will have different priorities (for example, events affecting availability vs. confidentiality of information might be read differently)

Thus, a useful application audit log message might look like this:

2010/12/31 10:00:01AM GMT+7 priority=3, system=mainserver, module=authentication, source=127.0.0.1, user=anton, action=login, object=PHI_database, status=failed, reason=“password incorrect”

By the way, notice that another field is added to the above example log message in order to explain the reason for failure. Also notice that the above examples is not in XML – as we mention above, human readability is a useful property to have in logs and computers can deal with name=value pairs just as well as with XML. XML Health Level Seven (HL7) based messages can be easily converted to text, for those application that can log in HL7.

We also mentioned above that being able to easily centralize logs is essential for distributed log analysis either across multiple systems or across multiple application components of a distributed application.  While syslog has been the king of log centralization due to its easy UDP delivery, modern cross-platform application frameworks like a call for publish/subscribe model for log delivery, similar to the ones used in modern Windows versions.  In this case security monitoring tool can request a subscription for a particular type of a logged event – and receive all relevant logs in near real-time, if needed.

Conclusions

In addition to that very basic conclusion – you must log access to sensitive healthcare data– we have to remind our readers that the importance of logging will only grow – along with growing application complexity.  In particular, the need to analyze application behavior and movement of sensitive information across distributed and cloud-based application calls for us to finally get application logging under control.

Software architects and implementers need to “get” logging – there is NO other way since infrastructure logging from network devices and operating systems won’t do it  for detecting and investigating application level threats to ePHI. Security team – ultimately responsible for log review – will need to guide developers towards useful and effective logging that can be used for both monitoring and investigative activities.

Certainly, logging standards such as MITRE CEE (cee.mitre.org) will help – but it might take a few years before they developed and their adoption increases. Pending a global standard, an organization should quickly build and then use its own application logging standard for applications. HIPAA compliance presents a great motivation to creating and adopting such logging standards.

Industry News

Pirate Bay hack exposes user booty
Security weaknesses in the hugely popular file-sharing website thepiratebay.org have exposed the user names, e-mail and Internet addresses of more than 4 million Pirate Bay users… An Argentinian hacker named Ch Russo said he and two of his associates discovered multiple SQL injection vulnerabilities that let them into the user database for the site.

Zeus is back with terrorism-themed spam run
Trojan-laden emails claiming to offer official terrorism information have been hitting inboxes… The emails are spoofed to look like they originate from the U.S. Department of Homeland Security, Pentagon or Transportation Security Administration. Users are encouraged to click on two links, supposedly leading to reports, but which are actually ZIP files containing the insidious Zeus, or Zbot, trojan.

Related Resource: Webinar – Learn how implementing change control in your enterprise can help you handle critical security challenges including BOTnet and zero-day attacks.

Database admin gets 12 months for hacking employer
A former database administrator for Houston’s GEXA Energy was sentenced to 12 months in prison and fined $100,000 for hacking into his former employer’s network. He remotely accessed the GEXA Energy network without authorization, impaired the availability of data and copied a database file containing personal information. GEXA Energy estimates that Kim’s actions resulted in a loss of at least $100,000.

Did you know?  Security violations by insiders are often the hardest to discover, but cause the greatest damage and cost the most to repair. EventTracker helps by monitoring all user and admin activity, automatically detecting policy violations and out-of ordinary or suspicious behavior

I am often asked that if Log Management is so important to the modern IT department, then how come more than 80% of the market that “should” have adopted it has not done so?

The cynic says “unless you have best practice as an enforced regulation (think PCI-DSS here)” then twill always be thus.

One reason why I think this is so is because earlier generations never had power tools and found looking at logs to be hard and relatively unrewarding work. That perception is hard to overcome even in this day and age after endless punditry and episode after episode has clarified the value.

Still resisting the value proposition? Then consider a recent column in the NY Times which quotes Dov Seidman, the C.E.O. of LRN who describes two kinds of values: “situational values” and “sustainable values.”

The article is in the context of the current political situation in the US but the same theme applies to many other areas.

“Leaders, companies or individuals guided by situational values do whatever the situation will allow, no matter the wider interests of their communities. For example, a banker who writes a mortgage for someone he knows can’t make the payments over time is acting on situational values, saying: I’ll be gone when the bill comes due.”

At the other end, people inspired by sustainable values act just the opposite, saying: I will never be gone. “I will always be here. Therefore, I must behave in ways that sustain — my employees, my customers, my suppliers, my environment, my country and my future generations.”

We accept that your datacenter grew organically, that back-in-the-day there were no power tools and you dug ditches with your bare hands outside when it was 40 below and tweets were for the birds…but…that was then and this is now.

Get Log Management, it’s a sustainable value.

–Ananth

Logging for Incident Response: Part 1 – Preparing the Infrastructure

From all the uses for log data across the spectrum of security, compliance, and operations, using logs for incident response presents a truly universal scenario –you can be forced to use logs for incident response at any moment, whether you’re prepared or not.  An incident response (IR) situation is one where having as much log data as possible is critical. You might not use it all, and you might have to work hard to find the proverbial needle in the haystack of logs – still, having reliable log data from all – affected and unaffected – systems is indispensable in a hectic post-incident environment.

The security mantra “prevention-detection-response” still defines most of the activities of today’s security professionals. Each of these three components is known to be of crucial importance to the organization’s security posture. However, unlike detection and prevention, the response is impossible to avoid. While it is not uncommon for an organization to have weak prevention and nearly non-existent detection capabilities, they will often be forced into response mode by attackers or their evil creations – malware. Even in cases where ignoring the incident that happened might be the chosen option, the organization will implicitly follow a response plan, even if it is as ineffective as to do nothing.

In this paper, we will focus on how to “incident-response-proof” your logging – how to prepare your logging infrastructure for incident response. The previous six articles focused on specific regulatory issues, and it is not surprising that many organizations are doing log management just to satisfy compliance mandates. Still, technology and processes implemented for PCI DSS or other external mandates are incredibly useful for other uses such as incident response.  On top of this, many of the same regulations prescribe solid incident response practices (for additional discussion see “Incident management in the age of compliance”)

Basics
Even though a majority of incidents are still discovered by third parties (seeVerizon Breach Report 2010 and other recent research), it is clear that organizations should still strive to detect incidents in order to limit the damage stemming from extensive, long-term compromises. On the other hand, even for incidents detected by the third parties, the burden of investigation – and thus using logs for figuring out what happened –falls on the organization itself.

We have therefore identified two focal points for use of logs in incident response:

• Detecting incidents
• Investigating incidents

Sometimes the latter use-case is called “forensics” but we will stay away from such definitions since we would rather reserve the term “forensics” for legal processes.

Incident Response Model and Logs
While incidents and incident response will happen whether you want it to or not, a structured incident response process is an effective way to reduce the damage suffered by the organization.  The industry-standard SANS incident response model organizes incident response in six distinct stages (see (http://www.sans.org/rr/whitepapers/incident/Incident Management 101 Preparation & Initial Response (aka Identification)  By: Robin Dickerson (posted on January 17, 2005)

Preparation includes tasks that need to be done before the incident: from assembling the team, training people, collecting, and building tools, to deploying additional monitoring and creating processes and incident procedures

• Identification starts when the signs of an incident are seen and then confirmed, so that incident is declared
• Containment is important for documenting what is going on, quarantining the affected systems, as well as possibly taking systems offline
• Eradication is preparing to return to normal by evaluating the available backups, and preparing for either restoration or rebuilding of the systems
• Recovery is where everything returns to normal operation
• Follow-Up includes documenting and discussing lessons learned, and reporting the incident to management

Logs are extremely useful, not just for identification and containment as we mention above, but for all phases of incident response process.  Specifically, here is how logs are used at each stage of the IR process:

1. Preparation: incident response logs help us verify controls (for example, review login success and failure histories), collect normal usage data (learn what log messages show up during routine system activity), and perform a baseline (create log-based metrics that describe such normal activity), etc.
2. Identification: logs containing attack traces, other evidence of a successful attack, or insider abuse are pin-pointed, or alerts might be sent to notify about an emerging incident; also, a quick search and review of logs helps to confirm an incident, etc.
3. Containment: logs help us scope the damage (for example, firewall logs show which other machines display the same scanning behavior in case of a worm or spyware infestation), and learn what else is lost by looking at logs from other systems that might contain traces similar to the one that is known to be compromised, etc.
4. Eradication: while restoring from backups, we need to also make a backup of logs and other evidence:  preserving logs for the future is required, especially if there is risk of a lawsuit (even if you don’t plan to sue, the other side might)
5. Recovery: logs are used for confirming the restoration and then measures are put in place to increase logging so that we have more data in case it happens again; incident response will be much easier next time
6. Follow-Up: apart from summarizing logs for a final report, we might use the incident logs for peaceful purposes: training the new team members, etc.

As a result, the IT infrastructure has to be prepared for incident response logging way before the first signs of an incident are spotted.

Preparing the Infrastructure
In light of predominantly 3rd party incident discovery, the incident response process might need to be activated at any moment when notification of a possible incident arrives.  From this point onward, the security team will try to contain the damage and investigate the reason for the attack or abuse based on initial clues. Having logs will allow an organization to respond better and faster!

What logs needs to be collected for effective IR? This is very simple: any and all logs from networks, hosts, applications, and other information systems can be useful during response to an incident. The same applies to context data – information about users, assets, and vulnerabilities will come in handy during the panic of incident response. As we say above, having as much log data as possible will allow your organization to effectively investigate what happened, and have a chance of preventing its recurrence in the future.

Specifically, make sure that the following log sources have logs enabled and centrally collected:

• Network Devices – routers and switches
• Firewalls – including firewall modules in other Network Devices
• IDS, IPS and UTM devices – while firewalls are ubiquitous and can create useful logs, IDS/IPS alerts add to a useful dimension to IR process
• Servers running Windows, Unix and other common operating systems; logging should include all server components such as web servers, email servers, DNS servers, and other server components
• VPN logs are often key to IR since they can reveal who was accessing your systems from remote locations
• Web proxies – these logs are extremely useful for tracking “drive-by downloads” and other web malware attacks
• Database – logs from RDBMS systems contain records indicating access to data as well as changes to database systems
• Applications ranging from large enterprise applications such as SAP to custom and vertical applications specific to a company

Detailed discussion of logging settings on all those systems goes beyond the scope of this paper and might justify not just reading a document, but engaging specialty consultants focused on logging and log management.

Tuning Log Settings for Incident Response
What logs should be enabled on the systems covered above? While “log everything” makes for a good slogan, it also makes log analysis a nightmare by mixing together more relevant log messages with debugging logs which are used much less often, if at all. Still, many logging defaults should be changed as described below.

A typical Unix (Solaris, AIX, etc.) or Linux system will log the following into syslog: various system status and error messages, local and remote login/logout, some program failures, and system start/stop/restart messages. Logs that will not be found will be all logs tracking access to files, running processes, and configuration changes. For example, to log file access on Linux, one needs to use a kernel audit facility, and not simply default syslog.

Similarly, on Windows systems the Event Log will contain a plethora of system status and error messages, login/logout records, account changes, as well as system and component failures.  To have more useful data for incident response , one needs to modify the audit policy to start logging access to files and other objects.

Most web servers (such as Apache and Microsoft IIS) will record access to web resources located on a server, as well as access errors. Unlike the OS platforms, there is not a pressing need for more logging, but one can modify the /etc/http/httpd.conf to add logging of additional details, such as referrer and browser type.

Databases such as Oracle and MS SQL Server log painfully little by default, even though the situation is improving in recent database versions such as Oracle 11g. With older databases, you have to assume to have no database logs if you have not enabled them during the incident preparation stage. A typical database will log only major errors, restarts, and some administrator access, but will not log access, or changes to data or database structures.

Firewalls typically log denied or blocked connections, but not the allowed connections by default: as our case study showed, connection allowed logs are one of the most indispensable for incident response. Follow the directions for your firewall to enable such logging.

VPN servers will log connections, user login/logouts, errors; default logging will be generally sufficient.  Making sure that successful logins – not just failures-  are logged is one of the important preparation tasks for VPN concentrators.

Network IDS and IPS will usually log their alerts, various failures, user access to the sensor itself; the only additional type of “logging” is recording full packet payload.

Implementing Log Management
Log management tools that can collect massive volumes of diverse log data without issues are hugely valuable for incident response.  Having a single repository for all activity records, audit logs, alerts, and other log types allows incident responders to quickly assess what was going on during an incident, and what led to a compromise or insider abuse.

After logging is enabled and configured for additional details and additional logged events, the logs have to be collected and managed to be useful for incident response.  Even if a periodic log review process is not occurring, the logs have to be available for investigations.  Following the maturity curve (see http://chuvakin.blogspot.com/2010/02/logging-log-management-and-log-review.html), even simply having logs is a huge step forward for many organizations.

When organizations start collecting and retaining logs, the question of retention policy comes to the forefront.  Some regulations give specific answers: PCI DSS for example, mandates storing logs for one year.  However, determining proper log storage for incident response can be more difficult. One year might still be a good rule of thumb for many organizations, since it is likely that investigating incidents more than one year after they happened will be relatively uncommon,but certainly possible – so longer retention periods such as three years may be useful).

In the next paper, we will address how to start reviewing logs for discovering incidents, and also how to review logs during incident response. At this point, we have made a huge step forward by making sure that logs will be around when we really need them!

Conclusions
Even though compliance might compel organizations to enable logging, deploy log management, and even start reviewing logs, incident response scenarios allow the value of logs to truly manifest itself.

However, in order to use logs for incident response, the IT environment has to be prepared – follow the guidance and tips from this paper in order to “IR-proof” your logging infrastructure.  A useful resource to jumpstart your  incident response log review is “Critical Log Review Checklist for Security Incidents” which can be obtained at http://chuvakin.blogspot.com/2010/03/simple-log-review-checklist-released.html in various formats.

About Author

Dr. Anton Chuvakin (http://www.chuvakin.org) is a recognized security expert in the field of log management and PCI DSS compliance.  He is an author of books “Security Warrior” and “PCI Compliance” and a contributor to “Know Your Enemy II”, “Information Security Management Handbook”; he is now working on a book about computer logs.  Anton has published dozens of papers on log management, correlation, data analysis, PCI DSS, security management (see list www.info-secure.org) . His blog http://www.securitywarrior.org is one of the most popular in the industry.

In addition, Anton teaches classes (including his own SANS class on log management) and presents at many security conferences across the world; he recently addressed audiences in United States, UK, Singapore, Spain, Russia and other countries.  He works on emerging security standards and serves on the advisory boards of several security start-ups.

Currently, Anton is building his security consulting practice www.securitywarriorconsulting.com, focusing on logging and PCI DSS compliance for security vendors and Fortune 500 organizations.  Dr. Anton Chuvakin was formerly a Director of PCI Compliance Solutions at Qualys. Previously, Anton worked at LogLogic as a Chief Logging Evangelist, tasked with educating the world about the importance of logging for security, compliance and operations. Before LogLogic, Anton was employed by a security vendor in a strategic product management role. Anton earned his Ph.D. degree from Stony Brook University.