Blog

With the rise of infrastructure complexity, organizations must improve their strategies to quickly investigate and mitigate unauthorized system access and internal identity threats. Teleport has already highlighted the importance of identity threat detection and response and introduced features to support security incident containment. This article builds on these ideas by presenting additional metrics to detect suspicious employee behavior and options for expanding detection capabilities. It explores practical steps and data sources that can be used in Teleport during an incident response.

Response

When an incident is detected, blocking mechanisms and data sources can help analyze the blast radius and understand the extent of the damage.
The response usually includes actions to track, contain, and eliminate the threat. Recovery and prevention of future occurrences are also part of the high-level plan.
Due to differences in processes, infrastructures, and tech stacks, companies need tailor-made incident response plans to guide internal staff during a breach.
Teleport offers many actionable features to assist with practical response operations.

Teleport in Your Response Strategy

Here are some core Teleport features usable in practical response operations:

• Session and Identity Locking – Administrators can disable a compromised user or Teleport agent by placing a lock on a session, user, or host identity.
• Moderated Sessions – Depending on requirements, users who join others’ sessions can observe, participate, or terminate rogue sessions.
• Access Monitoring – Administrators can run queries against cluster events, extending the threat analysis beyond initial IoCs.
• Exporting Teleport Audit Events – The Event Handler plugin can export audit events from Teleport to a log management platform or custom backend, providing a wide range of event data.
• Session Recording – Interactive sessions via ssh or kubectl exec -ti are recorded and can be replayed later.

As discussed in our previous article about “Detecting Impersonators and Compromised Identities with Teleport,” identifying rogue users within an organization is challenging. That article introduced some indicators of compromise (IoCs) in Teleport and Access Monitoring queries to collect them.

But what happens after the initial detection? While discovering illegitimate behaviors is difficult, once a rogue identity or node is confirmed, the process advances toward containment and blast radius evaluation.

Incident Containment – Locking Out Rogue Resources

Locking out rogue resources is often the first action after detecting malicious activity in the network. Once a lock is placed on a suspected resource, Teleport will reject new API requests and terminate any active SSH, database, desktop, and Kubernetes sessions to the locked resource.

Here are some examples and commands to leverage the available lock types:

# locking a rogue user. Useful in cases with complex roles-mesh to prevent lock of other legit identities
$ tctl lock [email protected] --message="Rogue activity." --ttl=10h
Created a lock with name "<LOCK_ID>".

# locking a specific role. Useful in cases where specific roles are assigned to teams or user-sources. E.g. SSO users could be locked all at once by assigned them sso-prefixed roles lockable in case of IdP compromise
$ tctl lock --role=sso-admin --message="All sso-admin access is disabled for 10h. The IdP was compromised" --ttl=10h
Created a lock with name "<LOCK_ID>".

# locking a specific device. Useful in case of lost / stolen device
$ tctl lock --device <DEVICE_ID> --message="Compromised device" --ttl=48h
Created a lock with name "<LOCK_ID>".

# locking an MFA device
$ tctl lock --mfa-device=<DEVICE_ID> --message="Rogue device locked for 10h." --ttl=10h
Created a lock with name "<LOCK_ID>".

# locking a potentially compromised server
$ tctl lock --server-id=<SERVER_ID> --message="The server running the Kubernetes Service and Database Service is under investigation for rogue activity." --ttl=10h
Created a lock with name "<LOCK_ID>".

# locking specific Windows Desktop instances
$ tctl lock --windows-desktop=<WIN_DESKTOP_ID> --ttl=10h
Created a lock with name "<LOCK_ID>".

Tracking & Responding to Identity Lateral Movements

Once malicious actors have a foothold in the infrastructure, they often move laterally to expand their actions and maintain access. Even if the initial exploited identity is detected, they can stay within the cluster using various compromised resources.

Therefore, the initial IoCs should be used to quickly track events and potential lateral movements from the confirmed rogue resource. We’ll use Teleport’s Access Monitoring feature to run queries against events involving a confirmed <COMPROMISED_IDENTITY>.

Multiple compromised identities could be present and used to hide escalations with legitimate access requests. Additionally, internal phishing operations might have led to unwanted approvals. The following query retrieves all reviewed Access Requests where the <COMPROMISED_IDENTITY> was either the requester or reviewer. This can reveal new identities to lock.

SELECT
  *
FROM
  access_request_create, access_request_review
WHERE
  access_request_create.id = access_request_review.id AND (access_request_review.reviewer = '<COMPROMISED_IDENTITY>' OR access_request_create.user = '<COMPROMISED_IDENTITY>')

If you find that access requests were abused, request locking will help contain the breach. This can be done using tctl:

$ tctl lock --access-request=<REQUEST_ID> --ttl=24h
Created a lock with name "<LOCK_ID>".

Apart from regular users, Teleport includes built-in agents with defined roles. Attackers might move to an agent role by creating a join token. The following query can be used to determine if the rogue user has created join tokens to obtain a built-in role in the cluster:

SELECT 
  user as compromised_identity, roles as exfiltered_token_roles, uid, time 
FROM
  join_token_create
WHERE 
  user = '<COMPROMISED_IDENTITY>'

While the impact of the created token may be limited depending on its type, removing or revoking them should be the next course of action. This can be done from the command line using tctl:

$ tctl tokens rm <TOKEN>
Token <TOKEN> has been deleted

To enumerate user accounts created or updated by the <COMPROMISED_IDENTITY>:

SELECT 
  user as compromised_identity, name as created_user, roles as assigned_role, time
FROM
  user_create
WHERE 
  user = '<COMPROMISED_IDENTITY>' or updated_by = '<COMPROMISED_IDENTITY>'

Attackers may attempt to achieve persistence by generating identity files. Such actions can be detected by querying for certificate creation actions performed by the malicious actor:

SELECT 
  identity_user as compromised_identity, event_time 
FROM 
  cert_create
WHERE 
  identity_user = '<COMPROMISED_IDENTITY>' AND event = 'cert.create' 
ORDER BY 
  event_time DESC;

Access to user accounts created by the attacker, as well as signed identities, can be prevented by locking the account. As mentioned in the “Incident Containment – Locking Out Rogue Resources” section, locking can be performed with the following command:

$ tctl lock [email protected] --message="Compromised account" --ttl=10h
Created a lock with name "<LOCK_ID>".

“Access Lists” are one of the main tools for access control in Teleport. Users and bots receive additional roles through their memberships in access lists. Editing access lists could be abused to hide memberships for new rogue identities.

Find out if attackers have edited the members of an access list:

SELECT 
  'access_list_member_create' AS table_name, uid, access_list_name, updated_by as compromised_identity, event, members as members_edit, time
FROM 
  access_list_member_create
WHERE 
  updated_by = '<COMPROMISED_IDENTITY>'

UNION ALL

SELECT 
  'access_list_member_delete' AS table_name, uid, access_list_name, updated_by as compromised_identity, event, members as members_edit, time
FROM 
  access_list_member_delete
WHERE 
  updated_by = '<COMPROMISED_IDENTITY>'

UNION ALL

SELECT 
  'access_list_member_update' AS table_name, uid, access_list_name, updated_by as compromised_identity, event, members as members_edit, time
FROM 
  access_list_member_update
WHERE 
  updated_by = '<COMPROMISED_IDENTITY>';

If any changes to the access list are detected, administrators and incident responders must mitigate the issue.
They should conduct a full review of the changes made by the attacker and restore the previous state of the access list, usually by manually reverting the unwanted changes.

Another useful tracking query involves finding the distinct resources accessed by the compromised identity, the relative IPs, and available session recordings.
With the session recording ID, the response team can replay and further inspect the actions taken by the attackers:

SELECT 
  user as compromised_identity, with_mfa, addr_remote as threat_actor_ip, event_time, event, login, namespace, proto, server_hostname, server_id, sid as session_recording_id
FROM 
  session_start
WHERE
  user = '<COMPROMISED_IDENTITY>'

Attackers with access to compromised credentials may try to execute SSH commands directly using tsh. Commands executed using this method are not subject to screen recording. However, they can be accessed from the exec event table.

In such cases, the executed commands can be retrieved using the following query:

SELECT 
  user as compromised_identity, command, event_time
FROM 
  exec 
WHERE 
  user = '<COMPROMISED_IDENTITY>'; 

The command can be further filtered for potentially dangerous actions, such as attempts to exploit known vulnerabilities or access sensitive system files.

What’s Next?

“Know your cluster” should be the starting point.

Analyze the cluster configuration to derive valuable queries and events to help find initial IoCs. This will enable effective tracking of movements like those in the real-world scenarios described above.

For example, if the cluster relies heavily on an external Identity Provider (IdP) as its user source, the incident response queries should be crafted to filter edits coming from the SSO plugin and users based on how they are referenced within the system.

Using Teleport events for detecting anomalies is a robust and flexible mechanism. This article contains just a few ideas for how Teleport’s Access Monitoring and Locking features can be used during threat response operations. We recommend reviewing the Access Monitoring documentation and Exporting Teleport Audit Events for more information and examples.