Audit log entry stale username field

[albertzaharovits]

The username audit field, which is present in most audit entries, might not reflect the authentication status when the entry is recorded.

Actions are first authenticated, and then continue execution (after authorization) using the point-in-time authentication. This is not a problem for one-off short lived actions.

But there are two broad situation where this behavior is confusing:

• when ML datafeed and watcher search input (possibly other, eg rollup) background jobs are audited, the username field reflects the authentication that has been performed when the job has been created.
• audited actions when using API keys, will show up with the username that created the API key.

These cases are confusing because the authenticating users might change (have a different set of roles) or might even be deleted, yet they still show up in the audit log as active.

Auditing the user roles, see #30032, and documenting this limitation might be enough?

Otherwise, I think, the "best" fix is to audit the creation of background jobs and API keys, attribute them an username-like identifier (which is included in the audit entry for the job/key creation), and use that identifier in all subsequent actions associated with the job/key. But there are more details we need to has out, for example the username (unlike the username-like identifier) allows for easiest attribution (when the authentication is not stale), so maybe keeping the field and signaling in some other way that the user might have changed is a better way.

[elasticmachine]

Pinging @elastic/es-security (:Security/Audit)

[albertzaharovits]

Another option is to record the timestamp of the authentication. So in these cases, the audit entry would contain a timestamp for eg. authorization and a timestamp for authentication.

[ywangd]

For this particular issue, I personally think 1) better documentation and 2) adding more description in the logs (e.g. timestamp as Albert suggested) would probably suffice.

But the "point-in-time" authentication (and to some extent authorization since role names are snapshot'd as well) could be a deeper issue to talk about. This issue touches it as well. I am not suggesting any actions. More of trying to understand people's thoughts on this.

[tvernum]

I think

• the username we audit is the correct one (a recurring background task like a ML datafeed is running on behalf of that user). I don't think we should mess with it
• including in the audit record that this is some sort of "background job" that runs without an explicit authentication event would be helpful. It's more explicit, and would assist in many of the cases where audit logs are needed. If that "flag" should refer to the job (e.g. watcher-watch-foo, ml-datafeed-bar) that would be even better.
• for non-API key audit records, the roles we audit are the correct names (they are the roles that this is executing under) even if that user no longer has those roles. The "background job" might be enough make that scenario explicit, but if there's another flag that would be useful to add, I would have no objections.
• for API keys, our audit records have a gap. API keys don't have named roles, they have privileges, and we ought to audit them more precisely (I think this point should be split into a separate issue).
• adding the timestamp for the authentication is interesting, but I wonder if really helpful. Happy to discuss.

[jkakavas]

including in the audit record that this is some sort of "background job" that runs without an explicit authentication event would be helpful. It's more explicit, and would assist in many of the cases where audit logs are needed. If that "flag" should refer to the job (e.g. watcher-watch-foo, ml-datafeed-bar) that would be even better.

+1

Another option is to record the timestamp of the authentication. So in these cases, the audit entry would contain a timestamp for eg. authorization and a timestamp for authentication.

adding the timestamp for the authentication is interesting, but I wonder if really helpful.

I don't dislike it either, but thinking out loud here: Doesn't this fall outside the scope of audit logging ? It feels like we're providing this as an explanatory piece of information, in order to help the reader of the log to realize how background jobs authentication works. From an audit perspective, isn't the original authentication audited already ? (when it happened )

[tvernum]

Doesn't this fall outside the scope of audit logging ?

I think it does, unless we're saying it's a proxy for a "session", so you can correlate your authz audit events back to the original authc event. That's a reasonable feature in an audit log.

[alexPolisevschi]

An Elastic support engineer directed me to this topic. I've been using Elasticsearch for over a year, for work. We have auditing enabled, as a requirement from our security department.

Recently I enabled CCR. I found it very confusing that the CCR entries from the audit log had as 'username' the person that configured the replication. Also, the number of audited events was huge. The solution was to create a dedicated CCR user with limited permissions and configure the job with this one (then ignore/filter this user from auditing).

In a similar scenario are ML jobs. I recently restarted an old ML job; it ran overnight and analysed a large amount of data, during which it also audited heavily (30GB in only a few hours). This put more pressure than I expected on the cluster, as various alarms were triggered. It took me a while to find out the cause - because even if I was logged in with my own user when restarting the datafeed, this particular job had been created by the elastic user. Also, there was no idicator that it was a ML job. All I had was the request type InternalScrollSearchRequest and event type transport.

So +1 on the below:

including in the audit record that this is some sort of "background job" that runs without an explicit authentication event would be helpful. It's more explicit, and would assist in many of the cases where audit logs are needed. If that "flag" should refer to the job (e.g. watcher-watch-foo, ml-datafeed-bar) that would be even better.

To be honest, I wasn't expecting CCR or ML to be executed & audited with the user that created them. I assumed there would be an internal user for this (just like we have beats_system, apm_system etc).
Because once created, these jobs aren't executed by user X, they are ran by "the system".

Not to mention AD integration - you have a user today configuring ML jobs. The next day the user leaves the company and is deleted from AD (the elasticsearch cluster has no control or awarness of this). You now have ML jobs running as user X that doesn't even exist, and this is, imo, wrong.

[albertzaharovits]

Thank you for the insights @alexPolisevschi . Much appreciated!

We've discussed this inside the team probably a month ago, but I've unfortunately deferred to update this thread.

The consensus was on the lines of what has been mentioned already in the thread, keeping in mind that auditing API keys and background jobs are distinct issues, ie :

• find a way to link the audit records to the background job that they are ascribed to
• document this behavior, similar to how ML does for data feeds https://www.elastic.co/guide/en/machine-learning/current/ml-dfeeds.html#ml-dfeeds

In addition, after further thought and after considering the feedback from @alexPolisevschi , I wish to contend @tvernum's

the username we audit is the correct one (a recurring background task like a ML datafeed is running on behalf of that user). I don't think we should mess with it

I think the intuition is that background tasks are executed "by the system", not by the user. The exponent example for this argument is the ShardFollowTaskExecutor for CCR, which does an indices stats on the remote cluster.
I also think that because the audit entry contains the roles, which confer the privileges that the task is operating under, as well as the username which might be assigned a different set of roles at the time when the audit record is produced, can give rise to confusion about what exactly the privileges of the task are.
This is in addition to the confusion, which has been discussed already, that the username can reference retired users.

I propose we keep the username designation only for synchronous actions, and instead use the started-by-username designation for background jobs (audit records of background jobs would not contain a username field anymore). @tvernum @jkakavas @ywangd @alexPolisevschi May I get your thoughts on this proposal? Regarding audit log ignore policies I believe they should continue working as they currently do, and filter audit logs of background jobs of specific users (even if the audit entry does not contain the username field, but contains the started-by-username field).

In terms of implementation details, we'll have to adjust the client thread context of the background tasks. The audit log would then interpret this thread context change, and slightly alter the audit entry, as described above, to signal that this corresponds to a background task. For reference here are all the background tasks I've been able to track down:

• shard follow task
• transforms
• ML dataframe analytics
• ML datafeed
• ML jobs
• rollup jobs (all the above are subclasses of PersistentTasksExecutor)
• Watcher Index action (ExecutableIndexAction)
• Watcher Search input (ExecutableSearchInput)

Other uses for tasks such as enrich policies are out of scope, since they are synchronous by comparison.

[tvernum]

I think the intuition is that background tasks are executed "by the system", not by the user

That may be the inituition, but it's not the reality from a security point of view. A running job has the privileges of the user that created it. Different tasks run with different sets of privileges and may do different things.
I do not understand the desire to hide that or pretends it's not true.

I propose we keep the username designation only for synchronous actions, and instead use the started-by-username designation for background jobs (audit records of background jobs would not contain a username field anymore). @tvernum @jkakavas @ywangd @alexPolisevschi May I get your thoughts on this proposal? Regarding audit log ignore policies I believe they should continue working as they currently do, and filter audit logs of background jobs of specific users (even if the audit entry does not contain the username field, but contains the started-by-username field).

I do not understand why introducing a new field, to record exactly the same information, and behave in exactly the same way, would be a gain here.
It doesn't reduce the "noise" in the audit log, it doesn't remove the fact that it could reference a user that no longer has access, it feels like a change for the sake of change without solving any real problem.

[ywangd]

I agree that the "system user" seems to be the right concept but lacks of concrete entity in these use cases. Privileges required by these jobs cannot be covered by a single system user since they always involve user data. My understanding is that we will never grant system users permissions for user indices. So it's not possible to assign a system user for it. Sure the superuser can do everything, but I'd argue using superuser to configure these things is a problem by itself and should be fixed separately.

The original problem seems to have two asks:

1. Somehow make it clear that the username in audit log is the one who create the job
2. An easy way to filter the audit log when a more common user (e.g. elastic) is used to create the job.

Instead of trying to move around the username, I wonder if it is possible to add another flag in the audit log entry to indicate it is triggered by the system (task or other origin etc.). This could help differentiate a background task from a synchronous one (with help from documentation improvement). And also allow filtering using this new flag?