feat(logger): add support for AWS_LAMBDA_LOG_LEVEL and POWERTOOLS_LOG_LEVEL (#1795)

* feat(logger): support advanced logging

* docs(logger): add alc info

* feat(logger): support alc

* docs: fix alc docs links

* tests(logger): add unit tests for the feature

* docs(logger): make POWERTOOLS_LOG_LEVEL default

Author: dreamorosi

Diff for: docs/core/logger.md

@@ -46,12 +46,12 @@ The library requires two settings. You can set them as environment variables, or
 
 These settings will be used across all logs emitted:
 
-| Setting                | Description                                                                                                      | Environment variable            | Default Value       | Allowed Values                             | Example Value       | Constructor parameter |
-| ---------------------- | ---------------------------------------------------------------------------------------------------------------- | ------------------------------- | ------------------- | ------------------------------------------ | ------------------- | --------------------- |
-| **Service name**       | Sets the name of service of which the Lambda function is part of, that will be present across all log statements | `POWERTOOLS_SERVICE_NAME`       | `service_undefined` | Any string                                 | `serverlessAirline` | `serviceName`         |
-| **Logging level**      | Sets how verbose Logger should be, from the most verbose to the least verbose (no logs)                          | `LOG_LEVEL`                     | `info`              | `DEBUG`, `INFO`, `WARN`, `ERROR`, `SILENT` | `ERROR`             | `logLevel`            |
-| **Log incoming event** | Whether to log or not the incoming event when using the decorator or middleware                                  | `POWERTOOLS_LOGGER_LOG_EVENT`   | `false`             | `true`, `false`                            | `false`             | `logEvent`            |
-| **Debug log sampling** | Probability that a Lambda invocation will print all the log items regardless of the log level setting            | `POWERTOOLS_LOGGER_SAMPLE_RATE` | `0`                 | `0.0` to `1`                               | `0.5`               | `sampleRateValue`     |
+| Setting                | Description                                                                                                      | Environment variable            | Default Value       | Allowed Values                                         | Example Value       | Constructor parameter |
+| ---------------------- | ---------------------------------------------------------------------------------------------------------------- | ------------------------------- | ------------------- | ------------------------------------------------------ | ------------------- | --------------------- |
+| **Service name**       | Sets the name of service of which the Lambda function is part of, that will be present across all log statements | `POWERTOOLS_SERVICE_NAME`       | `service_undefined` | Any string                                             | `serverlessAirline` | `serviceName`         |
+| **Logging level**      | Sets how verbose Logger should be, from the most verbose to the least verbose (no logs)                          | `POWERTOOLS_LOG_LEVEL`          | `INFO`              | `DEBUG`, `INFO`, `WARN`, `ERROR`, `CRITICAL`, `SILENT` | `ERROR`             | `logLevel`            |
+| **Log incoming event** | Whether to log or not the incoming event when using the decorator or middleware                                  | `POWERTOOLS_LOGGER_LOG_EVENT`   | `false`             | `true`, `false`                                        | `false`             | `logEvent`            |
+| **Debug log sampling** | Probability that a Lambda invocation will print all the log items regardless of the log level setting            | `POWERTOOLS_LOGGER_SAMPLE_RATE` | `0`                 | `0.0` to `1`                                           | `0.5`               | `sampleRateValue`     |
 
 #### Example using AWS Serverless Application Model (SAM)
 
@@ -71,7 +71,7 @@ These settings will be used across all logs emitted:
           Runtime: nodejs20.x
           Environment:
             Variables:
-              LOG_LEVEL: WARN
+              POWERTOOLS_LOG_LEVEL: WARN
               POWERTOOLS_SERVICE_NAME: serverlessAirline
     ```
 
@@ -394,9 +394,9 @@ The error will be logged with default key name `error`, but you can also pass yo
 
 ### Log levels
 
-The default log level is `INFO` and can be set using the `logLevel` constructor option or by using the `LOG_LEVEL` environment variable.
+The default log level is `INFO` and can be set using the `logLevel` constructor option or by using the `POWERTOOLS_LOG_LEVEL` environment variable.
 
-Logger supports the following log levels:
+We support the following log levels:
 
 | Level      | Numeric value |
 | ---------- | ------------- |
@@ -421,11 +421,48 @@ The `SILENT` log level provides a simple and efficient way to suppress all log m
 
 This feature is useful when you want to have your code instrumented to produce logs, but due to some requirement or business decision, you prefer to not emit them.
 
-By setting the log level to `SILENT`, which can be done either through the `logLevel` constructor option or by using the `LOG_LEVEL` environment variable, you can easily suppress all logs as needed.
+By setting the log level to `SILENT`, which can be done either through the `logLevel` constructor option or by using the `POWERTOOLS_LOG_LEVEL` environment variable, you can easily suppress all logs as needed.
 
 !!! note
     Use the `SILENT` log level with care, as it can make it more challenging to monitor and debug your application. Therefore, we advise using this log level judiciously.
 
+#### AWS Lambda Advanced Logging Controls (ALC)
+
+With [AWS Lambda Advanced Logging Controls (ALC)](https://docs.aws.amazon.com/lambda/latest/dg/monitoring-cloudwatchlogs.html#monitoring-cloudwatchlogs-advanced), you can control the output format of your logs as either `TEXT` or `JSON` and specify the minimum accepted log level for your application. Regardless of the output format setting in Lambda, we will always output JSON formatted logging messages.
+
+When you have this feature enabled, log messages that don’t meet the configured log level are discarded by Lambda. For example, if you set the minimum log level to `WARN`, you will only receive `WARN` and `ERROR` messages in your AWS CloudWatch Logs, all other log levels will be discarded by Lambda.
+
+```mermaid
+sequenceDiagram
+    title Lambda ALC allows WARN logs only
+    participant Lambda service
+    participant Lambda function
+    participant Application Logger
+
+    Note over Lambda service: AWS_LAMBDA_LOG_LEVEL="WARN"
+    Lambda service->>Lambda function: Invoke (event)
+    Lambda function->>Lambda function: Calls handler
+    Lambda function->>Application Logger: logger.warn("Something happened")
+    Lambda function-->>Application Logger: logger.debug("Something happened")
+    Lambda function-->>Application Logger: logger.info("Something happened")
+    
+    Lambda service->>Lambda service: DROP INFO and DEBUG logs
+
+    Lambda service->>CloudWatch Logs: Ingest error logs
+```
+
+**Priority of log level settings in Powertools for AWS Lambda**
+
+When the Advanced Logging Controls feature is enabled, we are unable to increase the minimum log level below the `AWS_LAMBDA_LOG_LEVEL` environment variable value, see [AWS Lambda service documentation](https://docs.aws.amazon.com/lambda/latest/dg/monitoring-cloudwatchlogs.html#monitoring-cloudwatchlogs-log-level) for more details.
+
+We prioritise log level settings in this order:
+
+1. `AWS_LAMBDA_LOG_LEVEL` environment variable
+2. Setting the log level in code using the `logLevel` constructor option, or by calling the `logger.setLogLevel()` method
+3. `POWERTOOLS_LOG_LEVEL` environment variable
+
+In the event you have set a log level in Powertools to a level that is lower than the ACL setting, we will output a warning log message informing you that your messages will be discarded by Lambda.
+
 ### Using multiple Logger instances across your code
 
 The `createChild` method allows you to create a child instance of the Logger, which inherits all of the attributes from its parent. You have the option to override any of the settings and attributes from the parent logger, including [its settings](#utility-settings), any [persistent attributes](#appending-persistent-additional-log-keys-and-values), and [the log formatter](#custom-log-formatter-bring-your-own-formatter). Once a child logger is created, the logger and its parent will act as separate instances of the Logger class, and as such any change to one won't be applied to the other. 

Diff for: docs/core/metrics.md

@@ -47,6 +47,10 @@ Install the library in your project:
 npm install @aws-lambda-powertools/metrics
 ```
 
+!!! warning "Caution"
+    
+    Using the Lambda [Advanced Logging Controls](...docs link) feature requires you to update your version of Powertools for AWS Lambda (TypeScript) to at least v1.15.0 to ensure metrics are reported correctly to Amazon CloudWatch Metrics.
+
 ### Usage
 
 The `Metrics` utility must always be instantiated outside of the Lambda handler. In doing this, subsequent invocations processed by the same instance of your function can reuse these resources. This saves cost by reducing function run time. In addition, `Metrics` can track cold start and emit the appropriate metrics.

Diff for: docs/index.md

@@ -307,7 +307,7 @@ Core utilities such as Tracing, Logging, and Metrics will be available across al
 | **POWERTOOLS_LOGGER_LOG_EVENT**              | Log incoming event                                                                                            | [Logger](core/logger.md)                | `false`             |
 | **POWERTOOLS_LOGGER_SAMPLE_RATE**            | Debug log sampling                                                                                            | [Logger](core/logger.md)                | `0`                 |
 | **POWERTOOLS_DEV**                           | Increase JSON indentation to ease debugging when running functions locally or in a non-production environment | [Logger](core/logger.md)                | `false`             |
-| **LOG_LEVEL**                                | Set logging level                                                                                             | [Logger](core/logger.md)                | `INFO`              |
+| **POWERTOOLS_LOG_LEVEL**                     | Sets how verbose Logger should be, from the most verbose to the least verbose (no logs)                       | [Logger](core/logger.md)                | `INFO`              |
 | **POWERTOOLS_PARAMETERS_MAX_AGE**            | Adjust how long values are kept in cache (in seconds)                                                         | [Parameters](utilities/parameters.md)   | `5`                 |
 | **POWERTOOLS_PARAMETERS_SSM_DECRYPT**        | Set whether to decrypt or not values retrieved from AWS Systems Manager Parameters Store                      | [Parameters](utilities/parameters.md)   | `false`             |
 | **POWERTOOLS_IDEMPOTENCY_DISABLED**          | Disable the Idempotency logic without changing your code, useful for testing                                  | [Idempotency](utilities/idempotency.md) | `false`             |

Diff for: docs/snippets/batch/templates/sam/dynamodb.yaml

@@ -10,7 +10,7 @@ Globals:
     Tracing: Active
     Environment:
       Variables:
-        LOG_LEVEL: INFO
+        POWERTOOLS_LOG_LEVEL: INFO
         POWERTOOLS_SERVICE_NAME: hello
 
 Resources:

Diff for: docs/snippets/batch/templates/sam/sqs.yaml

@@ -10,7 +10,7 @@ Globals:
     Tracing: Active
     Environment:
       Variables:
-        LOG_LEVEL: INFO
+        POWERTOOLS_LOG_LEVEL: INFO
         POWERTOOLS_SERVICE_NAME: hello
 
 Resources:

Diff for: examples/cdk/src/example-stack.ts

@@ -19,7 +19,7 @@ const commonProps: Partial<NodejsFunctionProps> = {
     NODE_OPTIONS: '--enable-source-maps', // see https://docs.aws.amazon.com/lambda/latest/dg/typescript-exceptions.html
     POWERTOOLS_SERVICE_NAME: 'items-store',
     POWERTOOLS_METRICS_NAMESPACE: 'PowertoolsCDKExample',
-    LOG_LEVEL: 'DEBUG',
+    POWERTOOLS_LOG_LEVEL: 'DEBUG',
   },
   bundling: {
     externalModules: [

Diff for: examples/sam/template.yaml

@@ -64,7 +64,7 @@ Resources:
           SAMPLE_TABLE: !Ref SampleTable
           POWERTOOLS_SERVICE_NAME: items-store
           POWERTOOLS_METRICS_NAMESPACE: PowertoolsSAMExample
-          LOG_LEVEL: Debug
+          POWERTOOLS_LOG_LEVEL: debug
       Events:
         Api:
           Type: Api
@@ -104,7 +104,7 @@ Resources:
           SAMPLE_TABLE: !Ref SampleTable
           POWERTOOLS_SERVICE_NAME: items-store
           POWERTOOLS_METRICS_NAMESPACE: PowertoolsSAMExample
-          LOG_LEVEL: Debug
+          POWERTOOLS_LOG_LEVEL: debug
       Events:
         Api:
           Type: Api
@@ -134,7 +134,7 @@ Resources:
         Variables:
           POWERTOOLS_SERVICE_NAME: items-store
           POWERTOOLS_METRICS_NAMESPACE: PowertoolsSAMExample
-          LOG_LEVEL: Debug
+          POWERTOOLS_LOG_LEVEL: debug
       # add ssm:getParameter permission to the function
       Policies:
         - SSMParameterWithSlashPrefixReadPolicy:
@@ -186,7 +186,7 @@ Resources:
           SAMPLE_TABLE: !Ref SampleTable
           POWERTOOLS_SERVICE_NAME: items-store
           POWERTOOLS_METRICS_NAMESPACE: PowertoolsSAMExample
-          LOG_LEVEL: Debug
+          POWERTOOLS_LOG_LEVEL: debug
       Events:
         Api:
           Type: Api

Diff for: packages/logger/src/Logger.ts

@@ -2,6 +2,7 @@ import { Utility } from '@aws-lambda-powertools/commons';
 import type { HandlerMethodDecorator } from '@aws-lambda-powertools/commons/types';
 import type { Context, Handler } from 'aws-lambda';
 import merge from 'lodash.merge';
+import { format } from 'node:util';
 import { Console } from 'node:console';
 import { randomInt } from 'node:crypto';
 import { EnvironmentVariablesService } from './config/EnvironmentVariablesService.js';
@@ -486,9 +487,13 @@ class Logger extends Utility implements LoggerInterface {
   /**
    * Set the log level for this Logger instance.
    *
+   * If the log level is set using AWS Lambda Advanced Logging Controls, it sets it
+   * instead of the given log level to avoid data loss.
+   *
    * @param logLevel The log level to set, i.e. `error`, `warn`, `info`, `debug`, etc.
    */
   public setLogLevel(logLevel: LogLevel): void {
+    if (this.awsLogLevelShortCircuit(logLevel)) return;
     if (this.isValidLogLevel(logLevel)) {
       this.logLevel = this.logLevelThresholds[logLevel];
     } else {
@@ -570,6 +575,30 @@ class Logger extends Utility implements LoggerInterface {
     merge(this.powertoolsLogData, attributes);
   }
 
+  private awsLogLevelShortCircuit(selectedLogLevel?: string): boolean {
+    const awsLogLevel = this.getEnvVarsService().getAwsLogLevel();
+    if (this.isValidLogLevel(awsLogLevel)) {
+      this.logLevel = this.logLevelThresholds[awsLogLevel];
+
+      if (
+        this.isValidLogLevel(selectedLogLevel) &&
+        this.logLevel > this.logLevelThresholds[selectedLogLevel]
+      ) {
+        this.warn(
+          format(
+            `Current log level (%s) does not match AWS Lambda Advanced Logging Controls minimum log level (%s). This can lead to data loss, consider adjusting them.`,
+            selectedLogLevel,
+            awsLogLevel
+          )
+        );
+      }
+
+      return true;
+    }
+
+    return false;
+  }
+
   /**
    * It processes a particular log item so that it can be printed to stdout:
    * - Merges ephemeral log attributes with persistent log attributes (printed for all logs) and additional info;
@@ -845,17 +874,21 @@ class Logger extends Utility implements LoggerInterface {
 
   /**
    * Sets the initial Logger log level based on the following order:
-   * 1. If a log level is passed to the constructor, it sets it.
-   * 2. If a log level is set via custom config service, it sets it.
-   * 3. If a log level is set via env variables, it sets it.
+   * 1. If a log level is set using AWS Lambda Advanced Logging Controls, it sets it.
+   * 2. If a log level is passed to the constructor, it sets it.
+   * 3. If a log level is set via custom config service, it sets it.
+   * 4. If a log level is set via env variables, it sets it.
    *
-   * If none of the above is true, the default log level applies (INFO).
+   * If none of the above is true, the default log level applies (`INFO`).
    *
    * @private
    * @param {LogLevel} [logLevel] - Log level passed to the constructor
    */
   private setInitialLogLevel(logLevel?: LogLevel): void {
     const constructorLogLevel = logLevel?.toUpperCase();
+
+    if (this.awsLogLevelShortCircuit(constructorLogLevel)) return;
+
     if (this.isValidLogLevel(constructorLogLevel)) {
       this.logLevel = this.logLevelThresholds[constructorLogLevel];
 

Diff for: packages/logger/src/config/EnvironmentVariablesService.ts

@@ -20,16 +20,35 @@ class EnvironmentVariablesService
   implements ConfigServiceInterface
 {
   // Reserved environment variables
+  private awsLogLevelVariable = 'AWS_LAMBDA_LOG_LEVEL';
   private awsRegionVariable = 'AWS_REGION';
   private currentEnvironmentVariable = 'ENVIRONMENT';
   private functionNameVariable = 'AWS_LAMBDA_FUNCTION_NAME';
   private functionVersionVariable = 'AWS_LAMBDA_FUNCTION_VERSION';
   private logEventVariable = 'POWERTOOLS_LOGGER_LOG_EVENT';
-  private logLevelVariable = 'LOG_LEVEL';
+  private logLevelVariable = 'POWERTOOLS_LOG_LEVEL';
+  private logLevelVariableLegacy = 'LOG_LEVEL';
   private memoryLimitInMBVariable = 'AWS_LAMBDA_FUNCTION_MEMORY_SIZE';
   private sampleRateValueVariable = 'POWERTOOLS_LOGGER_SAMPLE_RATE';
   private tzVariable = 'TZ';
 
+  /**
+   * It returns the value of the `AWS_LAMBDA_LOG_LEVEL` environment variable.
+   *
+   * The `AWS_LAMBDA_LOG_LEVEL` environment variable is set by AWS Lambda when configuring
+   * the function's log level using the Advanced Logging Controls feature. This value always
+   * takes precedence over other means of configuring the log level.
+   *
+   * @note we need to map the `FATAL` log level to `CRITICAL`, see {@link https://docs.aws.amazon.com/lambda/latest/dg/configuration-logging.html#configuration-logging-log-levels AWS Lambda Log Levels}.
+   *
+   * @returns {string}
+   */
+  public getAwsLogLevel(): string {
+    const awsLogLevelVariable = this.get(this.awsLogLevelVariable);
+
+    return awsLogLevelVariable === 'FATAL' ? 'CRITICAL' : awsLogLevelVariable;
+  }
+
   /**
    * It returns the value of the AWS_REGION environment variable.
    *
@@ -89,12 +108,19 @@ class EnvironmentVariablesService
   }
 
   /**
-   * It returns the value of the LOG_LEVEL environment variable.
+   * It returns the value of the `POWERTOOLS_LOG_LEVEL, or `LOG_LEVEL` (legacy) environment variables
+   * when the first one is not set.
+   *
+   * @note The `LOG_LEVEL` environment variable is considered legacy and will be removed in a future release.
+   * @note The `AWS_LAMBDA_LOG_LEVEL` environment variable always takes precedence over the ones above.
    *
    * @returns {string}
    */
   public getLogLevel(): string {
-    return this.get(this.logLevelVariable);
+    const logLevelVariable = this.get(this.logLevelVariable);
+    const logLevelVariableAlias = this.get(this.logLevelVariableLegacy);
+
+    return logLevelVariable !== '' ? logLevelVariable : logLevelVariableAlias;
   }
 
   /**