azureblobexporter

Azure Blob Exporter for OpenTelemetry Collector

Status
Stability	alpha: traces, metrics, logs
Distributions	contrib
Issues
Code coverage
Code Owners	@hgaol, @MovieStoreGuy

Configuration

The following settings are required:

• url: Must be specified if auth type is not connection_string. If auth type is connection_string, it's optional or will be override by the auth.connection_string. Azure storage account endpoint. This setting might be replaced with endpoint for future. e.g. https://.blob.core.windows.net/
• auth (no default): Authentication method for exporter to ingest data.
  • type (no default): Authentication type for expoter. supported values are: connection_string, service_principal, system_managed_identity, user_managed_identity and etc.
  • tenand_id: Tenand Id for the client, only needed when type is service_principal.
  • client_id: Client Id for the auth, only needed when type is service_principal and user_managed_identity.
  • client_secret: Secret for the client, only needed when type is service_principal.
  • connection_string: Connection string to the endpoint. Only needed for connection_string auth type. Once provided, it'll override the url parameter to the storage account.

The following settings can be optionally configured and have default values:

• container: container for metrics, logs and traces. A container organizes a set of blobs, similar to a directory in a file system. More details can refer this.
  • metrics (default metrics): container to store metrics. default value is metrics.
  • logs (default logs): container to store logs. default value is logs.
  • traces (default traces): container to store traces. default value is traces.
• blob_name_format: the final blob name will be blob_name +
  • metrics_format (default 2006/01/02/metrics_15_04_05.json): blob name format. The date format follows constants in Golang, refer here.
  • logs_format (default 2006/01/02/logs_15_04_05.json): blob name format.
  • traces_format (default 2006/01/02/traces_15_04_05.json): blob name format.
  • serial_num_range (default 10000): a range of random number to be appended after blob_name. e.g. blob_name_{serial_num}.
  • serial_num_before_extension (default false): places the serial number before the file extension if there is one. e.g blob_name_{serial_num}.json instead of blob_name.json_{serial_num}
• format (default json): json or proto. which present otel json or otel protobuf format, the file extension will be json or pb.
• encodings (default using encoding specified in format, which is json): if specified, uses the encoding extension to encode telemetry data. Overrides format.
  • logs (default nil): encoding component id.
  • metrics (default nil): encoding component id.
  • traces (default nil): encoding component id.
• append_blob: configures append blob behavior. When enabled, telemetry data is appended to a single blob instead of creating new blobs. This can be useful for aggregating data or reducing the number of blobs created.
  • enabled (default false): determines whether to use append blob mode.
  • separator (default \n): string to insert between appended data blocks.
• retry_on_failure
  • enabled (default = true)
  • initial_interval (default = 5s): Time to wait after the first failure before retrying; ignored if enabled is false
  • max_interval (default = 30s): Is the upper bound on backoff; ignored if enabled is false
  • max_elapsed_time (default = 120s): Is the maximum amount of time spent trying to send a batch; ignored if enabled is false

An example configuration is provided as follows:

extensions:
  zpages:
    endpoint: localhost:55679
  text_encoding:
    encoding: utf8
    marshaling_separator: "\n"
    unmarshaling_separator: "\r?\n"

exporter:
  azureblob/1:
    url: "https://<your-account>.blob.core.windows.net/"
    container:
      logs: "logs"
      metrics: "metrics"
      traces: "traces"
    auth:
      type: "connection_string"
      connection_string: "DefaultEndpointsProtocol=https;AccountName=<your-acount>;AccountKey=<account-key>;EndpointSuffix=core.windows.net"
    encodings:
      logs: text_encoding
    append_blob:
      enabled: true
      separator: "\n"

When append_blob is enabled:

• The exporter will create append blobs instead of block blobs
• New data will be appended to existing blobs rather than creating new ones
• The configured separator will be inserted between data blocks
• If the blob doesn't exist, it will be created automatically