Skip to main content

AWS Elastic Container Service (ECS)

Telemetry data can be collected from AWS ECS (both EC2 and Fargate) using OpenTelemetry (OTel) Collector.

OTel Collector needs to be deployed as a daemon service.

  1. Create an IAM role with the following policy and name it ECSOTELDaemonRole.

    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Sid": "VisualEditor0",
          "Effect": "Allow",
          "Action": ["ec2:DescribeInstances", "ecs:DescribeContainerInstances"],
          "Resource": "*"
        }
      ]
    }

  2. Add the following configuration to your EC2 launch template.

    launch-template
    # Create OTel config directory
    mkdir -p /etc/ecs/otel-config

    # Write config.yaml
    cat > /etc/ecs/otel-config/config.yaml << 'EOF'
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318

      hostmetrics:
        collection_interval: 60s
        scrapers:
          cpu:
          disk:
          load:
          filesystem:
          memory:
          network:

    processors:
      batch: {}
      resourcedetection:
        detectors: [ecs]
        timeout: 2s

    exporters:
      debug:
        verbosity: detailed
        sampling_initial: 5
        sampling_thereafter: 1

      otlphttp/metrics:
        metrics_endpoint: http://<cubeapm_endpoint>:3130/api/metrics/v1/save/otlp
        retry_on_failure:
          enabled: false
      otlphttp/logs:
        logs_endpoint: http://<cubeapm_endpoint>:3130/api/logs/insert/opentelemetry/v1/logs
        headers:
          Cube-Stream-Fields: severity, host.name
      otlp/traces:
        endpoint: <cubeapm_endpoint>:4317
        tls:
          insecure: true

    service:
      pipelines:
        traces:
          exporters:
            # - debug
            - otlp/traces
          processors:
            - batch
            - resourcedetection
          receivers:
            - otlp
        metrics:
          exporters:
            # - debug
            - otlphttp/metrics
          processors:
            - batch
            - resourcedetection
          receivers:
            - otlp
            - hostmetrics
        logs:
          exporters:
            # - debug
            - otlphttp/logs
          processors:
            - batch
            - resourcedetection
          receivers:
            - otlp
    EOF

    # Set permissions for OTel user
    chown -R 65532:65532 /etc/ecs/otel-config

  3. Create task definition for OTel Collector. Use the following configuration and edit the values in < > according to your setup.

    otel-collector-daemonset.json
    {
      "family": "otel-collector-daemon",
      "containerDefinitions": [
        {
          "name": "otel-collector",
          "image": "otel/opentelemetry-collector-contrib:0.145.0",
          "cpu": 0,
          "portMappings": [
            {
              "containerPort": 4317,
              "hostPort": 4317,
              "protocol": "tcp"
            },
            {
              "containerPort": 4318,
              "hostPort": 4318,
              "protocol": "tcp"
            }
          ],
          "essential": true,
          "command": [
            "--config=/etc/otelcol-contrib/config.yaml"
          ],
          "environment": [],
          "mountPoints": [
            {
              "sourceVolume": "otel-config",
              "containerPath": "/etc/otelcol-contrib/",
              "readOnly": true
            },
            {
              "sourceVolume": "host-root",
              "containerPath": "/hostfs",
              "readOnly": true
            }
          ],
          "volumesFrom": [],
          "logConfiguration": {
            "logDriver": "awslogs",
            "options": {
              "awslogs-group": "/ecs/otel-daemon",
              "awslogs-region": "<aws-region>",
              "awslogs-stream-prefix": "daemon"
            }
          },
          "systemControls": []
        }
      ],
      "taskRoleArn": "ECSOTELDaemonRole",
      "executionRoleArn": "<ecsTaskExecutionRole>",
      "networkMode": "host",
      "volumes": [
        {
          "name": "otel-config",
          "host": {
            "sourcePath": "/etc/ecs/otel-config"
          }
        },
        {
          "name": "host-root",
          "host": {
            "sourcePath": "/"
          }
        }
      ],
      "placementConstraints": [],
      "requiresCompatibilities": [
        "EC2"
      ],
      "cpu": "512",
      "memory": "1024",
      "runtimePlatform": {
        "cpuArchitecture": "<X86_64 or ARM64>",
        "operatingSystemFamily": "LINUX"
      }
    }

  4. Create a service using the above task definition and configure it as a daemon service by setting Scheduling strategy as daemon in Deployment configuration.

  5. Update your applications' configuration to send logs, metrics and traces to the OTel Collector daemon service. Applications can connect to OTel Collector on the fixed IP 172.17.0.1 (Docker bridge gateway IP), which will always point to the OTel Collector running on the same EC2 host as the application container.

Troubleshooting​

  1. Verify that OTel Collector is running on the EC2 instances, and Otel Collector config.yaml file is placed at /etc/ecs/otel-config/config.yaml on the EC2 instances.

  2. Check OTel Collector logs.