Skip to main content

Alert Rules

The CubeAPM Alert APIs allow you to programmatically manage Alert Rules, Receiver Groups (Notifications), and Mute Groups.

Authentication​

These APIs can be accessed programmatically using the Admin Port 3199.

Headers Required:

  • (Optional) Authorization: Bearer <ADMIN_TOKEN>
  • Content-Type: application/json
info

When http-token-admin is enabled in CubeAPM's config.properties, requests must include the corresponding token in the Authorization header when using curl to CREATE, UPDATE, or DELETE alert rules.

Alert rules define the condition (query) under which an alert fires.

Create Alert Rule​

Endpoint: POST http://<cubeapm-admin-host>:3199/api/alerts/api/v1/rules

Request Parameters​

ParameterTypeDescription
namestringA descriptive string identifier for your alert rule.
datasourcestringThe source platform to query against. Options are:
"prometheus" for metrics
"vlogs" for logs
"traces" for traces
kindstring"static" for a standard threshold alert.
statusstring"ACTIVE" to enable the alert, or "PAUSED" to temporarily disable it.
intervalintegerHow often (in seconds) the backend evaluates the query (e.g., 60).
exprstringThe query used to evaluate the alert condition.
expr2string(Optional) Secondary query expression used for advanced comparative evaluations or baseline conditions.
forintegerThe duration (in seconds) that the condition must be met before firing the alert.
repeat_intervalintegerOnce an alert is firing, how often to resend the notification (in seconds).
grouping_disablebooleanDefaults to false. Set to true to disable notification grouping (which normally batches multiple instances of the same alert).
labelsobjectKey-value pairs:
Notification Context: Custom tags (e.g., "severity": "critical") are included in notification payloads (Slack, Email) for extra context.

Alert Grouping: Setting the "group" key (e.g., "group": "Database Alerts") dictates which group the alert appears under on the CubeAPM UI.
annotationsobjectKey-value pairs for descriptive metadata (e.g., {"summary": "CPU is high"}).
config.receiver_group_idsarrayArray of IDs linking this alert to predefined Receiver Groups.
config.mute_group_idsarrayArray of IDs linking this alert to predefined Mute Groups.
config.modelobject(Optional) JSON used to save the visual state of the rule in the CubeAPM UI's Query Builder. If omitted, the API will still create the alert successfully, but it will be displayed in the UI as a raw query.
receiverobjectDefines an inline custom receiver (e.g., {"slack_configs": [...]}). Functions identically to a Receiver Group but exclusively for this rule.
muteobjectDefines an inline custom mute configuration (e.g., {"time_intervals": [...]}). Functions identically to a Mute Group but exclusively for this rule.
permissionsarrayDetermines access. Pass an empty array [] for default "Role Based" access (based on global CubeAPM roles). Pass an array of objects to grant "Custom" roles (viewer, editor) to specific users or teams.
info

You can configure inline receivers, mutes and permissions for alert rules.

Note: By placing these inline objects directly inside the root JSON payload of your POST or PUT request, the backend automatically attaches them exclusively to the single alert rule you are creating or updating.

Curl Example​

Base Example (Metrics):

curl -X POST "http://<cubeapm-admin-host>:3199/api/alerts/api/v1/rules" \
     -H "Content-Type: application/json" \
     -d '{
           "name": "High CPU Usage",
           "datasource": "prometheus",
           "kind": "static",
           "status": "ACTIVE",
           "grouping_disable": false,
           "interval": 60,
           "expr": "sum(rate(node_cpu_seconds_total{mode!=\"idle\"}[5m])) > 80",
           "for": 300,
           "repeat_interval": 3600,
           "labels": {
             "severity": "critical",
             "team": "database-reliability",
             "group": "Default Group"
           },
           "annotations": {
             "summary": "CPU usage exceeded 80%"
           },
           "config": {
             "receiver_group_ids": [],
             "mute_group_ids": []
           },
           "receiver": {
             "slack_configs": [
               { "channel": "high-cpu-specific-alerts" }
             ]
           },
           "mute": {
             "time_intervals": []
           },
           "permissions": []
         }'

For Logs & Traces: Use the exact same payload as above, but update the datasource and expr fields:

{
  "...": "...",
  "datasource": "vlogs",
  "expr": "{app=\"nginx\"} AND error | stats count() > 50"
}

Response Parameter​

The response is a JSON object with the following structure:

FieldTypeDescription
idintegerUnique identifier for the alert rule.
datasourcestringThe source platform queried (prometheus, vlogs, traces).
namestringThe name of the alert rule.
intervalintegerEvaluation interval in seconds.
exprstringThe query expression evaluated.
expr2stringAn optional secondary query or expression, typically used for baseline evaluations, comparisons, or anomaly detection.
kindstringRule type (static).
forintegerDuration condition must be met before firing.
labelsobjectKey-value pairs:
Notification Context: Custom tags (e.g., "severity": "critical") are included in notification payloads (Slack, Email) for extra context.

Alert Grouping: Setting the "group" key (e.g., "group": "Database Alerts") dictates which folder the alert appears under on the CubeAPM UI.
statusstringCurrent status (ACTIVE, PAUSED, etc.).
grouping_disablebooleanWhether alert grouping is disabled.
configobjectContains receiver_group_ids and mute_group_ids.
receiverobjectContains inline custom receiver definitions configured exclusively for this rule.
muteobjectContains inline custom mute definitions configured exclusively for this rule.
repeat_intervalintegerNotification repeat interval in seconds.
permissionsarrayDetermines access. Pass an empty array [] for default "Role Based" access (based on global CubeAPM roles). Pass an array of objects to grant "Custom" roles (viewer, editor) to specific users or teams.

Note: If you provide IDs in config.receiver_group_ids or config.mute_group_ids, those groups must exist in your account. Passing an invalid or non-existent ID will result in a 500 error (all receiver groups must exist). If you have not created any Receiver Groups yet, pass an empty array [] as shown in the examples.

For example:

{
  "id": 1,
  "datasource": "prometheus",
  "name": "High CPU Usage",
  "interval": 60,
  "expr": "sum(rate(node_cpu_seconds_total{mode!=\"idle\"}[5m])) > 80",
  "expr2": "",
  "kind": "static",
  "for": 300,
  "labels": {
    "group": "Default Group",
    "severity": "critical",
    "team": "database-reliability"
  },
  "annotations": {
    "summary": "CPU usage exceeded 80%"
  },
  "status": "ACTIVE",
  "grouping_disable": false,
  "config": {
    "receiver_group_ids": [],
    "mute_group_ids": []
  },
  "receiver": {
    "slack_configs": [
      {
        "channel": "high-cpu-specific-alerts"
      }
    ]
  },
  "mute": {
    "time_intervals": [
      {
        "times": [
          {
  "start_time": "00:00",
  "duration_minutes": 1440
          }
        ],
        "weekdays": [
          "saturday",
          "sunday"
        ],
        "location": "UTC"
      }
    ]
  },
  "repeat_interval": 3600,
  "permissions": [
    {
      "entity_type": "user",
      "entity_id": "john.doe@company.com",
      "permission": "editor"
    },
    {
      "entity_type": "team",
      "entity_id": "4",
      "permission": "viewer"
    }
  ]
}

Get Alert Rules​

Endpoint: GET http://<cubeapm-admin-host>:3199/api/alerts/api/v1/rules

Curl Examples & Response Format​

The alert rule JSON object has the following structure:

FieldTypeDescription
alertsarrayA list of specific alert instances that are currently firing or pending. (Note: Returns null if the rule is currently inactive).
annotationsobjectCurrent annotations evaluated for the alert rule.
configobjectRule configuration, including receiver and mute group mappings.
datasourcestringThe source platform queried (prometheus, vlogs, traces).
durationintegerThe duration condition the rule must meet before firing. Note: The GET API returns this as "duration", but it corresponds exactly to the "for" parameter used when creating the alert.
groupingDisablebooleanWhether alert grouping is disabled.
healthstringThe health status of the rule evaluation (e.g., ok).
idintegerUnique identifier for the alert rule.
intervalintegerEvaluation interval in seconds.
kindstringRule type (static).
labelsobjectKey-value pairs:
Notification Context: Evaluated tags (e.g., "severity": "critical") are included in notification payloads for extra context.

Alert Grouping: Setting the "group" key (e.g., "group": "Database Alerts") dictates which folder the alert appears under on the CubeAPM UI.
lastEvaluationstringTimestamp of the last time the rule was evaluated.
muteobjectContains inline custom mute definitions configured exclusively for this rule.
namestringThe name of the alert rule.
permissionstringThe access level of the requesting user.
querystringThe query expression evaluated.
query2stringAn optional secondary query or expression, typically used for baseline evaluations, comparisons, or anomaly detection.
readonlybooleanWhether the rule is read-only.
receiverobjectContains inline custom receiver definitions configured exclusively for this rule.
repeatIntervalintegerNotification repeat interval in seconds.
statestringThe current state of the alert (firing, pending, inactive).
statusstringCurrent rule status (ACTIVE, PAUSED).

For example:

Request:

curl -X GET "http://<cubeapm-admin-host>:3199/api/alerts/api/v1/rules?id=1"

Response:

The response format is a JSON array of alert rule objects.

{
  "alerts": null,
  "annotations": {
    "alertname": "Testing-Alert",
    "cube.kind": "static",
    "datasource": "prometheus",
    "interval": "30",
    "value": "{{ printf \"%.2f\" .Value }}"
  },
  "config": {
    "model": {
      "type": "quick",
      "calculate": "latency_average",
      "value": "90",
      "labelPairs": [
        {
          "label": "service",
          "operator": "=",
          "values": [
            "analytics-service"
          ],
          "options": []
        }
      ],
      "groupBy": []
    }
  },
  "datasource": "prometheus",
  "duration": 0,
  "groupingDisable": false,
  "health": "ok",
  "id": 2,
  "interval": 30,
  "kind": "static",
  "labels": {
    "group": "Default Group",
    "rule_id": "{{ .GroupID }}"
  },
  "lastEvaluation": "0001-01-01T00:00:00Z",
  "mute": {
    "time_intervals": []
  },
  "name": "Testing-Alert",
  "permission": "admin",
  "query": "(sum(increase(cube_apm_latency_total{service=\"analytics-service\", span_kind=~\"server|consumer\"} default 0)) * 1000 / sum(increase(cube_apm_calls_total{service=\"analytics-service\", span_kind=~\"server|consumer\"} default 0))) > 20",
  "query2": "",
  "readonly": false,
  "receiver": {
    "email_configs": [],
    "slack_configs": [
      {
        "channel": "C0AUECYSYSJ",
        "type": "slack",
        "send_resolved": true,
        "cube_show_query": false,
        "cube_show_sample_log": false,
        "valid": true
      }
    ],
    "pagerduty_configs": [],
    "jira_configs": [],
    "opsgenie_configs": [],
    "googlechat_configs": [],
    "msteamsv2_configs": [],
    "webhook_configs": []
  },
  "repeatInterval": 300,
  "state": "inactive",
  "status": "PAUSED"
}

Update / Delete Alert Rules​

Update an Alert Rule (PUT)​

Endpoint: PUT http://<cubeapm-admin-host>:3199/api/alerts/api/v1/rules

Send the same payload as POST, but include the "id" field in the JSON payload.

warning

The PUT request replaces the entire rule configuration. You must include all existing configurations (such as labels, inline receiver, inline mute, and permissions) in your payload; otherwise, they will be overwritten and removed!

How to pause an alert: To pause an alert, change the "status" field to "PAUSED" and send the full payload in the PUT request. Do not send just the "status" field, or your entire rule configuration will be wiped out!

For example:

curl -X PUT "http://<cubeapm-admin-host>:3199/api/alerts/api/v1/rules" \
     -H "Content-Type: application/json" \
     -d '{
           "id": 1,
           "name": "High CPU Usage",
           "datasource": "prometheus",
           "kind": "static",
           "status": "PAUSED",
           "interval": 60,
           "expr": "sum(rate(node_cpu_seconds_total{mode!=\"idle\"}[5m])) > 80",
           "for": 300,
           "repeat_interval": 3600,
           "labels": {
             "severity": "critical",
             "team": "database-reliability",
             "group": "Default Group"
           },
           "annotations": {
             "summary": "CPU usage exceeded 80%"
           },
           "config": {
             "receiver_group_ids": [],
             "mute_group_ids": []
           },
           "receiver": {
             "slack_configs": [
               { "channel": "high-cpu-specific-alerts" }
             ]
           },
           "mute": {
             "time_intervals": []
           },
           "permissions": []
         }'

Include every other field from your rule's current configuration (such as expr2, grouping_disable, annotations, and any inline receiver/mute/permissions you had set) — the example above is abridged for brevity.

Delete an Alert Rule (DELETE)​

Endpoint: DELETE http://<cubeapm-admin-host>:3199/api/alerts/api/v1/rules?id={id}

Deletes an alert rule permanently.

For example:

curl -X DELETE "http://<cubeapm-admin-host>:3199/api/alerts/api/v1/rules?id=1"

How to Configure Inline Receivers​

While you can assign alerts to shared global Receiver Groups using config.receiver_group_ids, you can also configure Inline Receivers directly on a specific Alert Rule. This is useful when you want the alert to have a specific routing that isn't shared by any other rules.

To define a receiver inline, you define the exact same configuration schema you would use for a Receiver Group, but place it inside the "receiver" object of the Alert Rule payload.

Integration Parameters:

ParameterTypeDescription
send_resolvedbooleanWhether or not to notify when the alert is resolved (e.g. returns to a healthy state). Default is true.
cube_show_querybooleanIf true, includes the raw PromQL/Metrics query that triggered the alert inside the notification payload. Default is false.
cube_show_sample_logbooleanIf true, includes a sample log line from the evaluation (if applicable for Log-based alerts). Default is false.
{
  "...": "...",
  "receiver": {
    "slack_configs": [
      { 
        "channel": "production-alerts", 
        "send_resolved":true,
        "cube_show_query":false,
        "cube_show_sample_log":false
      }
    ]
  }
}

How to Configure Inline Mutes​

While you can link alerts to shared global Mute Groups using config.mute_group_ids, you can also configure Inline Mutes directly on a specific Alert Rule. Place the exact same configuration schema you would use for a Mute Group inside the "mute" object of the Alert Rule payload.

{
  "...": "...",
  "mute": {
    "time_intervals": [
      {
        "times": [
          { "start_time": "00:00", "duration_minutes": 1440 }
        ],
        "weekdays": ["saturday", "sunday"],
        "location": "America/New_York"
      }
    ]
  }
}

How to Configure Role Based and Custom Permissions​

The permissions array in the JSON payload dictates who can view and edit your alert rule.

For global roles, teams, and how permissions work in the UI, see Roles and Permissions and Teams.

1. Role Based (Default)​

When "Role Based" is used, the rule relies entirely on the user's global workspace role (Global Viewers can view it, Global Editors can edit it).

To enforce this via the API, simply pass an empty array:

For example:

{
  "...": "...",
  "permissions": [] 
}

2. Custom​

When "Custom" is used, you explicitly grant specific roles (viewer or editor) to specific users (by email) or teams (by ID) for this rule exclusively.

To configure this via the API, populate the permissions array with one or more objects specifying the entity_type, entity_id, and permission:

For example:

{
  "...": "...",
  "permissions": [
    {
      "entity_type": "user",
      "entity_id": "john.doe@company.com",
      "permission": "editor"
    },
    {
      "entity_type": "team",
      "entity_id": "4", 
      "permission": "viewer"
    }
  ]
}
info

Custom permissions form an allowlist in the CubeAPM UI. Only listed users and teams can access the rule there. Effective permission is capped at the user's global role — a global viewer cannot edit even if granted editor on the resource.

info

Admin Port APIs (port 3199) bypass per-resource ACL. They can list, get, update, and delete all alert rules regardless of Custom permissions.