/
Pulse IBM Cloud Token Authorization Failures — Diagnosis and Fix

Pulse IBM Cloud Token Authorization Failures — Diagnosis and Fix

This page documents a recurring incident pattern where Pulse intermittently loses connectivity to IBM Planning Analytics (TM1) instances hosted on IBM Cloud. It explains the symptoms, root causes, how to diagnose the issue, and remediation/prevention steps—without referencing any specific customer, ticket, or environment names.

Summary

During certain time windows (often aligning with upstream maintenance periods), Pulse may intermittently lose connections to one or more IBM Planning Analytics (TM1) instances behind the IBM Cloud gateway. During these windows, Pulse can receive:

  • HTTP 502 responses when calling TM1 APIs (backend temporarily unavailable behind the gateway)

  • Repeated HTTP 401 responses after successfully retrieving a new access token (token not being accepted/authorized for subsequent API calls)

If Pulse exhausts its retry/authorization limits, one or more environments may require manual re-authentication in the Pulse UI once the upstream services recover.

Environment and Scope (Generic)

  • Pulse component: Workspace/Agent connections to IBM Planning Analytics on IBM Cloud

  • Affected scope: One or more Pulse environments pointing to IBM Cloud-hosted TM1 instances

  • Typical pattern: Intermittent failures during a recurring weekly maintenance window (varies by region/IBM schedule)

Symptoms Observed

  • Intermittent TM1 connectivity failures from Pulse with HTTP 502 returned by the IBM Cloud gateway when querying instance state or calling TM1 APIs.

  • Pulse retrieves a new access token, but subsequent API requests return 401 Unauthorized repeatedly.

  • Pulse UI shows instances disconnected until a manual “Login” (re-authentication) is performed on the Environment page after the window.

  • In some cases, Pulse logs show an agent identity/mapping problem (e.g., “Unable to find a matching Agent ID”), requiring environment re-initialization.

Evidence from Logs (Examples)

Agent identity mismatch example

ERROR ServerWorkspace Unable to find a matching Agent ID

Token retrieved but gateway returns 401 repeatedly

INFO ClientWorkspace New access token retrieved for https://<ibm-cloud-gateway-host> ERROR ClientWorkspace MaxUnauthorizedAttemptsReached: https://<ibm-cloud-gateway-host>/api/.../api/v1/Threads... ERROR ClientWorkspace Credentials to be set to invalid for <environment> (<auth-method>)

Gateway/backend 502 example (typical symptom)

ERROR ClientWorkspace Request failed with status 502 (Bad Gateway) calling https://<ibm-cloud-gateway-host>/api/...

What the Error Was

1) Gateway 502 responses

Pulse can reach the IBM Cloud gateway URL, but downstream TM1 instance calls intermittently return HTTP 502. This indicates backend service unavailability or a maintenance window behind the gateway, rather than a Pulse-side configuration issue.

2) Authorization 401 loop (post-token retrieval)

In some cases Pulse successfully obtains a new access token, but subsequent API calls return 401 Unauthorized. Pulse retries token validation/authorization up to the configured limits; once exhausted, Pulse marks the stored credentials invalid to prevent infinite loops. Manual re-authentication is then required.

Why It Happened (Most Common Causes)

  • IBM Cloud maintenance or transient instability affecting the TM1 backend behind the gateway (commonly surfaces as 502).

  • Gateway-side authorization inconsistencies where a freshly retrieved token is not accepted for API calls (surfaces as repeated 401s).

  • Agent/environment identity mismatch after changes, restarts, or prolonged connectivity issues (surfaces as “Unable to find a matching Agent ID”).

Note: Log lines such as “Day of the Week set to …” are informational and not an error condition.

How It Was Fixed

Immediate remediation

  • Once IBM Cloud services stabilize, perform a manual “Login” (re-authentication) in Pulse → Environments for any affected environments whose credentials were invalidated.

  • If you see Agent ID mismatch errors:

    • Re-initialize/re-map the environment via the Pulse Environment page (or follow your standard Agent re-registration procedure).

Stabilization / prevention

  • Schedule a Pulse service restart shortly after the known maintenance window to ensure clean session re-initialization and reduce the chance of stale connections/credentials persisting after upstream recovery.

  • If repeated 401 after token retrieval persists:

    • Treat it as a likely IBM gateway authorization issue and engage IBM support with timestamps and log excerpts.

Configuration Note: MaxTokenRefreshAttempt

Definition: Maximum number of times Pulse retries to refresh and authorize an IBM Cloud access token before marking the stored credentials invalid and requiring manual re-authentication in the Pulse Environments page.

  • Default: 5

  • Purpose: Prevent infinite retry loops during gateway-side authorization failures.

  • Guidance: Increasing this value will not fix gateway authorization issues; it only increases how long Pulse retries before invalidating credentials if 401s continue.

Operational Runbook

When reconnect issues occur

Look for:

  • HTTP 502 responses from the gateway to TM1 backend calls

  • Repeated HTTP 401 after “New access token retrieved …”

  • “Unable to find a matching Agent ID”

Actions:

  1. Verify IBM Cloud / TM1 service health outside Pulse (to separate upstream outage from Pulse issues).

  2. If credentials are invalidated, go to Pulse → Environments → Login to re-authenticate.

  3. If Agent ID mismatch appears, re-initialize the environment mapping.

  4. If the pattern aligns with a maintenance window, consider a scheduled Pulse restart after the window.