Engage - Vipps integration issues

Summary

On the afternoon of 20 March, we experienced a disruption affecting Vipps and Treazure integrations. The investigation showed that the issue was caused by a very large burst of webhook-triggering updates from one customer workflow. The issue was mitigated the same day by disabling the affected workflow, after which service returned to normal.

Customer Impact

Multiple customers experienced failed or interrupted external API call activities connected to Vipps flows, and some impact was also seen in Treazure flows. This resulted in timeouts and failed processing during the incident window.

Root Cause

The incident was caused by a massive simultaneous burst of webhook-triggering updates from one customer workflow, which overloaded the shared integration service. This created performance issues in the integration layer and affected Vipps and Treazure integrations using the same service.

Mitigation

After the source of the traffic spike was identified, the affected webhook endpoint was disabled and the customer integration with Treazure was temporarily deactivated. This reduced the load on the integration service and restored normal behaviour for other tenants.

Next Steps

To reduce the risk of similar incidents in the future, we are:

• Reviewing throttling and rate-limiting for webhook traffic
• Reviewing monitoring and alerting for the integration
• Strengthening guidance and communication around high-volume customer automations

We apologize for the inconvenience this incident may have caused and appreciate your understanding.