Difficulty in getting tracking event

Incident Report for Shipup

Postmortem

We wanted to take the time to provide a transparent summary of an incident that affected our system from January 6th to the 14th. During this period, we experienced difficulties in retrieving package events and sending associated notifications.

The root cause of the issue was an ongoing upgrade to our database, which inadvertently put excessive read/write operations on the disk. This slowdown significantly impacted critical operational processes, including event retrieval and notification scheduling.

Before starting the migration, we deliberately targeted a low-activity period to minimize disruptions to our customers' experience. The production migration started on December 29th, but at first, we didn't see any issues with the system. However, it wasn't until January 6th that the processing of large tables caused all database operations to slow down dramatically.

Our team worked diligently to optimize operations and restore normal system performance over the course of several days. We initially believed that notifications were being sent despite delays in tracker event processing, only to discover on Thursday 9th later that no notifications had been sent since Tuesday 7th morning. We quickly addressed this issue and restored notification functionality on Thursday evening.

The migration operation was completed on January 14th, and our system is now back to normal regarding event retrieval and notification sending.

Lessons learned:

We recognized that internal alerts didn't provide accurate information twice during the incident. We're working to resolve these issues and ensure no false-positive alarms in the future.
Although testing concluded before the migration, we acknowledge the importance of warning customers about ongoing changes, even if it's just a heads-up or a scheduled maintenance window. This would have allowed for more proactive communication and potentially minimized the impact.

Our goal is to provide our customers with a reliable and trustworthy experience. We take incidents like this seriously and are continually working to improve our processes and communication channels.

Posted Jan 17, 2025 - 10:12 CET

Resolved

The incident has ended on the 14/01 around 4pm.

After monitoring the situation, we confirm that this incident is now closed.

A Post Mortem will be shared here soon.

Posted Jan 15, 2025 - 15:41 CET

Monitoring

Huge improvement has been noticed. We are monitoring the situation closely and we will send an email today to share news about this issue.

Posted Jan 14, 2025 - 15:23 CET

Update

Huge improvement has been noticed. We are monitoring the situation closely and we will send an email today to share news about this issue.

Posted Jan 14, 2025 - 14:59 CET

Update

We are still experiencing difficulties in retrieving carrier events. The retrieval frequency is less regular and more spaced out.
In terms of impact :
- Shipup notifications are sent with a delay. They may not be sent at all if they fall within our obsolescence rules.
- Shipup data analysis is momentarily impacted.
- Once the problem is solved, we will recover all events. The data willl therefore be available for analysis shortly.

We expect to be back to normal state tomorrow evening.

Posted Jan 13, 2025 - 13:52 CET

Update

We are continuing to work on a fix for this issue.

The return to normal should occur by Tuesday, January 14th.

Posted Jan 10, 2025 - 10:42 CET

Update

We are continuing to work on a fix for this issue.

Posted Jan 09, 2025 - 17:57 CET

Update

- Shipup notifications are sent with a delay. They may not be sent at all if they fall within our obsolescence rules.
- No notifications were sent between January 7 and January 9. Notifications have been sent again since today, including, progressively, for orders placed in the last few days.
- Shipup data analysis is momentarily impacted.
- Once the problem is solved, we will recover all events. The data will therefore be available for analysis shortly.

IMPORTANT: your clients are still receiving the notifications sent by the carriers (even if you have the "Filter carrier notifications" on - we have disabled it until the resolution). Therefore, your clients receive the tracking status of their orders.

Posted Jan 09, 2025 - 15:40 CET

Update

We are experiencing difficulties in retrieving carrier events. The retrieval frequency is less regular and more spaced out.
In terms of impact :
- Shipup notifications are sent with a delay. They may not be sent at all if they fall within our obsolescence rules.
- Shipup data analysis is momentarily impacted.
- Once the problem is solved, we will recover all events. The data willl therefore be available for analysis shortly.

We expect to be back to normal state tomorrow evening.

Posted Jan 08, 2025 - 11:42 CET

Identified

We have been experiencing again some issues with retrieving tracker events related to our internal migration process. Some customers may receive their tracking notifications with a few hours' delay.

Posted Jan 07, 2025 - 16:07 CET

This incident affected: API (Retrieve Orders / Trackers from Shipup).