Bug Fix

Fixed Inconsistent Notification Delivery Issues

Resolved delays and failures in notification delivery to ensure timely and reliable updates.

January 4, 2025

What Was Wrong

The root cause was identified in the notification processing pipeline:

  1. The message queue handler in notify/service/queue_manager.py occasionally dropped tasks when buffer limits were exceeded during high traffic periods.
  2. An outdated dependency in lib/msg_dispatch_22.so on Linux systems caused failures in parsing user notification preferences, leading to skipped or misrouted notifications.
  3. A race condition in core/api/notify_dispatcher.js resulted in duplicate notifications for some users and none for others.

Impact

  • Windows: Notifications were delayed by up to 10 minutes, particularly under heavy loads.
  • Linux: Certain user-specific notifications failed to send, with error logs showing MSG_PARSE_FAIL (code 412).
  • macOS: Notifications were intermittently delayed or failed for users on Ventura 13.0 and higher.

Resolution

  • Updated queue_manager.py to dynamically scale buffer limits based on traffic patterns.
  • Recompiled lib/msg_dispatch_22.so with updated dependencies to ensure compatibility.
  • Patched notify_dispatcher.js to prevent race conditions by introducing a locking mechanism.

Next Steps

  • Enhance traffic monitoring for the notification service.
  • Add end-to-end tests for the notification pipeline to prevent similar issues in the future.
  • Conduct load tests in simulated high-traffic environments to verify the fixes.

Notifications are now delivered promptly and consistently across all platforms. If you experience further issues, please contact Swyft Support with relevant logs.