DEFER Command Overview
Purpose
The DEFER command is a critical component of EnSync's event processing system, providing the ability to postpone event processing for a specified time period. This capability is essential for implementing resilient distributed systems that can gracefully handle temporary failures and schedule future work.
Key Capabilities
Retry Logic
When a service encounters a temporary failure (such as a downstream service being unavailable), it can defer the event for a short period rather than failing completely. This implements the retry pattern, allowing the system to automatically attempt processing again after the specified delay.
Backoff Strategies
By using progressively longer delays for subsequent deferrals, services can implement exponential backoff strategies to prevent overwhelming recovering systems and to handle transient failures efficiently.
Scheduled Processing
Events can be scheduled for future processing at specific times, enabling time-based workflows and delayed actions without requiring a separate scheduling system.
Load Balancing
During high-load periods, non-critical events can be deferred to reduce system pressure, effectively implementing a form of load shedding that preserves all events while prioritizing system stability.
Technical Implementation
The DEFER command works by:
Moving events from the client's processing queue to a deferred queue
Storing the event with a score representing its future delivery time
Using a background process to periodically check for events that are ready for delivery
Automatically returning ready events to the pending queue with their specified priority preserved (defaults to priority 1 if not specified)
This implementation ensures that deferred events are never lost and will be processed exactly once when their delay period expires.
Integration Points
The DEFER command integrates with:
Priority System: Allows specifying priority levels (1-5, where 5 is highest) for deferred events and preserves these levels throughout the deferral period
Competing Consumer Pattern: Works with multiple instances of the same service
Monitoring System: Provides visibility into deferred events and their scheduled delivery times
Audit System: Records all deferrals with their reasons for compliance and debugging
Use Cases
Implementing retry logic for transient failures
Scheduling notifications or reminders for future delivery
Managing rate limits for external API calls
Implementing time-based workflow steps
Throttling event processing during system maintenance
Coordinating distributed processes with time dependencies