Problem
In crypto data streams, maintaining data consistency was a critical risk for a system dependent on external providers. Any interruption, latency, or inconsistency in data from sources like CoinGecko and CoinAPI could lead to publishing incorrect or incomplete information on the platform.
This was not just a technical issue—it represented a direct financial risk.
Root Cause
The issue was not the data providers themselves.
The system was not designed to tolerate external dependency failures
The data update process was not atomic
The cache layer had become a single point of failure
Data integrity was not prioritized over freshness
As a result, the system was prone to publishing incorrect data, especially under failure conditions.
What I Did (System)
I redesigned the data flow to improve system resilience:
Implemented an atomic transition model using A/B collection swap strategy
Introduced safeguards to prevent publishing empty data snapshots
Designed a fallback event-driven mechanism to eliminate Redis dependency as a single point of failure
Reframed system behavior to remain safe and consistent under failure scenarios
The goal was not to make the system faster, but to prevent it from producing incorrect data.
Outcome
The system became resilient to external dependency failures
Data consistency was preserved by prioritizing correctness over freshness
Single points of failure were eliminated
The system continued operating in a controlled manner instead of failing under error conditions
More importantly, the system evolved from one that simply “works” to one that remains correct even when it breaks.
Key Insight
In distributed systems, the core problem is rarely performance—it is correctness.
Most systems are designed to function under ideal conditions.
True resilience, however, is defined by:
how the system behaves under failure.
In data systems, the most critical decision is not when to update data,
but when to deliberately choose not to update it.