Context
The cryptocurrency market operates 24/7 with near-zero tolerance for errors. In such an environment, the accuracy of symbol data becomes one of the most critical components of the system.
SymbolService, developed within the HemenBasvur platform, was aggregating data from external providers such as CoinGecko and CoinAPI. This introduced a fundamental challenge:
How can we guarantee data consistency without causing service disruption?
Problem
- Outages or delays from external data providers could disrupt data flow
- Partial or incorrect data could be published to the system
- Incorrect data could lead to financial losses
- The system was not designed to behave safely under failure conditions
Constraints
- The system depended on external APIs (CoinGecko, CoinAPI)
- Data flow had to remain uninterrupted
- Publishing incorrect data was worse than publishing no data
- Redis was a potential single point of failure
Root Cause
- Data update operations were not atomic
- The system was not tolerant to external dependency failures
- The cache layer (Redis) acted as a single point of failure
- Freshness was prioritized over data integrity
- There was no safeguard for empty data scenarios
Key Design Decisions
A/B Collection Swap for Atomic Data Transition
New data was prepared in a separate collection instead of the active one
A single-step switch activated the new dataset
Eliminated dirty reads and partial data risks
Empty Snapshot Guard
If external sources returned empty data, the system skipped the update
Prevented accidental deletion of the entire dataset
Dual Event (Fallback) Strategy
If Redis was operational → a reference event was published
If Redis failed → full payload was sent
Prevented data flow interruption due to cache dependency
Startup Health Check
Validated active data integrity at system startup
Trade-offs
- Data freshness was intentionally delayed in certain scenarios
- Accuracy was prioritized over performance
- Increased latency was accepted during Redis fallback situations
Failure Scenarios
- If external APIs return empty data → no update is performed
- If Redis fails → fallback event mechanism is triggered
- System operates in a fail-safe rather than fail-fast mode
- Instead of propagating incorrect data, the last known correct data is preserved
Results
- Eliminated dirty reads and partial data risks
- Continued operation during external dependency failures
- Maintained data flow even during Redis outages
- Minimized the risk of publishing incorrect data
- System behavior became controlled and predictable under failure conditions
Key Insight
In distributed systems, the real problem is not performance — it is correctness.
Systems are often designed to function when everything works. True resilience, however, is defined by how a system behaves when things break.
The most critical decision in this system was not when to update the data,
but choosing not to update it when necessary.