Unpacking Magento 2's Redis Cache Bloat: The cache:id_tags:* Accumulation Mystery

Unpacking Magento 2's Redis Cache Bloat: The cache:id_tags:* Accumulation Mystery

For large-scale Magento 2 stores, efficient caching is paramount for performance and stability. Many production environments rely on Redis (or its fork, Valkey) as a robust cache backend. However, a recently highlighted GitHub issue (Issue #41004) sheds light on a significant problem that can plague long-running Magento installations: the indefinite accumulation of cache:id_tags:* keys, leading to severe Redis memory bloat and performance degradation.

The Silent Killer: Unbounded cache:id_tags:* Growth

The core of the issue, reported on Magento Open Source 2.4.9 with Valkey (Redis) cache, is that while cache entries are correctly invalidated and removed, their associated reverse-index metadata – specifically keys prefixed with cache:id_tags:* – appear to persist indefinitely. This happens even after the corresponding cache objects are long gone.

In a typical Magento setup using Redis for default, full-page, and session caches, this bug manifests over months of production usage. Instead of a stable cache footprint, administrators observe a continuous, unchecked growth in Redis memory consumption. The issue author provided compelling evidence:

• After several months, a production DB0 (default cache) could accumulate approximately 2.6 million cache:id_tags:* keys out of 2.7 million total keys.
• This metadata alone could consume hundreds of megabytes (e.g., 370 MB for cache:tags:f73_BLOCK_HTML).
• Total Valkey memory usage soared to around 2.7 GB, largely due to this accumulated metadata.

Impact on Production Environments

The consequences of this accumulation are far-reaching for any serious e-commerce operation:

• Steadily Increasing Redis/Valkey Memory Usage: A direct and obvious impact, potentially leading to out-of-memory errors or requiring costly scaling of Redis instances.
• Slower Cache Operations: As the cache databases grow, operations like reading, writing, and especially invalidating cache entries become significantly slower.
• Degraded bin/magento cache:flush Performance: The command, crucial for maintenance, takes much longer to execute due to the massive number of keys it has to process or attempt to process.
• Large Metadata Structures: The sheer volume of metadata overshadows the actual active cache data, making the cache less efficient.

Unmasking the Root Cause: A Deep Dive into Cache Invalidation

The issue's author conducted a thorough investigation, including flushing the cache databases (valkey-cli -n 0 FLUSHDB and valkey-cli -n 1 FLUSHDB). This immediately dropped Valkey memory from 2.7 GB to a mere 30 MB, with Magento continuing to function normally and rebuilding its cache from a small baseline. This confirmed that the problem lies with the long-term metadata growth, not the active cache.

Upon examining the Magento source code, a potential discrepancy was identified in how cache entries are removed:

• The Symfony::remove() method, used for individual item removal, appears to correctly handle reverse indices via $this->adapter->onRemove($cleanId);.
• However, bulk invalidation paths, which often execute $this->adapter->deleteByIds($ids);, do not explicitly show the removal of corresponding cache:id_tags:* entries.

During a cache:flush operation, Redis MONITOR logs showed numerous SREM cache:all_ids ... commands, but no visible deletion of cache:id_tags:* keys, reinforcing the hypothesis that these specific reverse-index entries are being orphaned.

Suggested Investigation and Moving Forward

The issue author's suggested investigation points to a critical area for Magento developers:

Please verify that all bulk cache invalidation paths (especially `deleteByIds()`) correctly remove the associated `cache:id_tags:*` reverse-index entries (created by `RedisTagAdapter::onSave()`), as they appear to accumulate indefinitely during normal operation.

Particular attention may be warranted for the interaction between:

`Symfony::clean*()`
`RedisTagAdapter::deleteByIds()`
`RedisTagAdapter::onRemove()`

to ensure reverse-index cleanup occurs for all invalidation paths.

This detailed report is invaluable for the Magento community. While no immediate solution or workaround was provided within the issue comments (as of the provided information), the comprehensive analysis offers a clear roadmap for developers to diagnose and potentially fix this significant performance bottleneck. For merchants and system administrators, understanding this issue is crucial for monitoring their Redis instances and planning for potential cache maintenance strategies, such as periodic full cache flushes, until a permanent fix is implemented.

Addressing this bug is vital for ensuring the long-term stability and optimal performance of Magento 2 stores, especially those with high traffic and extensive product catalogs, reinforcing Magento's reputation as a robust e-commerce platform.