In Ceph clusters, managing monitor weights correctly is crucial to maintaining system stability. A common issue we’ve encountered, particularly in clusters deployed before 2019, involves manager failures caused by inconsistent mon_weight values. If newer monitors are added to an older cluster with a default weight of 0, while existing monitors have a weight of 10, the cluster can experience crashes, leading to serious operational disruptions.

This blog post will dive into how to detect and resolve this issue, referencing Bug #64814, which outlines the root cause of the manager failures. We’ll also explore the specific cases we’ve seen at 42on, particularly in Red Hat Ceph to Community Ceph conversions. If you’re managing a Ceph cluster that was provisioned before 2019, this is essential information that could prevent unexpected crashes and keep your environment running smoothly.

Keep reading to learn more about detecting and fixing these monitor weight discrepancies before they cause serious problems.

Checking mon weight

Prior to 2019 mon_weight defaulted to 10, the new default is 0. In clusters initially provisioned before 2019, if monitors are added on a newer version with mon_weight 0 managers will fail to start and crash. The bug is documented in detail at https://tracker.ceph.com/issues/64814

The bug can be detected and resolved by changing mon_weight to 0 with the following commands:

ceph mon dump -f json|jq '.mons[]|(.name), (.weight)'
#for each mon with a weight > 0
ceph mon set-weight $MON_NAME 0

Explanation

At least 4 instances among customers of this issue have been observed within 42on’s customer base. In our case these were specific to environments that had transitioned from Red Hat Ceph to Community Ceph. While this may occur in any cluster deployed before 2019 we suspect that some part of modern upgrade processes handles this due to us only observing this in Red Hat conversions.

The problem is not the actual weight value, rather when the weight value is not uniform, meaning there is no difference between all mons on 10 and all mons on 0, but if there is a variation within these weights the mgr and other services are unable to handle it properly.

Conclusion

Inconsistent monitor weights in Ceph clusters, particularly those provisioned before 2019, can lead to serious disruptions, including manager failures and system crashes. As highlighted in Bug #64814, the root cause stems from a mismatch in mon_weight values between older and newer monitors. This issue is especially prevalent in environments that have transitioned from Red Hat Ceph to Community Ceph, though it can potentially affect any older cluster.

Fortunately, the solution is straightforward: ensuring uniformity in mon_weight across all monitors in the cluster, typically by setting the value to 0. By addressing this inconsistency, you can prevent the crashes that result from these discrepancies and maintain the smooth operation of your Ceph environment.

For anyone managing a pre-2019 Ceph cluster, particularly those undergoing version transitions, we recommend proactively checking and adjusting the mon_weight settings to avoid unexpected failures. By doing so, you can safeguard your system’s stability and avoid costly downtime.

This blog was created in collaboration with Spencer Macphee.