Fixing GlusterFS split-brain automagically

My filesystem is in split-brain!

Hi everyone! When using GlusterFS (nutellino.it uses it too) it can happen that, because of network partitioning or some nodes crashing, some of the files in the glusterfs volumes are no longer accessible. This happens when one or more of the replica files start to diverge from each other. GlusterFS is then no longer able to resolve it automatically and start a self heal process.

The best-known process is probably the manual one, described in the official guide here: https://github.com/gluster/glusterfs/blob/master/doc/debugging/split-brain.md

It gets really long and tedious though, especially if the only file going into split brain is a log file.

Luckily, since version 3.8 they've introduced a configuration parameter among the volume options:

cluster.favorite-child-policy

This option can take the following values:

• none: Default value, in this case you have to proceed manually.
• ctime: Selects and heals the volume using the file with the highest ctime.
• mtime: Selects and heals the volume using the file with the highest mtime.
• size: Selects and heals the volume using the file with the largest size.
• majority: Selects the file with identical mtime and size on more than half of the bricks in the replica (for volumes with 3 or more replicas).

How do I check the state of my volume?

With the following command:

gluster volume heal miovolume info split-brain

How do I change the cluster.favorite-child-policy parameter?

By changing the volume values:

gluster volume set miovolume cluster.favorite-child-policy mtime

And then running a heal operation on the volume:

gluster volume heal miovolume

Once the split brain is resolved, you can set the value back to none, or leave it automatic if the files in the volume aren't critical.