Steffan

Question:

I want to make a glusterFS with replication on 2 hosts for my webserver cluster to share.

If i on webserver1 and webserver2 mount the GlusterFS using the command you specified above:

[HOST]:/[VOLUME] /[MOUNT] glusterfs defaults,_netdev 0 0

WHAT IF, [HOST] goes down ? how does it know to connect to the other host instead?? so i have high availability? is this even possible?

james.coyle

Hi Steffan.

I have updated the article as you have a good point – for this type of replication, a single server address is a critical single point of failure. Please see the revised article on how to do what you require.

Regards

James.

Steffan

THANKS!
I really like your guide, you seem to know what you are doing!.
However i do miss some more details and explanations on what the different config variables are there for, and what they do..

The stuff i’m curious about are marked in bold below:

/etc/glusterfs/datastore.vol [MOUNT] glusterfs rw,allow_other,default_permissions,max_read=131072 0 0
“max_read=…” in the above, why that exact number? what happens if i turn it up? do i get more performance or what does it do?

volume writebehind
type performance/write-behind
option window-size 1MB
subvolumes replicate
end-volume

volume cache
type performance/io-cache
option cache-size 512MB
subvolumes writebehind
end-volume
The above is from your /etc/glusterfs/datastore.vol example, i curious about the two lines i marked with bold…
Windows-size: What is this, do i get more performance with a bigger window?
cache-size: Where does the cache get saved? on disk, or in memory(RAM)?

I hope you can answer my questions, i really like your blog. Is there any way i can subscribe so i get an email when you post something new?

james.coyle

Hi Stephen,

I am working on a couple of performance related posts now, but they won’t be ready for publishing for a few more days.

To answer your above questions…

Firstly, it’s important to note that there is no hard rule when it comes to performance attributes. On different systems, and under different workloads you will notice different results and therefore you must tune each attribute to your environment.

max-read – This is taken from NFS and is often used when mounting remote filesystems. It’s the maximum amount of data which will be read in one operation. There is also a max_write which can help you tune write performance. Generally, set this larger for large files, smaller for small files. As I say above, test this with different values under different workloads to test what works best.
window-size – this is the size of the write behind, per file buffer. This holds data of a changed file which has not yet been committed to disk. Use this with care!
cache-size – This is the size in bytes of the local file cache. The GlusterFS client will cache read operations on the client to save the amount of network IO traffic. I am not sure where this data resides, however even if it is not explicitly committed to RAM, it’s likely the kernel would cache it.

Ivan Rossi

To have mount redundancy you write that it is necessary to create a volume config file.

By reading gluster admin manual at page 25, it seems that the backupvolfile=myredundanteserver option in /etc/fstab should get the same result.

Do you imply that this piece of documentation is not correct?

junaid

Hi James,

Does glusterfs work on the disks created with openstack as well. I am trying to back up my openstack disk with glusterfs. Not sure if this will work. Example this is what I have right now,
—–
-rw-rw-rw- 2 root root 5368709120 Dec 24 08:35 volume-10891405-569b-4630-aa62-9deb9a6f1087
—-

Thats the disk I am trying to back up with glusterfs.

cheis

Hi,
I’ve configured a redundant datashare with a volume config file. The problem is the automounting at boot. In a debian Wheezy I get the error:
[2014-02-10 09:08:35.156119] I [glusterfsd.c:1910:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.4.1 (/usr/sbin/glusterfs –log-file=/var/log/gluster.log –fuse-mountopts=allow_other,default_permissions,max_read=131072,loglevel=WARNING –volfile=/etc/glusterfs/datashare.vol –fuse-mountopts=allow_other,default_permissions,max_read=131072,loglevel=WARNING /mnt/share)
[2014-02-10 09:08:35.195355] E [mount.c:267:gf_fuse_mount] 0-glusterfs-fuse: cannot open /dev/fuse (No such file or directory)
[2014-02-10 09:08:35.195369] E [xlator.c:390:xlator_init] 0-fuse: Initialization of volume ‘fuse’ failed, review your volfile again

It seems a problem with the order of fuse during boot. I’ve found this link:

 but it's for Squeeze. In Wheezy fuse has been removed.
So what can I do?

Tnx.
Cheis.

Florian

Thanks for all your guides!

I’m trying to mount two gluster servers (with a replicated volume) for use in proxmox. Since it is only possible to specify one server through the GUI, I must do it manually, using your guide here…

This all works fine, but I’m unsure how to add the mounted volume as a “shared” storage to proxmox (so that CTs can be migrated away from *dead* nodes)? If I add it as “directory” in the proxmox storage GUI, the storage is still considered local and I can’t migrate a CT that was running on a just-failed node away from that node (I get the usual 595 no route to host error, similar to the case when using “regular” local storage…)

Thanks for any pointers!

Florian

I should have read your other article (/how-to/533-glusterfs-storage-mount-in-proxmox) more closely: There’s no need to add all server IPs, one’s enough as it communicates the other redundant server IPs automatically…

That said, in case of a particular gluster server failure, there’s a 42s delay before things continue to work, so if you are testing this – give it a minute ;-)

semiosis

Use dns round-robin for a more redundant mount, as described here: http://goo.gl/ktI6p

Joe Julian

Besides the “backupvolfile=myredundanteserver” option which is fine in very small installations, for larger installations the correct method for having redundant mount servers is to use round robin dns (rrdns). http://edwyseguru.wordpress.com/2012/01/09/using-rrdns-to-allow-mount-failover-with-glusterfs/

Mounting using volfiles precludes the ability to make live changes to your volumes and is much more open to user-error that can have catastrophic results (ie, loss of access, loss of data, etc.)

Peter H

I followed your post, trying to mount using a volfile. However, this does not work?
Mounting from any one of the 2 storage servers works fine, every change on that volume immediately replicates to second server, so the cluster in itself works.
It’s just mounting using your volfile setup that throws an error:
E [client-handshake.c:1778:client_query_portmap_cbk] 0-remote1: failed to get the port number for remote subvolume. Please run ‘gluster volume status’ on server to see if brick process is running.
Any ideas what’s going wrong?
The cluster works, gluster volume status & gluster volume info shows everything is ok. No firewall, selinux permissive, Centos 6.5 on all servers.

Andreas

hm, i do experience the same problem like Peter H on a debian wheezy, gluster version 3.5.2. Mounting the volume on the client manually works, but it doesn’t work during reboot. Either fuse isn’t loaded on time or before network is up, or something else is…

Andreas

hm, the problem seems to be quite … interesting:

mount -t glusterfs /etc/glusterfs/datastore.vol /mnt/glusterdata

[2014-10-06 13:54:38.639843] E [client-handshake.c:1760:client_query_portmap_cbk] 0-remote2: failed to get the port number for remote subvolume. Please run ‘gluster volume status’ on server to see if brick process is running.
[2014-10-06 13:54:38.639902] I [client.c:2229:client_rpc_notify] 0-remote2: disconnected from 192.168.94.88:24007. Client process will keep trying to connect to glusterd until brick’s port is available

but the glusterfs servers aren’t listening on port 24007 – they listen on port 49152. wtf?

Andreas

no good day today, should just start thinking before posting a comment… forget the comment above and this one, only the first one is relevant.

Yiorgos Stamoulis

Using Centos7,

I also got the “ERROR: Server name/volume name unspecified cannot proceed further..” error reported by others above while trying with a datastore.vol configuration file .

Looking into /sbin/mount.glusterfs (bash script), it does not seem capable of handling .vol type config files (as its sed’ing $1 for hostname:/nfs-type-export and bailing out with above when if does not find them).

Not much lost though as “-o backupvolfile-server=xx” works just fine. Will have to check however how important are all the other options that appear in the .vol file above and whether they can also be supplied as options.

Hope this was helpful.

Roger Marcus

Question: I get different results depending if I mount a glusterfs with mount “-t glusterfs” or “-t nfs”. Some files are missing from the glusterfs mount.

Details: I have two bananapi’s with a volume setup following your two bananapi how to sheet. The client to the volume is an Ubuntu server running 14.04 LTS.

From the two bananas, everything is intact. Also when the server is mounted with nfs, the missing files are accessible and show up with an “ls”. It’s just the -t glusterfs which doesn’t seem to see 50% of the files.

Any thoughts?

Tam

Thanks for the wonderful info. However I could not make the following scenario work.

Our setup is as follows,
– 2 gluster server with replication.
– from another server, I am mounting the gluster volume as redundant mount point.

Query,

Everything works fine as expected. If any one of the gluster server goes down, within 60 seconds from the third server I am able to access the gluster volume seamlessly. Now consider the case when the first server (mentioned in /etc/fstab) is brought down. The third server can access the gluster volume from the second gluster server. Good and it worked as expected. Now if I umount the mount point and try to re-mount, then it did not work when the first gluster server being down. Any solution to this ?

Sebastian Krätzig

Hey,
thanks for this guide, but you should update these lines:
> option remote-subvolume [VOLNAME]
…to something like this:
> option remote-subvolume [/path/to/brick]

VOLNAME would be the name of a volume, which will be shown by the command “gluster volume info/status”:
> :~# gluster volume info
>
> Volume Name: gfsvbackup
> Type: Distribute
> Volume ID: 0b7b6027-0b35-497d-bddb-4d64b34828b0
> Status: Started
> Number of Bricks: 3
> Transport-type: tcp
> Bricks:
> Brick1: server01:/export/vdc1/brick
> Brick2: server02:/export/vdd1/brick
> Brick3: server03:/export/vde1/brick

But in this case, you really need the path to the brick. For example “/export/vdc1/brick”.

Thanks again,
Sebastian

fdir

This gave me errors when trying to mount
gfs1.jamescoyle.net:/datastore /mnt/datastore glusterfs defaults,_netdev,backupvolfile-server=gfs2.jamescoyle.net 0 0

needed an extra dash (-) and an “s” to work on 3.10 for me, “backup-volfile-servers” instead of “backupvolfile-server”
gfs1.jamescoyle.net:/datastore /mnt/datastore glusterfs defaults,_netdev,backup-volfile-servers=gfs2.jamescoyle.net 0 0

m4rcc

Hi,
There is something i don’t get, this “backupvolfile-server” described here it is used just first time we mount the GlusterFS volume, right? I mean once we established the connectivity with that server FIRST TIME, from the GlusterFS Docs pasted below i understand we won’t need any further “backupvolfile-server”, is this right?

######The server specified in the mount command is only used to fetch the gluster configuration volfile describing the volume name. Subsequently, the client will communicate directly with the servers mentioned in the volfile (which might not even include the one used for mount).#######

Dragomir

Volume config file doesn’t work anymore in Debian 10, the fstab method should be used instead.