REX-Ray + Isilon Troubleshooting

When I installed REX-Ray + Isilon for the first time, I got a lot of errors and struggled to solve those issues. So I post the troubleshooting logs for the record.

Initial configuration parameters

It seemed to be fine for me, however, causes a lot of errors so don't refer this configuration parameters as an example.

[root@sunny ~]# cat  /etc/rexray/config.yml_for_Isilon
libstorage:
  service: isilon
isilon:
  endpoint: https://192.168.20.40:8080
  group: wheel
  username: root
  password: ****
  insecure: true
  volumePath: /ifs/volumes
  nfsHost: 192.168.20.40
  dataSubnet: 192.168.20.0

Issue 1: Rex-Ray failed to create a new volume

After setting/etc/rexray/config.yml file and REX-Ray serivce started, the following error showed up when createing a rexray volume.

[root@sunny ~]# rexray volume create rexray_isilon_1 --size=20
WARN[0000] executor not supported                        host=unix:///var/run/libstorage/480463663.sock server=rainbow-leg-mu service=isilon storageDriver=libstorage time=1497501440712
WARN[0000] cannot get local deviecs                      error=executor not supported host=unix:///var/run/libstorage/480463663.sock integrationDriver=linux osDriver=linux server=rainbow-leg-mu service=isilon storageDriver=libstorage time=1497501440753 txCR=1497501440 txID=d5c2a9b5-284c-49b0-5cb8-c2fcce550e78
FATA[0000] error creating volume                         error.status=404 error.inner.resourceID="rexray_isilon_1" volume=rexray_isilon_1

Check if API endpoint is working
Ran "rexray volume ls", then the following result came back. Even though it throwbacked a few warnings, seemingly API got information from the directory set as volumePath in the config file. So I assumed Isilon API was working normally.

[root@sunny ~]# rexray volume ls
WARN[0000] executor not supported                        host=unix:///var/run/libstorage/025799782.sock server=puzzle-hawk-tm service=isilon storageDriver=libstorage time=1497503127890
WARN[0000] cannot get local deviecs                      error=executor not supported host=unix:///var/run/libstorage/025799782.sock integrationDriver=linux osDriver=linux server=puzzle-hawk-tm service=isilon storageDriver=libstorage time=1497503127927 txCR=1497503127 txID=443f84df-3b2d-4101-41a0-d9638f34503a
ID          Name        Status     Size
.snapshot   .snapshot   available  0
README.txt  README.txt  available  0
data        data        available  0
home        home        available  0

Set Isilon NFS access privilege
Assuming API is working normally, then, there should be some reasons REX-Ray failed to create a new volume under the volumePath directory.
I searched google about Isilon NFS privilege and found the link below.
https://github.com/codedellemc/libstorage/issues/236
To use Isilon, create a new directory named volumes under menu File System -> File System Explorer. Use all defaults for RW access.
Next go to menu item Protocols -> UNIX Sharing (NFS). Create a New Export. Select the/ifs/volumes folder . Enable Mounting of sub-directories. Change the Root Permissions from nobody to root. Root can be found under FILE:system.
So I moved to Isilon OneFS management, change the NFS export to enable:
- Mount access to subdirectory
- Root user mapping to "root"user

But I still got the same error. Looking at the error log, it says "error=executor not supported". So next, I doubted if the REX-Ray was properly configured to use Isilon NFS.
Mount Isilon NFS
Tried to mount Isilon NFS to the REX-Ray server, then, this error returned.

[root@sunny ~]# mount -t nfs 192.168.20.40:/ifs/volumes /mnt/isilon
mount: wrong fs type, bad option, bad superblock on 192.168.20.40:/ifs/volumes,
       missing codepage or helper program, or other error
       (for several filesystems (e.g. nfs, cifs) you might
       need a /sbin/mount.<type> helper program)

       In some cases useful info is found in syslog - try
       dmesg | tail or so.

It means, the server was not ready to mount NFS. So installed nfs-utils.

yum install nfs-utils

As a result, NFS became successfully able to be mounted and writable.

[root@sunny ~]# mount -t nfs 192.168.20.40:/ifs/volumes  /mnt/isilon
[root@sunny ~]# ls /mnt/isilon/
[root@sunny ~]# vi /mnt/isilon/test_at_isilon_ifs_volumes
[root@sunny ~]# ls /mnt/isilon/
test_at_isilon_ifs_volumes

Issue 2: REX-Ray service failed to start

After the configuration change, however, REX-Ray came to fail starting its service.

[root@sunny ~]# rexray status
● rexray.service - rexray
   Loaded: loaded (/etc/systemd/system/rexray.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Thu 2017-06-15 16:30:52 JST; 6s ago
  Process: 7819 ExecStart=/usr/bin/rexray start -f (code=exited, status=2)
 Main PID: 7819 (code=exited, status=2)
(omitted)
Jun 15 16:30:52 sunny.weather.local systemd[1]: Unit rexray.service entered failed state.
Jun 15 16:30:52 sunny.weather.local systemd[1]: rexray.service failed.

To investigate this issue, used debug mode like below. Option "-l"means logging level. By setting logging level as "debug", REX-Ray shows you debug info when running a command.

rexray start -f -l debug

Set CIDR explicitly
In the debug info when REX-Ray failed to start the service, there was the following line saying CIDR is invalid.

ERRO[0001] error: api call failed                        error.dataSubnet=192.168.20.0 host=unix:///var/run/libstorage/602344592.sock route=serviceInspect server=plume-bolt-ki time=1497513046332 tls=false txCR=1497513046 txID=44eeacc4-6b3d-4e76-7a69-1acdeef71f00
DEBU[0001] api call error json                           apiErr={"message":"invalid data subnet","status":500,"error":{"dataSubnet":"192.168.20.0","inner":"invalid CIDR address: 192.168.20.0"}} host=unix:///var/run/libstorage/602344592.sock route=serviceInspect server=plume-bolt-ki time=1497513046333 tls=false txCR=1497513046 txID=44eeacc4-6b3d-4e76-7a69-1acdeef71f00
PANI[0001] error initializing instance ID cache          inner.inner.dataSubnet=192.168.20.0 inner.status=500

So I added CIDR explicitly to "dataSubnet"in the config.yml file like "192.168.20.0/24". As a result, REX-Ray started the service successfully. Here, "test_at_isilon_ifs_volumes"is the file created during Isilon NFS mounting test earlier.

[root@sunny ~]# rexray volume ls
ID                          Name                        Status     Size
test_at_isilon_ifs_volumes  test_at_isilon_ifs_volumes  available  0

Issue 3: Still REX-Ray failed to create a new volume

It all seemed to be fine, but unfortunately, I still got an error when creating a new volume.

[root@sunny ~]# rexray volume create rexray_isilon_1 --size=20
FATA[0000] error creating volume                         error.inner.resourceID="rexray_isilon_1" error.status=404 volume=rexray_isilon_1

Debug mode returned enormous amount of logs to me, but it didn't have any useful information for me to get over this issue.

[root@sunny ~]# rexray volume create rexray_isilon_1 --size=20 -l debug
INFO[0000] updated log level                             logLevel=debug
DEBU[0000] os.args                                       time=1497516974875 val=[rexray volume create rexray_isilon_1 --size=20 -l debug]
DEBU[0000] activating libStorage                         cmd=create time=1497516974875
DEBU[0000] parseSafeHost - no change                     postParse=unix:///var/run/libstorage
(omitted)
{{end}} time=1497516975398
FATA[0000] error creating volume                         error.inner.resourceID="rexray_isilon_1" error.status=404 volume=rexray_isilon_1

Is this because of privilege? In libstorage document , it says the user access to the Isilon cluster needs to have the following privilge. But I use "root"for REX-Ray, so this must be no problem.
The account used to access the Isilon cluster must be in a role with the following privileges:
Namespace Access (ISI_PRIV_NS_IFS_ACCESS)
Platform API (ISI_PRIV_LOGIN_PAPI)
NFS (ISI_PRIV_NFS)
Restore (ISI_PRIV_IFS_RESTORE)

Solution: Upgrade REX-Ray

I didn't have no clue at all at that time. There was nothing I could do. Only thing left was to ask help to REX-Ray developers. So I posted about this issue at https://github.com/codedellemc/rexray/issues/ .
Then I got a reply from a developer saying like "this behavior was a regression with REX-Ray 0.9.0, which was released the last Friday. So it should be fixed in the latest release."
So I upgraded REX-Ray to the latest version.
https://github.com/codedellemc/rexray/blob/master/.docs/about/release-notes.md

[root@sunny ~]# curl -sSL https://dl.bintray.com/emccode/rexray/install | sh

rexray has been installed to /usr/bin/rexray

REX-Ray
-------
Binary: /usr/bin/rexray
Flavor: client+agent+controller
SemVer: 0.9.1
OsArch: Linux-x86_64
Branch: v0.9.1
Commit: 2373541479478b817a8b143629e552f404f75226
Formed: Sat, 10 Jun 2017 04:23:38 JST

libStorage
----------
SemVer: 0.6.1
OsArch: Linux-x86_64
Branch: v0.9.1
Commit: fd26f0ec72b077ffa7c82160fbd12a276e12c2ad
Formed: Sat, 10 Jun 2017 04:23:05 JST

After the upgrade, finally, a new volume was successfully created.

[root@sunny ~]# rexray volume create rexray_isilon_1 --size=2
ID               Name             Status     Size
rexray_isilon_1  rexray_isilon_1  available  0

Memo: Proper configuration parameters

This is the final version of/etc/rexray/config.yml file.

[root@sunny ~]# cat  /etc/rexray/config.yml_for_Isilon
libstorage:
  service: isilon
isilon:
  endpoint: https://192.168.20.40:8080
  group: wheel
  username: root
  password: ****
  insecure: true
  volumePath: /ifs/volumes
  nfsHost: 192.168.20.40
  dataSubnet: 192.168.20.0/24