nfs-ganesha编译

Ethereal Lv4
  2024-06-14 22:13:36 2024-06-14 22:13 Created   2025-02-10 18:31:49 2025-02-10 18:31 Updated  

1. 编译

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
git clone https://github.com/nfs-ganesha/nfs-ganesha.git
cd nfs-ganesha
git submodule update --init
git checkout V5-stable
mkdir build
cd build
sudo apt install -y krb5-user libpam-krb5 libpam-ccreds libkrb5-dev
sudo apt install libcephfs-dev
sudo apt install libacl1-dev
sudo apt install liburcu-dev
sudo apt install gcc git cmake autoconf libtool bison flex
sudo apt-get install uuid-dev
sudo apt-get install librgw-dev
cmake -DUSE_ACL_MAPPING=ON -DUSE_FSAL_RGW=ON -DUSE_FSAL_CEPH=ON ../src/
make & make install

其中，-DUSE_ACL_MAPPING=ON使得nfs-ganesha可以读取到正确的ACL，否则会报错。-DUSE_FSAL_RGW=ON -DUSE_FSAL_CEPH=ON开启使用ceph。

此外，针对OpenEuler，其命令为

1
2
3
4
5
6
7
8
9
10
11
12
13
git clone https://github.com/nfs-ganesha/nfs-ganesha.git
cd nfs-ganesha
git submodule update --init
git checkout V5-stable
mkdir build
cd build
dnf install -y krb5-workstation pam_krb5 krb5-devel
dnf install libacl-devel userspace-rcu-devel libnsl2-devel
dnf install gcc g++ git cmake autoconf libtool bison flex
dnf install libuuid-devel make
cmake -DUSE_ACL_MAPPING=ON ../src/ # 这里也可按照上述使用-DUSE_FSAL_RGW=ON -DUSE_FSAL_CEPH=ON
make
make install

2. 服务启动

如果使用docker，那么必须以--privileged模式运行

1
docker run --privileged -v /root/Softwares:/root/Softwares --name openeuler -d hub.oepkgs.net/openeuler/openeuler:latest /bin/bash -c "tail -f /dev/null"

检查服务名称是否正确

1
2
# mv nfs-ganesha-lock.service.el7 nfs-ganesha-lock.service
# mv nfs-ganesha.service.el7 nfs-ganesha.service

安装kerberos

1
2
sudo apt-get install krb5-kdc krb5-admin-server
sudo systemctl restart krb5-admin-server

针对OpenEuler为

1
2
sudo dnf install krb5-server krb5-workstation krb5-libs
sudo systemctl restart kadmin

新建realm

1
2
kdb5_util create
# krb5_newrealm

记下当前的REALM，例如EXAMPLE.COM

确认host存在

1
2
cat /etc/krb5.conf
# 查看对应REALM的域名，然后在/etc/hosts中添加，ip可以通过ifconfig获得

新建nfs-ganesha需要的key

1
2
3
4
5
6
7
8
9
10
11
kadmin.local
add_principal nfs/hostname # 输入密码
list_principals
quit
ktutil
ktutil:  addent -password -p nfs/hostname@EXAMPLE.COM -k 1 -e aes256-cts-hmac-sha1-96
Password for nfs/hostname@EXAMPLE.COM:  # 输入principal的密码
ktutil:  wkt /etc/krb5.keytab
ktutil:  quit
sudo klist -k /etc/krb5.keytab
kinit -k -t /etc/krb5.keytab nfs/hostname@EXAMPLE.COM

安装rpcbind

1
sudo apt install rpcbind

针对OpenEuler为

1
2
3
sudo dnf install rpcbind
systemctl status rpcbind
# rpcbind

编写文件

1
vim /etc/ganesha/ganesha.conf

文件内容如下

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
EXPORT
{
        # Export Id (mandatory, each EXPORT must have a unique Export_Id)
        Export_Id = 12345;

        # Exported path (mandatory)
        Path = /etc;

        # Pseudo Path (required for NFSv4 or if mount_path_pseudo = true)
        Pseudo = /etc;

        # Restrict the protocols that may use this export.  This cannot allow
        # access that is denied in NFS_CORE_PARAM.
        Protocols = 3,4;

        # Access type for clients.  Default is None, so some access must be
        # given. It can be here, in the EXPORT_DEFAULTS, or in a CLIENT block
        Access_Type = RW;

        # Whether to squash various users.
        Squash = root_squash;

        # Allowed security types for this export
        Sectype = sys,krb5,krb5i,krb5p;

        # Exporting FSAL
        FSAL {
                Name = VFS;
        }
}

启动nfs-ganesha

1
2
sudo systemctl daemon-reload
sudo systemctl start nfs-ganesha

针对docker容器内程序，直接启动

1
2
3
4
mkdir -p /run/ganesha
mkdir -p /var/log/ganesha
mkdir -p /var/lib/nfs/ganesha
./ganesha.nfsd -L /var/log/ganesha.log -f /etc/ganesha/ganesha.conf

3. 测试

1
2
3
sudo showmount -e hostname
sudo mount -v -t nfs4 ethereal-vm:/etc /home/ethereal/Documents/tmp
sudo umount /home/ethereal/Documents/tmp

4. 其他

查看日志

1
journalctl -u nfs-ganesha --since "2024-06-21 10:35:00"

kerberos相关：

新建Realm

1
krb5_newrealm

新建数据库

1
kdb5_util create -s -r HADOOP.COM

新建principal

1
2
3
4
kadmin.local
add_principal nfs/hostname # 输入密码
list_principals
quit

查看principal

1
2
3
kadmin.local
list_principals
quit

新建条目

1
ktutil:  addent -password -p nfs/hostname@REALM -k 1 -e aes256-cts-hmac-sha1-96

写入到数据库条目表

1
ktutil:  wkt /etc/krb5.keytab

列出数据库条目表中内容

1
sudo klist -k /etc/krb5.keytab

5. 配置kerberos鉴权

5.1 kerberos服务端

  1. 安装
1
yum -y install krb5-server krb5-workstation
  1. 编辑 /etc/krb5.conf 文件， 取消注释所有行，并替换kdc, admin server的设置，这里用default realm EXAMPLE.COM , 最后配置文件如下：
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
[logging]
 default = FILE:/var/log/krb5libs.log
 kdc = FILE:/var/log/krb5kdc.log
 admin_server = FILE:/var/log/kadmind.log

[libdefaults]
 dns_lookup_realm = false
 ticket_lifetime = 24h
 renew_lifetime = 7d
 forwardable = true
 rdns = falsei
 default_realm = EXAMPLE.COM
 default_ccache_name = KEYRING:persistent:%{uid}

[realms]
 EXAMPLE.COM = {
  kdc = remote.example.com
  admin_server = remote.example.com
 }

[domain_realm]
 .example.com = EXAMPLE.COM
 example.com = EXAMPLE.COM
  1. 修改 /etc/hosts 文件
1
2
3
192.168.57.3 server.example.com remote
192.168.57.2 remote.example.com server
192.168.57.204 client.example.com client

  1. 修改 /var/kerberos/krb5kdc/kdc.conf 文件
    由于我们使用default realm 所以无需修改此文件，若用另外的realm，需替换此文件中的EXAMPLE.COM

  2. 运行命令以下命令创建**数据库，并按照提示设置数据库密码

1
2
3
4
5
6
7
[root@remote ~]# kdb5_util create  -s
Loading random data
Initializing database '/var/kerberos/krb5kdc/principal' for realm 'EXAMPLE.COM',
master key name 'K/M@EXAMPLE.COM'
You will be prompted for the database Master Password.
It is important that you NOT FORGET this password.
Enter KDC database master key:
  1. 启动kerberos
1
2
systemctl start krb5kdc kadmin 
systemctl enable krb5kdc kadmin
  1. 运行kadmin.local 来管理kdc 并添加nfs server 和 client的 主机信息
1
2
3
4
5
6
7
8
[root@remote ~]# kadmin.local
Authenticating as principal root/admin@EXAMPLE.COM with password.
kadmin.local:  addprinc -randkey host/server.example.com
WARNING: no policy specified for host/server.example.com@EXAMPLE.COM; defaulting to no policy
Principal "host/server.example.com@EXAMPLE.COM" created.
kadmin.local:  addprinc -randkey host/client.example.com
WARNING: no policy specified for host/client.example.com@EXAMPLE.COM; defaulting to no policy
Principal "host/client.example.com@EXAMPLE.COM" created.
  1. 添加nfs server 和 client的 nfs 服务主机和客户端
1
2
3
4
5
6
kadmin.local:  addprinc -randkey nfs/server.example.com
WARNING: no policy specified for nfs/server.example.com@EXAMPLE.COM; defaulting to no policy
Principal "nfs/server.example.com@EXAMPLE.COM" created.
kadmin.local:  addprinc -randkey nfs/client.example.com
WARNING: no policy specified for nfs/client.example.com@EXAMPLE.COM; defaulting to no policy
Principal "nfs/client.example.com@EXAMPLE.COM" created.
1
2
3
4
5
6
7
8
9
10
11
kadmin.local:  listprincs
K/M@EXAMPLE.COM
host/client.example.com@EXAMPLE.COM
host/server.example.com@EXAMPLE.COM
kadmin/admin@EXAMPLE.COM
kadmin/changepw@EXAMPLE.COM
kadmin/server.example.com@EXAMPLE.COM
krbtgt/EXAMPLE.COM@EXAMPLE.COM
nfs/client.example.com@EXAMPLE.COM
nfs/server.example.com@EXAMPLE.COM
[root@remote ~]#
  1. 生成密钥
1
2
3
4
5
kadmin.local:  ktadd -k /tmp/server.keytab nfs/server.example.com
[以下省略输出]
kadmin.local:  ktadd -k /tmp/client.keytab nfs/client.example.com
[以下省略输出]
kadmin.local:  quit

5.2 nfs服务端

  1. 配置host

  2. 复制文件

1
2
scp remote:/etc/krb5.conf /etc/krb5.conf   
scp remote:/tmp/server.keytab /etc/krb5.keytab
  1. 导出文件路径
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
EXPORT
{
        # Export Id (mandatory, each EXPORT must have a unique Export_Id)
        Export_Id = 12345;

        # Exported path (mandatory)
        Path = /etc/nfs;

        # Pseudo Path (required for NFSv4 or if mount_path_pseudo = true)
        Pseudo = /etc/nfs;

        # Restrict the protocols that may use this export.  This cannot allow
        # access that is denied in NFS_CORE_PARAM.
        Protocols = 3,4;

        # Access type for clients.  Default is None, so some access must be
        # given. It can be here, in the EXPORT_DEFAULTS, or in a CLIENT block
        Access_Type = RW;

        # Whether to squash various users.
        Squash = no_root_squash;

        # Allowed security types for this export
        Sectype = sys,krb5,krb5i,krb5p;

        # Exporting FSAL
        FSAL {
                Name = VFS;
        }
}

NFS_KRB5
{
   PrincipalName = nfs ;
   KeytabPath = /etc/krb5.keytab ;
   Active_krb5 = YES ;
}
  1. 初始化
1
sudo kinit -k -t /etc/krb5.keytab nfs/openeuler-vm-2203@EXAMPLE.COM

5.3 客户端配置

  1. 配置host

  2. 复制文件
    scp remote:/etc/krb5.conf /etc/krb5.conf
    scp remote:/tmp/server.keytab /etc/krb5.keytab

  3. 挂载

1
sudo mount -v -t nfs4 -o sec=krb5 openeuler-vm-2203:/etc/nfs /mnt/tmp
  1. 初始化
    sudo kinit -k -t /etc/krb5.keytab nfs/openeuler-vm-2203@EXAMPLE.COM

6. 系统测试

6.1 使用文件测试工具测试

6.2 使用kdevops /pynfs 测试

pynfs中含有使用python编写的测试pnfs的测试脚本，因此可以直接用来测试。

  1. 安装anaconda

  2. 使用python2.7

  3. 执行构建

1
2
python setup.py build
python setup.py install
  1. 在原路径中运行
1
python ./testserver.py localhost:/etc/nfs pnfs
  1. 需要注意的是服务器必须使用MDS（ceph），保证存在flag为EXCHGID4_FLAG_USE_PNFS_MDS。否则报错为：
1
2
CSID7    st_current_stateid.testOpenLayoutGet                     : FAILURE
           NameError: global name 'fail' is not defined

位置为devkernel/pynfs/nfs4.1/nfs4client.py中的new_pnfs_client_session。

服务端返回此标识位置为src/Protocols/NFS/nfs4_op_exchange_id.c中的nfs4_op_exchange_id中的

1
2
3
4
5
6
7
8
if ((arg_EXCHANGE_ID4->eia_flags & EXCHGID4_FLAG_MASK_PNFS) == 0) {
    if (nfs_param.nfsv4_param.pnfs_mds)
        server_pnfs_flags |= EXCHGID4_FLAG_USE_PNFS_MDS;
    else
        server_pnfs_flags |= EXCHGID4_FLAG_USE_NON_PNFS;
    if (nfs_param.nfsv4_param.pnfs_ds)
        server_pnfs_flags |= EXCHGID4_FLAG_USE_PNFS_DS;
}

当设置参数后将启用此标识，参数解析位于src/support/nfs_read_conf.c

1
2
3
4
5
.blk_desc.name = "NFSv4",
.blk_desc.type = CONFIG_BLOCK,
.blk_desc.flags = CONFIG_UNIQUE,  /* too risky to have more */
.blk_desc.u.blk.init = noop_conf_init,
.blk_desc.u.blk.params = version4_params,

因此只需要在/etc/ganesha/ganesha.conf中添加

1
2
3
4
NFSv4
{
   PNFS_MDS = YES ;
}

重启nfs-ganesha服务后，发现即可正确使用pnfs。

但是，pynfs要求必须使用BLOCK布局，位置在devkernel/pynfs/nfs4.1/server41tests/st_getdevicelist.py

1
blocksize = get_blocksize(sess, use_obj(env.opts.path))

否则返回，对应服务端代码位于src/FSAL/FSAL_GLUSTER/mds.c中的handle_ops_pnfs与pnfs_layout_get。

1
2
3
4
GETLAYOUT1 st_getdevicelist.testGetLayout                         : FAILURE
           layout_type does not contain BLOCK
LAYOUTCOMMIT1 st_getdevicelist.testLayoutCommit                   : FAILURE
           layout_type does not contain BLOCK

将get_blocksize注释掉后显示

1
2
3
GETLAYOUT1 st_getdevicelist.testGetLayout                         : FAILURE
           OP_LAYOUTGET should return NFS4_OK, instead got
           NFS4ERR_LAYOUTUNAVAILABLE

对应服务端代码位置在src/FSAL/default_methods.c，其中def_handle_ops变量定义了函数指针。

1
2
3
4
5
6
7
static nfsstat4 layoutget(struct fsal_obj_handle *obj_hdl,
              XDR *loc_body,
              const struct fsal_layoutget_arg *arg,
              struct fsal_layoutget_res *res)
{
    return NFS4ERR_LAYOUTUNAVAILABLE;
}

标准文件系统不支持此操作，但是CEPH等支持。

参考

nfs-ganesha(用户态NFS)编译、安装和使用_nfs-ganesha 编译-CSDN博客

Ubuntu1804安装MIT Kerberos_ubuntu kerberos-CSDN博客

GitHub - nfs-ganesha/nfs-ganesha: NFS-Ganesha is an NFSv3,v4,v4.1 fileserver that runs in user mode on most UNIX/Linux systems

ceph/doc/cephfs/nfs.rst at main · ceph/ceph · GitHub

nfs-ganesha/src/COMPILING_HOWTO.txt at V5-stable · nfs-ganesha/nfs-ganesha · GitHub

Make Error when Linking C shared library libganesha_nfsd.so · Issue #882 · nfs-ganesha/nfs-ganesha · GitHub

RPCSEC_GSS · nfs-ganesha/nfs-ganesha Wiki · GitHub

【求助kerberos配置】Cannot contact any KDC for realm ‘EXAMPLE.COM’ -CSDN社区

nfs 连接用户名 nfs密码认证指定用户_kekenai的技术博客_51CTO博客

一文搞定Kerberos - 知乎

Kerberos认证原理及基于Kerberos认证的NFS文件共享 - 佟晖 - 博客园

RPCSEC_GSS · nfs-ganesha/nfs-ganesha Wiki

centos7 部署kerberos实现nfs用户密码认证挂载-腾讯云开发者社区-腾讯云

Kerberos常用命令用法_addprinc-CSDN博客

ubuntu - Kerberized nfs4 mounts ERROR: No credentials found for connection to server - Server Fault

Kerberized nfs4 挂载错误：找不到连接到服务器的凭据 - server

kerberos使用详解_krb5-CSDN博客

【求助kerberos配置】Cannot contact any KDC for realm ‘EXAMPLE.COM’ -CSDN社区

linux-kdevops/kdevops: kdevops main repository: Generalized devops infrastructure for Linux kernel development

kdevops/docs/nfs.md at main · linux-kdevops/kdevops

Installing Ansible — Ansible Community Documentation

(80 条消息) ansible执行命令的时候报错msg”: “Missing sudo password 有知道的吗？ - 知乎

ansible踩坑指南_missing sudo password-CSDN博客

virt-install找不到default网络_没有网络与名称 ‘default’ 映射-CSDN博客

dang/pynfs: Clone of git://git.linux-nfs.org/projects/bfields/pynfs.git

[pynfs] Move to xdrlib3 - Patchwork

Parallel NFS (pNFS)

  • Title: nfs-ganesha编译
  • Author: Ethereal
  • Created at: 2024-06-14 22:13:36
  • Updated at: 2025-02-10 18:31:49
  • Link: https://ethereal-o.github.io/2024/06/14/nfs-ganesha编译/
  • License: This work is licensed under CC BY-NC-SA 4.0.
 Comments
On this page
nfs-ganesha编译
  1. 1. 编译
  2. 2. 服务启动
  3. 3. 测试
  4. 4. 其他
  5. 5. 配置kerberos鉴权
    1. 5.1 kerberos服务端
    2. 5.2 nfs服务端
    3. 5.3 客户端配置
  6. 6. 系统测试
    1. 6.1 使用文件测试工具测试
    2. 6.2 使用kdevops /pynfs 测试
  7. 参考