After numa source code make install, apt-get install libnuma-dev can not work again, both on x86 and ARM.
So numa has to install from source code

1. apt-get install libnuma-dev
   success

2,
$git clone https://github.com/numactl/numactl.git
$cd numactl
$./autogen.sh
./configure, make; sudo make install

Can remove library in /use/lib, or sudo make uninstall

3. apt-get install libnuma-dev not work again

On x86
apt-get remove libnuma-dev
apt-get install libnuma-dev

ls /usr/lib/x86_64-linux-gnu/libnuma.*
/usr/lib/x86_64-linux-gnu/libnuma.a /usr/lib/x86_64-linux-gnu/libnuma.so

Only has link libnuma.so, but no libnuma.so.1.0.0

$ ll /usr/lib/x86_64-linux-gnu/libnuma.so
lrwxrwxrwx 1 root root 16 10月 20 2017 /usr/lib/x86_64-linux-gnu/libnuma.so -> libnuma.so.1.0.0
$ ls /usr/lib/x86_64-linux-gnu/libnuma.so.1.0.0
ls: cannot access '/usr/lib/x86_64-linux-gnu/libnuma.so.1.0.0': No such file or directory

On ARM
$ sudo apt-get remove libnuma-dev
$ sudo apt-get install libnuma-dev
$ numactl
numactl: error while loading shared libraries: libnuma.so.1: cannot open shared object file: No such file or directory
$ ll /usr/lib/aarch64-linux-gnu/libnuma*
-rw-r--r-- 1 root root 77642 Sep 8 2018 /usr/lib/aarch64-linux-gnu/libnuma.a
lrwxrwxrwx 1 root root 16 Sep 8 2018 /usr/lib/aarch64-linux-gnu/libnuma.so -> libnuma.so.1.0.0
Only link, no real library

Dear developers:
Thank you for your checking. In the method test_sec_cipher_async, the lock &test_sec_mutex maybe not be released correctly when the control flow goes to the else if branch and go to out, bypassing the unreleased lock.

static int test_sec_cipher_async(void *arg)
{
	...;
	pthread_mutex_lock(&test_sec_mutex);
	...;
	do {
try_do_again:
		ret = wd_do_cipher_async(h_sess, req);
		if (ret == -EBUSY) { // busy
			usleep(100);
			goto try_do_again;
		} else if (ret) {
			SEC_TST_PRT("test sec cipher send req is error!\n");
			goto out;   // bypass the unlock, leading to unreleased lock
		}
		cnt--;
	} while (cnt);
	pthread_mutex_unlock(&test_sec_mutex);
	...;
out:
	if (h_sess)
		wd_cipher_free_sess(h_sess);

	return ret;
}

Best,

digest stream mode must be 64-byte aligned!

https://github.com/Linaro/linux-kernel-warpdrive/tree/uacce-devel-5.12
make defconfig

1. buildin, cmdline:
   hisi_zip.uacce_mode=2 hisi_hpre.uacce_mode=2 hisi_sec2.uacce_mode=2

2. Or use module:
   make menuconifg
   -- Cryptographic API --->
   [] Hardware crypto devices --->
   Support for HiSilicon SEC2 crypto block cipher accelerator
   Support for HiSilicon ZIP accelerator
   Support for HISI HPRE accelerator

sudo insmod hisi_qm.ko
sudo insmod sec2/hisi_sec2.ko uacce_mode=2
sudo insmod hpre/hisi_hpre.ko uacce_mode=2

$ cat /sys/class/uacce/hisi_sec2-0/flags
128

3. uadk v1
   // verify kae for v1
   git clone https://github.com/Linaro/uadk
   // conf.sh build v1

•   --with-openssl_dir=~/work/openssl \
  

make clean
./conf.sh; make -j4
sudo make install

4. kae
   git clone https://github.com/kunpengcompute/KAE
   // install to system /usr/local
   cd KAE
   ./configure
   LIBRARY_PATH=/usr/local/lib C_INCLUDE_PATH=~/work/uadk/v1 make
   sudo make install
   openssl engine -c kae

// Or kae.sh for local build
#!/bin/bash -e

export WORKSPACE=${PWD}

cd ${WORKSPACE}/uadk-master
./cleanup.sh; ./autogen.sh;
./configure
--prefix=${WORKSPACE}/uadk-shared-v2/usr/local
--includedir=${WORKSPACE}/uadk-shared-v2/usr/local/include/uadk
--disable-static
--enable-shared
make -j$(nproc)
make install

#sudo \

LD_LIBRARY_PATH=${WORKSPACE}/uadk-shared-v2/usr/local/lib/ \

PATH=${WORKSPACE}/uadk-shared-v2/usr/local/bin:${PATH} \

C_INCLUDE_PATH=${WORKSPACE}/uadk-shared-v2/usr/local/include/ \

${WORKSPACE}/uadk-master/test/sanity_test.sh

cd ${WORKSPACE}/KAE
make clean
./configure --openssl_path=${WORKSPACE}/uadk-shared-v2
LIBRARY_PATH=${WORKSPACE}/uadk-shared-v2/usr/local/lib C_INCLUDE_PATH=${WORKSPACE}/uadk-master/v1 make
make install
export LD_LIBRARY_PATH=${WORKSPACE}/uadk-shared-v2/usr/local/lib
openssl engine -c kae

5. issues
   linaro@ubuntu:/tmp/ci$ openssl speed -elapsed -multi 1 -engine kae rsa2048
   Forked child 0
   unable to open /var/log/kae.log, Permission denied
   engine "kae" set.
   out bytes 0 error!
   [2021-05-19 06:11:22][error][alg/pkey/hpre_wd.c:372:hpre_rsa_sync()] hpre do rsa fail!
   +DTP:2048:private:rsa:10
   out bytes 0 error!
   [2021-05-19 06:11:22][error][alg/pkey/hpre_wd.c:372:hpre_rsa_sync()] hpre do rsa fail!
   out bytes 0 error!

$ openssl engine -t uadk
(uadk) uadk hardware engine support
[ available ]
281473677588864:error:25066067:DSO support routines:dlfcn_load:could not load the shared library:crypto/dso/dso_dlfcn.c:117:filename(libproviders.so): libproviders.so: cannot open shared object file: No such file or directory
281473677588864:error:25070067:DSO support routines:DSO_load:could not load the shared library:crypto/dso/dso_lib.c:162:
281473677588864:error:0E07506E:configuration file routines:module_load_dso:error loading dso:crypto/conf/conf_mod.c:221:module=providers, path=providers
281473677588864:error:0E076071:configuration file routines:module_run:unknown module name:crypto/conf/conf_mod.c:162:module=providers

commit b3db20a
Author: Haojian Zhuang [email protected]
Date: Wed Dec 30 20:38:35 2020 +0800

wd: replace mutex lock by spin lock in ctx structure

/bin/bash ../../libtool --tag=CC --mode=link aarch64-linux-gnu-gcc -Wall -fno-strict-aliasing -I../../include -pthread -g -O2 -Wl,-rpath,'/usr/local/lib' -o test_hisi_hpre test_hisi_hpre.o ../../.libs/libwd.a ../../.libs/libwd_crypto.a ../../.libs/libhisi_hpre.a -ldl
libtool: link: aarch64-linux-gnu-gcc -Wall -fno-strict-aliasing -I../../include -pthread -g -O2 -Wl,-rpath -Wl,/usr/local/lib -o test_hisi_hpre test_hisi_hpre.o ../../.libs/libwd.a ../../.libs/libwd_crypto.a ../../.libs/libhisi_hpre.a -ldl -pthread
/usr/bin/ld: ../../.libs/libhisi_hpre.a(hisi_hpre.o): in function ecc_sign_out_transfer': /home/buildslave/uadk/drv/hisi_hpre.c:1225: undefined reference to wd_sm2_get_sign_out_params'
/usr/bin/ld: ../../.libs/libhisi_hpre.a(hisi_hpre.o): in function sm2_enc_out_transfer': /home/buildslave/uadk/drv/hisi_hpre.c:1279: undefined reference to wd_sm2_get_enc_out_params'
/usr/bin/ld: ../../.libs/libhisi_hpre.a(hisi_hpre.o): in function sm2_kg_out_transfer': /home/buildslave/uadk/drv/hisi_hpre.c:1259: undefined reference to wd_sm2_get_kg_out_params'
/usr/bin/ld: ../../.libs/libhisi_hpre.a(hisi_hpre.o): in function ecc_prepare_pubkey': /home/buildslave/uadk/drv/hisi_hpre.c:847: undefined reference to wd_ecc_get_pubkey_params'
/usr/bin/ld: ../../.libs/libhisi_hpre.a(hisi_hpre.o): in function ecc_prepare_prikey': /home/buildslave/uadk/drv/hisi_hpre.c:784: undefined reference to wd_ecc_get_prikey_params'
/usr/bin/ld: ../../.libs/libhisi_hpre.a(hisi_hpre.o): in function ecc_prepare_sign_in': /home/buildslave/uadk/drv/hisi_hpre.c:970: undefined reference to wd_ecc_get_prikey_params'
collect2: error: ld returned 1 exit status
make[3]: *** [Makefile:395: test_hisi_hpre] Error 1
make[3]: Leaving directory '/home/buildslave/uadk/test/hisi_hpre_test'
make[2]: *** [Makefile:352: all-recursive] Error 1
make[2]: Leaving directory '/home/buildslave/uadk/test'
make[1]: *** [Makefile:722: all-recursive] Error 1
make[1]: Leaving directory '/home/buildslave/uadk'
make: *** [Makefile:516: all] Error 2

https://github.com/Linaro/uadk/tree/nginx-fork

test_hisi_sec --digest 0 --optype 0 --pktlen 16 --keylen 16 --times 1 --sync --multi 1
SEC BD digest fail! done=0x1, etype=0x11

In the middle of sec_digest_sync_once, fork a process start sec_digest_sync_once again,
report error: SEC BD digest fail! done=0x1, etype=0x11

After appended rand case in "example.c", stream mode is failed in NOSVA scenario. It could be observed in "hzhuang1-sva-3.2" branch.

https://github.com/Linaro/linux-kernel-warpdrive/tree/uacce-devel-5.12
make defconfig
make menuconifg
-- Cryptographic API --->
[] Hardware crypto devices --->
Support for HiSilicon SEC2 crypto block cipher accelerator
Support for HiSilicon ZIP accelerator
Support for HISI HPRE accelerator

sudo insmod hisi_qm.ko
sudo insmod hisi_sec2.ko uacce_mode=2

$ cat /sys/class/uacce/hisi_sec2-0/flags
128

./cleanup.sh;./autogen.sh;
./configure
--host aarch64-linux-gnu
--target aarch64-linux-gnu
--includedir=/usr/local/include/uadk
--with-openssl_dir=~/work/openssl
--enable-static --with-static_drv --disable-shared

make

$ ./v1/test/hisi_sec_test/test_hisi_sec
Segmentation fault (core dumped)

//async
test_hisi_hpre rsa-sgn --mode=crt --perf --trd_mode=async -t 1 --key_bits=2048

//sync
test_hisi_hpre rsa-sgn --mode=crt --perf --trd_mode=sync --key_bits=2048

sync:
23:<< sync 2 thread rsa-gen crt mode 2048 key_bits at 20899.988 ops!
45:<< sync 2 thread rsa-gen crt mode 2048 key_bits at 18341.109 ops!
67:<< sync 2 thread rsa-gen crt mode 2048 key_bits at 21087.043 ops!
89:<< sync 2 thread rsa-gen crt mode 2048 key_bits at 21136.064 ops!
111:<< sync 2 thread rsa-gen crt mode 2048 key_bits at 20772.770 ops!

async:
22:<< async 1 thread rsa-gen crt mode 2048 key_bits at 3570.233 ops!
43:<< async 1 thread rsa-gen crt mode 2048 key_bits at 58572.301 ops!
64:<< async 1 thread rsa-gen crt mode 2048 key_bits at 156542.062 ops!
85:<< async 1 thread rsa-gen crt mode 2048 key_bits at 4922.516 ops!
106:<< async 1 thread rsa-gen crt mode 2048 key_bits at 27717.783 ops!

async+bind:
numactl --cpubind=0 --membind=0 test_hisi_hpre rsa-sgn --mode=crt --perf --trd_mode=async -t 1 --key_bits=2048
linaro@ubuntu:~/test$ grep -rn "async 1 thread rsa-gen crt" log.2
22:<< async 1 thread rsa-gen crt mode 2048 key_bits at 6391.080 ops!
43:<< async 1 thread rsa-gen crt mode 2048 key_bits at 10102.447 ops!
64:<< async 1 thread rsa-gen crt mode 2048 key_bits at 18771.873 ops!
85:<< async 1 thread rsa-gen crt mode 2048 key_bits at 11188.066 ops!
106:<< async 1 thread rsa-gen crt mode 2048 key_bits at 8320.621 ops!
127:<< async 1 thread rsa-gen crt mode 2048 key_bits at 19878.707 ops!
148:<< async 1 thread rsa-gen crt mode 2048 key_bits at 6042.568 ops!
169:<< async 1 thread rsa-gen crt mode 2048 key_bits at 7707.399 ops!
190:<< async 1 thread rsa-gen crt mode 2048 key_bits at 1772.593 ops!
211:<< async 1 thread rsa-gen crt mode 2048 key_bits at 5408.329 ops!
232:<< async 1 thread rsa-gen crt mode 2048 key_bits at 4921.700 ops!
253:<< async 1 thread rsa-gen crt mode 2048 key_bits at 19461.562 ops!
274:<< async 1 thread rsa-gen crt mode 2048 key_bits at 17051.154 ops!
295:<< async 1 thread rsa-gen crt mode 2048 key_bits at 17449.238 ops!
316:<< async 1 thread rsa-gen crt mode 2048 key_bits at 4229.448 ops!
337:<< async 1 thread rsa-gen crt mode 2048 key_bits at 17965.992 ops!
358:<< async 1 thread rsa-gen crt mode 2048 key_bits at 19314.020 ops!
379:<< async 1 thread rsa-gen crt mode 2048 key_bits at 18230.029 ops!
400:<< async 1 thread rsa-gen crt mode 2048 key_bits at 4987.757 ops!
421:<< async 1 thread rsa-gen crt mode 2048 key_bits at 17950.393 ops!
442:<< async 1 thread rsa-gen crt mode 2048 key_bits at 18874.283 ops!
463:<< async 1 thread rsa-gen crt mode 2048 key_bits at 3388.029 ops!
484:<< async 1 thread rsa-gen crt mode 2048 key_bits at 120603.016 ops!
505:<< async 1 thread rsa-gen crt mode 2048 key_bits at 16211.062 ops!
526:<< async 1 thread rsa-gen crt mode 2048 key_bits at 21487.039 ops!
547:<< async 1 thread rsa-gen crt mode 2048 key_bits at 58381.984 ops!
568:<< async 1 thread rsa-gen crt mode 2048 key_bits at 10157.367 ops!
589:<< async 1 thread rsa-gen crt mode 2048 key_bits at 15453.960 ops!
610:<< async 1 thread rsa-gen crt mode 2048 key_bits at 5289.558 ops!
631:<< async 1 thread rsa-gen crt mode 2048 key_bits at 20218.229 ops!

In guest system (openEuler), init2 fails since wd_get_usable_list failed to get usable devices(-19)!
Host system init2 works.

Guest:
cat /sys/class/uacce/hisi_sec2-0/device/numa_node
-1

uadk_tool test --m sec --digest 0 --sync --optype 0 --pktlen 16 --keylen 16 --times 1 --multi 1 --init 2
driver_type=3
HW
numa_id=-1 p=0x3cdbd9e0
p=(nil)
failed to get usable devices(-19)!

Host:
HW
numa_id=2 p=0xaaaac695a030
numa_id=0 p=0xaaaac695a440

GCC 12 emits this error:

wd.c: In function 'wd_is_isolate':
wd.c:193:21: error: the comparison will always evaluate as 'true' for the address of 'dev_root' will never be NULL [-Werror=address]
  193 |         if (!dev || !dev->dev_root)
      |                     ^
In file included from wd.c:21:
./include/wd.h:131:14: note: 'dev_root' declared here
  131 |         char dev_root[PATH_STR_SIZE];
      |              ^~~~~~~~

dev_root being an initialized array, it cannot be NULL.
It should be fixed with this patch:

--- a/wd.c
+++ b/wd.c
@@ -190,7 +190,7 @@ int wd_is_isolate(struct uacce_dev *dev)
        int value = 0;
        int ret;
 
-       if (!dev || !dev->dev_root)
+       if (!dev)
                return -WD_EINVAL;
 
        ret = access_attr(dev->dev_root, "isolate", F_OK);

test zip using cmd:
zip_sva_perf -b 4096 -l 1 -v -m 0
host sva and guest vsva is enabled.
host: success
guest: fail, both host and guest report errors

detail steps:

insmod zip driver in host:
insmod uacce.ko
insmod hisi_qm.ko
insmod hisi_zip.ko uacce_mode=1 vfs_num=2

start vm, using 75:00.1:
echo 0000:75:00.1 > /sys/bus/pci/drivers/hisi_zip/unbind
echo vfio-pci > /sys/devices/pci0000:74/0000:74:00.0/0000:75:00.1/driver_override
echo 19e5 a251 > /sys/bus/pci/drivers/vfio-pci/new_id

./qemu-system-aarch64
-enable-kvm
-kernel ../Image
-smp 4
-m 4G
-machine virt,kernel_irqchip=on,gic-version=3,iommu=smmuv3
-device pcie-root-port,port=0x8,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x1
-net none
-device virtio-net-pci,netdev=net0,mac="52:54:11:12:34:55"
-netdev type=tap,id=net0,script=no,downscript=no,ifname=tap0
-device vfio-pci,host=75:00.1,bus=pci.1,addr=0x0,id=acc2
-initrd ../v688fs-guest
-cpu host
-nographic
-append "rdinit=init console=ttyAMA0 earlycon=pl011,0x9000000 ip=192.168.100.3"

insmod zip driver in guest:
insmod /uacce.ko
insmod /hisi_qm.ko
insmod /hisi_zip.ko uacce_mode=1

test zip using cmd:
zip_sva_perf -b 4096 -l 1 -v -m 0

[ 262.127989] hisi_zip 0000:75:00.0: qm_db_random_invalid [error status=0x1000] found
[ 262.127989] hisi_zip 0000:75:00.0: qm_db_random_invalid [error status=0x1000] found

guest zip driver(hisi_zip 0000:01:00.0 ) reports errors and ras error:

[ 23.974799] hisi_zip 0000:01:00.0: uacce release queue's qp_id = 0, BD sqe head = 0, tail = 0

NOTICE: RasIntBase = 0x140090000,ErrMapping = 0x29
NOTICE: [RasEriInterrupt]:[140L]NodeTYP0Status = 0x0
NOTICE: [RasEriInterrupt]:[156L]NodeTYP1Status = 0x1
NOTICE: [NimbusZipNodeType1]:[2254L]This is zip, Base = 0x141000000

NOTICE: [NimbusZipHandle]:[2188L]QM_PF_ABNORMAL_INT_SOURCE = 0x40

NOTICE: [NimbusZipHandle]:[2189L]ZIP_CORE_INT_SOURCE = 0x0

NOTICE: [PrintSecurityType]:[349L] SecurityType is RECOVERABLE!

NOTICE: [RasErrorDataPcieOemProcessor]:[1400L]BDF[0x75:0x0:0x0]

NOTICE: [HestGhesV2SetGenericErrorData]:[121L]
NOTICE: [HestGhesV2SetGenericErrorData]:[171L] Fill in HEST TABLE ,AckRegister=44010000
NOTICE: [ErrorBlockUpdateStatusStructure]:[38L] memset_s() run failed!
NOTICE: [HestNotifiedOS]:[18L]
NOTICE: [RasEriInterrupt]:[172L]NodeTYP2Status = 0x0
ERROR: [FiqInterruptHandler]:[115L] GicInterrupt1 : 0x1d3!!!

[ 1129.637064] {1}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 0
[ 1129.637064] {1}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 0
[ 1129.656343] {1}[Hardware Error]: event severity: recoverable
[ 1129.656343] {1}[Hardware Error]: event severity: recoverable
[ 1129.667667] {1}[Hardware Error]: Error 0, type: recoverable
[ 1129.667667] {1}[Hardware Error]: Error 0, type: recoverable
[ 1129.678991] {1}[Hardware Error]: section_type: PCIe error
[ 1129.678991] {1}[Hardware Error]: section_type: PCIe error
[ 1129.690139] {1}[Hardware Error]: version: 4.0
[ 1129.690139] {1}[Hardware Error]: version: 4.0
[ 1129.699198] {1}[Hardware Error]: command: 0x0006, status: 0x0010
[ 1129.699198] {1}[Hardware Error]: command: 0x0006, status: 0x0010
[ 1129.711566] {1}[Hardware Error]: device_id: 0000:75:00.0
[ 1129.711566] {1}[Hardware Error]: device_id: 0000:75:00.0
[ 1129.722540] {1}[Hardware Error]: slot: 0
[ 1129.722540] {1}[Hardware Error]: slot: 0
[ 1129.730728] {1}[Hardware Error]: secondary_bus: 0x00
[ 1129.730728] {1}[Hardware Error]: secondary_bus: 0x00
[ 1129.741006] {1}[Hardware Error]: vendor_id: 0x19e5, device_id: 0xa250
[ 1129.741006] {1}[Hardware Error]: vendor_id: 0x19e5, device_id: 0xa250
[ 1129.754244] {1}[Hardware Error]: class_code: 120000
[ 1129.754244] {1}[Hardware Error]: class_code: 120000
[ 30.004443] hisi_zip 0000:01:00.0: QM mailbox operation timeout!

[ 30.004875] hisi_zip 0000:01:00.0: Failed to dump sqc!

[ 30.005242] hisi_zip 0000:01:0[0.0: Failed to drain out data for stopping!

[ 30.005853] hisi_zip 0000:01:00.0: uacce release queue's qp_id = 1, BD sqe head = 0, tail = 0

[ 30.007466] hisi_zip 0000:01:00.0: QM mailbox is busy to start!

[ 30.007880] hisi_zip 0000:01:00.0: Failed to dump sqc!

[ 30.008236] hisi_zip 0000:01:00.0: Failed to drain out data for stopping!

[ 30.008715] hisi_zip 0000:01:00.0: uacce release queue's qp_id = 2, BD sqe head = 0, tail = 0

[ 30.010319] hisi_zip 0000:01:00.0: QM mailbox is busy to start!

[ 30.010733] hisi_zip 0000:01:00.0: Failed to dump sqc!

[ 30.011093] hisi_zip 0000:01:00.0: Failed to drain out data for stopping!

[ 30.011563] hisi_zip 0000:01:00.0: uacce release queue's qp_id = 3, BD sqe head = 0, tail = 0

[ 30.013155] hisi_zip 0000:01:00.0: QM mailbox is busy to start!

[ 30.013639] hisi_zip 0000:01:00.0: Failed to dump sqc!

[ 30.013998] hisi_zip 0000:01:00.0: Failed to drain out data for stopping!

https://github.com/Linaro/linux-kernel-uadk/tree/uacce-devel-5.12
make defconfig
make -j4

linaro@ubuntu:$ zip_sva_perf
Couldn't open file /sys/kernel/debug/tracing/events/iommu/dev_fault/id
IOPF statistic unavailable
NOTICE: RasIntBase = 0x140090000,ErrMapping = 0x29
NOTICE: [RasEriInterrupt]:[141L]NodeTYP0Status = 0x0
NOTICE: [RasEriInterrupt]:[157L]NodeTYP1Status = 0x1
NOTICE: [NimbusZipNodeType1]:[2593L]This is zip, Basbad status(ctx_st=0x0, s=0x41, t=3)
Compress bz=512000 nb=1×10, speed=184.3 MB/s (±0.0% N=1) overall=161.7 MB/s (±0.0%)
e = 0x141000000
NOTICE: [NimbusZipHandle]:[2542L]QM_PF_ABNORMAL_INT_SOURCE = 0x0
NOTICE: [NimbusZipHandle]:[2543L]ZIP_CORE_INT_SOURCE = 0xc
NOTICE: [PrintSecurityType]:[389L] SecurityType is RECOVERABLE!
NOTICE: [RasErrorDataPcieOemProcessor]:[1748L]BDF[0x75:0x0:0x0]
NOTICE: [HestGhesV2SetGenericErrorData]:[188L] Fill in HEST TABLE ,AckRegister=44010000
NOTICE: [Hestlinaro@ubuntu:$ NotifiedOS]:[37L]
NOTICE: [RasEriInterrupt]:[173L]NodeTYP2Status = 0x0
[ 849.220545] {7}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 0
[ 849.228776] {7}[Hardware Error]: event severity: recoverable
[ 849.234409] {7}[Hardware Error]: Error 0, type: recoverable
[ 849.240043] {7}[Hardware Error]: section_type: PCIe error
[ 849.245589] {7}[Hardware Error]: version: 4.0
[ 849.250097] {7}[Hardware Error]: command: 0x0006, status: 0x0010
[ 849.256248] {7}[Hardware Error]: device_id: 0000:75:00.0
[ 849.261707] {7}[Hardware Error]: slot: 0
[ 849.265784] {7}[Hardware Error]: secondary_bus: 0x00
[ 849.270897] {7}[Hardware Error]: vendor_id: 0x19e5, device_id: 0xa250
[ 849.277479] {7}[Hardware Error]: class_code: 120000

linaro@ubuntu:~$ test_hisi_sec
this is a hisi sec test.
Test sec Cipher parameter default, alg:ecb(aes), set_times:10,set_pktlen:16 bytes, set_keylen:128 bit.
hisi sec init Kunpeng920!
req src--------->:
\0x00\0x11\0x22\0x33\0x44\0x55\0x66\0x77
\0x88\0x99\0xaa\0xbb\0xcc\0xdd\0xee\0xff

cipher req key--------->:
SEC BD cipher fail! done=0x1, etype=0xd
SEC BD cipher fail! done=0x1, etype=0xd
SEC BD cipher fail! done=0x1, etype=0xd
SEC BD cipher fail! done=0x1, etype=0xd
SEC BD cipher fail! done=0x1, etype=0xd
SEC BD cipher fail! done=0x1, etype=0xd
SEC BD cipher fail! done=0x1, etype=0xd
SEC BD cipher fail! done=0x1, etype=0xd
SEC BD cipher fail! done=0x1, etype=0xd
SEC BD cipher fail! done=0x1, etype=0xd
time_used:275 us, send task num:10
Pro-5277, thread_id-5277, speed:36363.637 ops, Perf: 568 KB/s
Test cipher sync function: output dst-->
\0x00\0x00\0x00\0x00\0x00\0x00\0x00\0x00
\0x00\0x00\0x00\0x00\0x00\0x00\0x00\0x00

linaro@ubuntu:~$ test_hisi_hpre

test start run rsa-gen :
key_bits = 1024
trd_mode = sync
trd_num = 2
core_mask = [0x0][0x0]
msg_type = 0
msg_len = 0xffffffff
id_len = 0xffffffff
hash_type = 1
rand_type = 1
rand_len = 0xffffffff
data_from = 0
perf_test = 0
check = 0
cycles = 100
seconds = 0
HPRE do rsa fail!done=0x2, etype=0x4
Proc-5278, T-5279:hpre rsa-gen 0th status=0 fail!
HPRE do rsa fail!done=0x2, etype=0x4
Proc-5278, T-5280:hpre rsa-gen 0th status=0 fail!
<< test finish!
NOTICE: RasIntBase = 0x2010a0000,ErrMapping = 0x29
NOTICE: [RasEriInterrupt]:[141L]NodeTYP0Status = 0x0
NOTICE: [RasEriInterrupt]:[157L]NodeTYP1Status = 0x2
NOTICE: [NimbusHpreNodeType1]:[2405L]This is hpre, Base = 0x208000000
NOTICE: [NimbusHpreHandle]:[2348L] HpreHacIntSt = 0xe408
NOTICE: [NimbusHpreHandle]:[2349L] HpreQmIntStatus = 0x0
NOTICE: [PrintSecurityType]:[389L] SecurityType is RECOVERABLE!
NOTICE: [RasElinaro@ubuntu:~$ rrorDataPcieOemProcessor]:[1748L]BDF[0x79:0x0:0x0]
NOTICE: [HestGhesV2SetGenericErrorData]:[188L] Fill in HEST TABLE ,AckRegister=44010000
NOTICE: [HestNotifiedOS]:[37L]
NOTICE: [RasEriInterrupt]:[173L]NodeTYP2Status = 0x0
[ 1090.772613] {8}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 0
[ 1090.780842] {8}[Hardware Error]: event severity: recoverable
[ 1090.786474] {8}[Hardware Error]: Error 0, type: recoverable
[ 1090.792107] {8}[Hardware Error]: section_type: PCIe error
[ 1090.797653] {8}[Hardware Error]: version: 4.0
[ 1090.802161] {8}[Hardware Error]: command: 0x0006, status: 0x0010
[ 1090.808311] {8}[Hardware Error]: device_id: 0000:79:00.0
[ 1090.813771] {8}[Hardware Error]: slot: 0
[ 1090.817848] {8}[Hardware Error]: secondary_bus: 0x00
[ 1090.822961] {8}[Hardware Error]: vendor_id: 0x19e5, device_id: 0xa258
[ 1090.829543] {8}[Hardware Error]: class_code: 100000

if you try to build using make -jN (where N is >= 8), the build fails:
/usr/bin/ld: cannot find -lwd_crypto

wd_util.c
num = fmin(config_numa->async_poll_num, config_numa->async_ctx_num);

depend on -lm
core lib should not depend too much library

It could be observed in "master" branch.

UADK multi-threaded performance cannot be increased to the hardware speed limit
8 threads：
algname: length: perf: iops: CPU_rate:
aes-128-cbc 1024Bytes 1254809.0KB/s 1225.2Kops 785.40%

16 threads：
algname: length: perf: iops: CPU_rate:
aes-128-cbc 1024Bytes 1420551.0KB/s 1387.2Kops 1564.60%

32 threads：
algname: length: perf: iops: CPU_rate:
aes-128-cbc 1024Bytes 539262.2KB/s 526.6Kops 3138.20%

In contrast, in the same synchronization mode, warpdrive can reach the hardware speed limit by adding threads:
8 threads：
algname: length: perf: iops: CPU_rate:
aes-128-cbc 1024Bytes 2970516.8KB/s 2900.8Kops 799.20%

16 threads：
algname: length: perf: iops: CPU_rate:
aes-128-cbc 1024Bytes 5664155.0KB/s 5531.4Kops 1597.80%

32 threads：
algname: length: perf: iops: CPU_rate:
aes-128-cbc 1024Bytes 8293766.6KB/s 8099.2Kops 3194.80%

test aes-cbc, use sync mode. Treads == queues. packet size = 1K. The lengths of other packets are similar. maybe worse.
NO-SVA SVA
Threads = 1, 502.4 MB/s 555.7 MB/s
Threads = 2, 989.3 MB/s 1067.6 MB/s
Threads = 4, 1963.9 MB/s 1910.8 MB/s
Threads = 8, 13604.3 MB/s 3189.9 MB/s
Threads = 16, 6571.5 MB/s 1911.6 MB/s
Threads = 32, 9403.3 MB/s 966.9 MB/s

DVM: "Distributed Virtual Memory", an Arm interconnect
protocol. When sharing a process address space, TLB invalidations from the
CPU are broadcasted to the SMMU, so we don't have to send commands through
the SMMU queue to invalidate IOTLBs

guest: fail
estuary:/mnt$ ./test_sva_bind -s 0x1000 -b 0x1000 -k t
TLB test failed, broken invalidate! VA=0xffff9d3f9000-0xffff9d3fafff

host: succeed
sudo ./test_sva_bind -s 0x1000 -b 0x1000 -k t bad status (s=66, t=3, expected)
wd_sched_work: -14
TLB test success
SUCCESS

board: 201

kernel: https://github.com/Linaro/linux-kernel-uadk/tree/uacce-devel-5.11
make defconfig
make menuconfig
Device Drivers --->
-- IOMMU Hardware Support --->
[] Shared Virtual Addressing support for the ARM SMMUv3
Misc devices --->
<> Accelerator Framework for User Land
-- Cryptographic API --->
[] Hardware crypto devices --->
<> Support for HiSilicon SEC2 crypto block cipher accelerator
<> Support for HiSilicon ZIP accelerator
<> Support for HISI HPRE accelerator
[*] Tracers --->

uadk: master version: update to 2.2.14

openssl-uadk: master: a96fbbf ras: work with nginx via both async and sync mode
https://github.com/Linaro/openssl-uadk
https://github.com/Linaro/openssl-uadk/blob/master/INSTALL.md

Test:
for((i=0; i<10; i++))
do
echo $i
openssl speed -elapsed -engine uadk -async_jobs 36 rsa2048

done

Hung since poll->recv can not get new data

Sometime report ras error first, then hung.
NOTICE: [RasEriInterrupt]:[141L]NodeTYP0Status = 0x0
NOTICE: [RasEriInterrupt]:[157L]NodeTYP1Status = 0x2
NOTICE: [NimbusHpreNodeType1]:[2405L]This is hpre, Base = 0x208000000
NOTICE: [NimbusHpreHandle]:[2348L] HpreHacIntSt = 0x33e7c8
NOTICE: [NimbusHpreHandle]:[2349L] HpreQmIntStatus = 0x22
NOTICE: [PrintSecurityType]:[389L] SecurityType is RECOVERABLE!
NOTICE: [RasErrorDataPcieOemProcessor]:[1748L]BDF[0x79:0x0:0x0]
NOTICE: [HestGhesV2SetGenericErrorData]:[188L] Fill in HEST TABLE ,AckRegister=44010000
NOTICE: [HestNotifiedOS]:[37L]
NOTICE: [RasEriInterrupt]:[173L]NodeTYP2Status = 0x0
[ 496.016498] {2}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 0
[ 496.024727] {2}[Hardware Error]: event severity: recoverable
[ 496.030361] {2}[Hardware Error]: Error 0, type: recoverable
[ 496.035994] {2}[Hardware Error]: section_type: PCIe error
[ 496.041539] {2}[Hardware Error]: version: 4.0
[ 496.046047] {2}[Hardware Error]: command: 0x0006, status: 0x0010
[ 496.052199] {2}[Hardware Error]: device_id: 0000:79:00.0
[ 496.057659] {2}[Hardware Error]: slot: 0
[ 496.061735] {2}[Hardware Error]: secondary_bus: 0x00
[ 496.066849] {2}[Hardware Error]: vendor_id: 0x19e5, device_id: 0xa258
[ 496.073431] {2}[Hardware Error]: class_code: 100000

root@ubuntu:/sys/kernel/debug/hisi_hpre/0000:79:00.0/qm# cat regs
QM_ECC_1BIT_CNT = 0x00000000
QM_ECC_MBIT_CNT = 0x00000000
QM_DFX_MB_CNT = 0x0000248e
QM_DFX_DB_CNT = 0x2ce1cada
QM_DFX_SQE_CNT = 0x08971cfc
QM_DFX_CQE_CNT = 0x08971cfc
QM_DFX_SEND_SQE_TO_ACC_CNT = 0x1670ef9d
QM_DFX_WB_SQE_FROM_ACC_CNT = 0x00000000
QM_DFX_ACC_FINISH_CNT = 0x1670ef9b
QM_DFX_CQE_ERR_CNT = 0x00000206
QM_DFX_FUNS_ACTIVE_ST = 0x00000002
QM_ECC_1BIT_INF = 0x00010045
QM_ECC_MBIT_INF = 0x000600c6
QM_DFX_ACC_RDY_VLD0 = 0x01000000
QM_DFX_ACC_RDY_VLD1 = 0x0000ff00
QM_DFX_AXI_RDY_VLD = 0x00001802
QM_DFX_FF_ST0 = 0x00000ff4
QM_DFX_FF_ST1 = 0x0208c000
QM_DFX_FF_ST2 = 0x01d73fff
QM_DFX_FF_ST3 = 0x00000000
QM_DFX_FF_ST4 = 0x0fffffff
QM_DFX_FF_ST5 = 0x00ff00ff
QM_DFX_FF_ST6 = 0x00078081
QM_IN_IDLE_ST = 0x00000000

uadk master can not reproduce since only support -t 2
test_hisi_hpre rsa-sgn --mode=crt --perf --trd_mode=async --seconds=10 -t 2

If -t 3, report "failed to send: retry exit!"

Sanity_test.sh depends on /var/log/syslog to run correctly.

https://github.com/Linaro/uadk/blob/4cd0b3e82205767ac151835e69736c61aca4eda8/test/sanity_test.sh#L362C2-L362C24

However, on openEuler 22.04, there is no such a file. Somebody need to fix this.

UADK's scheduling for multiple queues will decrease performance after more than 16 queues.

In synchronous mode, 4threads，16 queues
algname: length: perf: iops: CPU_rate:
aes-128-cbc 1024Bytes 1026159.6KB/s 1002.0Kops 398.80%

In synchronous mode, 4threads， 24 queues
algname: length: perf: iops: CPU_rate:
aes-128-cbc 1024Bytes 1001670.6KB/s 978.0Kops 398.60%

In synchronous mode, 4threads， 32 queues
algname: length: perf: iops: CPU_rate:
aes-128-cbc 1024Bytes 821041.8KB/s 801.6Kops 386.40%

1/test/hisi_sec_test_sgl/Makefile.am:9:test_hisi_sec_sgl_LDADD=../../../.libs/libwd.la $(with_openssl_dir)/libcrypto_wd.so
v1/test/hisi_sec_test_sgl/Makefile.am:11:test_hisi_sec_sgl_LDADD=../../../.libs/libwd.so $(with_openssl_dir)/libcrypto_wd.so
v1/test/hisi_hpre_test/Makefile.am:20:test_hisi_hpre_LDADD+=$(with_openssl_dir)/libcrypto_wd.so
v1/test/hisi_hpre_test/Makefile.am:21:hpre_test_tools_LDADD+=$(with_openssl_dir)/libcrypto_wd.so
v1/test/hisi_hpre_test/Makefile.am:22:test_hisi_hpre_times_LDADD+=$(with_openssl_dir)/libcrypto_wd.so
v1/test/hisi_sec_test/Makefile.am:9:test_hisi_sec_LDADD=../../../.libs/libwd.la $(with_openssl_dir)/libcrypto_wd.so
v1/test/hisi_sec_test/Makefile.am:11:test_hisi_sec_LDADD=../../../.libs/libwd.so $(with_openssl_dir)/libcrypto_wd.so
test/hisi_hpre_test/Makefile.am:17:test_hisi_hpre_LDADD+= $(with_openssl_dir)/libcrypto_wd.so

环境：
硬件：920
OS: openeuler + kernel 6.1.x
步骤：
host执行:
1.modprobe hisi_sec2 uacce_mode=1 pf_q_num=256 ctx_q_num=2 vfs_num=0
2.echo 63 >/sys/bus/pci/devices/0000:76:00.0/sriov_numvfs
3.echo 0000:76:01.0 > /sys/bus/pci/devices/0000:76:01.0/driver/unbind
4. echo vfio-pci > /sys/bus/pci/devices/0000:76:01.0/driver_override
5. echo 0000:76:01.0 > /sys/bus/pci/probe

启动虚拟机参数：
smp 1 -device vfio-pci,host=76:01.0
kill一次虚拟机再重启。

vm执行业务：
1.modprobe hisi_sec2 uacce_mode=1
2.while true; do kp_utools u_accv2 -a cipher -n encrypt-cbc-aes -k 16-16-1000000 -s 0 -e 1||exit;done
16-16-1000000：(key长16，iv长16，输入1000000 )
业务流程：CBC-AES算法（SEC其他算法也可），单进程单线程同步模式，硬件加密，再进行硬件环回解密。

nginx can not support since cipher and digest can not re-entry.
prevented by
if (wd_cipher_setting.config.ctx_num) {
return -WD_EEXIST;

Nginx master process will init cipher and digest, doing negotiation
then fork work process, init cipher and digest, doing process, here failed.

How to setup nginx, see https://docs.qq.com/doc/DRVdaZ2tqTmdzc1hI

As the pull request #599 & #600 mentioned, there's an issue on hw_ifl_perf(). The test environment is mentioned in #599, and the workaround is mentioned in #600.

When test multiple blocks in block mode, case in "example.c" fails. It could be observed in both "master" and "hzhuang1-sva-3.3" branch.

Is io page fault for intel x86 platform supported in this project?

SM4 CTR/XTS/CBC/ECB/OFB modes, and AES ECB/CTR/XTS/CBC modes are supported by wd v1, should also all be supported by uadk

Currently, we found the version number in the ChangeLog, however, it it not convenient to DFX. For example, we can easily obtain the version number in the interface.

prepare:
dd if=/dev/urandom of=origin bs=1K count=512
zip_sva_perf --in origin --dest

test:
zip_sva_perf -d --in dest --out origin2

result:
Missing IN list file!
Fail to run hw_ifl_perf() (-22)!

I think the error is caused by the 'load_ilist'. But I can't understand the use of 'ilist'. So I need you help.
@hzhuang1