1/test/hisi_sec_test_sgl/Makefile.am:9:test_hisi_sec_sgl_LDADD=../../../.libs/libwd.la $(with_openssl_dir)/libcrypto_wd.so
v1/test/hisi_sec_test_sgl/Makefile.am:11:test_hisi_sec_sgl_LDADD=../../../.libs/libwd.so $(with_openssl_dir)/libcrypto_wd.so
v1/test/hisi_hpre_test/Makefile.am:20:test_hisi_hpre_LDADD+=$(with_openssl_dir)/libcrypto_wd.so
v1/test/hisi_hpre_test/Makefile.am:21:hpre_test_tools_LDADD+=$(with_openssl_dir)/libcrypto_wd.so
v1/test/hisi_hpre_test/Makefile.am:22:test_hisi_hpre_times_LDADD+=$(with_openssl_dir)/libcrypto_wd.so
v1/test/hisi_sec_test/Makefile.am:9:test_hisi_sec_LDADD=../../../.libs/libwd.la $(with_openssl_dir)/libcrypto_wd.so
v1/test/hisi_sec_test/Makefile.am:11:test_hisi_sec_LDADD=../../../.libs/libwd.so $(with_openssl_dir)/libcrypto_wd.so
test/hisi_hpre_test/Makefile.am:17:test_hisi_hpre_LDADD+= $(with_openssl_dir)/libcrypto_wd.so

As the pull request #599 & #600 mentioned, there's an issue on hw_ifl_perf(). The test environment is mentioned in #599, and the workaround is mentioned in #600.

Is io page fault for intel x86 platform supported in this project?

环境：
硬件：920
OS: openeuler + kernel 6.1.x
步骤：
host执行:
1.modprobe hisi_sec2 uacce_mode=1 pf_q_num=256 ctx_q_num=2 vfs_num=0
2.echo 63 >/sys/bus/pci/devices/0000:76:00.0/sriov_numvfs
3.echo 0000:76:01.0 > /sys/bus/pci/devices/0000:76:01.0/driver/unbind
4. echo vfio-pci > /sys/bus/pci/devices/0000:76:01.0/driver_override
5. echo 0000:76:01.0 > /sys/bus/pci/probe

启动虚拟机参数：
smp 1 -device vfio-pci,host=76:01.0
kill一次虚拟机再重启。

vm执行业务：
1.modprobe hisi_sec2 uacce_mode=1
2.while true; do kp_utools u_accv2 -a cipher -n encrypt-cbc-aes -k 16-16-1000000 -s 0 -e 1||exit;done
16-16-1000000：(key长16，iv长16，输入1000000 )
业务流程：CBC-AES算法（SEC其他算法也可），单进程单线程同步模式，硬件加密，再进行硬件环回解密。

Sanity_test.sh depends on /var/log/syslog to run correctly.

https://github.com/Linaro/uadk/blob/4cd0b3e82205767ac151835e69736c61aca4eda8/test/sanity_test.sh#L362C2-L362C24

However, on openEuler 22.04, there is no such a file. Somebody need to fix this.

digest stream mode must be 64-byte aligned!

DVM: "Distributed Virtual Memory", an Arm interconnect
protocol. When sharing a process address space, TLB invalidations from the
CPU are broadcasted to the SMMU, so we don't have to send commands through
the SMMU queue to invalidate IOTLBs

guest: fail
estuary:/mnt$ ./test_sva_bind -s 0x1000 -b 0x1000 -k t
TLB test failed, broken invalidate! VA=0xffff9d3f9000-0xffff9d3fafff

host: succeed
sudo ./test_sva_bind -s 0x1000 -b 0x1000 -k t bad status (s=66, t=3, expected)
wd_sched_work: -14
TLB test success
SUCCESS

$ openssl engine -t uadk
(uadk) uadk hardware engine support
[ available ]
281473677588864:error:25066067:DSO support routines:dlfcn_load:could not load the shared library:crypto/dso/dso_dlfcn.c:117:filename(libproviders.so): libproviders.so: cannot open shared object file: No such file or directory
281473677588864:error:25070067:DSO support routines:DSO_load:could not load the shared library:crypto/dso/dso_lib.c:162:
281473677588864:error:0E07506E:configuration file routines:module_load_dso:error loading dso:crypto/conf/conf_mod.c:221:module=providers, path=providers
281473677588864:error:0E076071:configuration file routines:module_run:unknown module name:crypto/conf/conf_mod.c:162:module=providers

https://github.com/Linaro/linux-kernel-warpdrive/tree/uacce-devel-5.12
make defconfig

1. buildin, cmdline:
   hisi_zip.uacce_mode=2 hisi_hpre.uacce_mode=2 hisi_sec2.uacce_mode=2

2. Or use module:
   make menuconifg
   -- Cryptographic API --->
   [] Hardware crypto devices --->
   Support for HiSilicon SEC2 crypto block cipher accelerator
   Support for HiSilicon ZIP accelerator
   Support for HISI HPRE accelerator

sudo insmod hisi_qm.ko
sudo insmod sec2/hisi_sec2.ko uacce_mode=2
sudo insmod hpre/hisi_hpre.ko uacce_mode=2

$ cat /sys/class/uacce/hisi_sec2-0/flags
128

3. uadk v1
   // verify kae for v1
   git clone https://github.com/Linaro/uadk
   // conf.sh build v1

•   --with-openssl_dir=~/work/openssl \
  

make clean
./conf.sh; make -j4
sudo make install

4. kae
   git clone https://github.com/kunpengcompute/KAE
   // install to system /usr/local
   cd KAE
   ./configure
   LIBRARY_PATH=/usr/local/lib C_INCLUDE_PATH=~/work/uadk/v1 make
   sudo make install
   openssl engine -c kae

// Or kae.sh for local build
#!/bin/bash -e

export WORKSPACE=${PWD}

cd ${WORKSPACE}/uadk-master
./cleanup.sh; ./autogen.sh;
./configure
--prefix=${WORKSPACE}/uadk-shared-v2/usr/local
--includedir=${WORKSPACE}/uadk-shared-v2/usr/local/include/uadk
--disable-static
--enable-shared
make -j$(nproc)
make install

#sudo \

LD_LIBRARY_PATH=${WORKSPACE}/uadk-shared-v2/usr/local/lib/ \

PATH=${WORKSPACE}/uadk-shared-v2/usr/local/bin:${PATH} \

C_INCLUDE_PATH=${WORKSPACE}/uadk-shared-v2/usr/local/include/ \

${WORKSPACE}/uadk-master/test/sanity_test.sh

cd ${WORKSPACE}/KAE
make clean
./configure --openssl_path=${WORKSPACE}/uadk-shared-v2
LIBRARY_PATH=${WORKSPACE}/uadk-shared-v2/usr/local/lib C_INCLUDE_PATH=${WORKSPACE}/uadk-master/v1 make
make install
export LD_LIBRARY_PATH=${WORKSPACE}/uadk-shared-v2/usr/local/lib
openssl engine -c kae

5. issues
   linaro@ubuntu:/tmp/ci$ openssl speed -elapsed -multi 1 -engine kae rsa2048
   Forked child 0
   unable to open /var/log/kae.log, Permission denied
   engine "kae" set.
   out bytes 0 error!
   [2021-05-19 06:11:22][error][alg/pkey/hpre_wd.c:372:hpre_rsa_sync()] hpre do rsa fail!
   +DTP:2048:private:rsa:10
   out bytes 0 error!
   [2021-05-19 06:11:22][error][alg/pkey/hpre_wd.c:372:hpre_rsa_sync()] hpre do rsa fail!
   out bytes 0 error!

board: 201

kernel: https://github.com/Linaro/linux-kernel-uadk/tree/uacce-devel-5.11
make defconfig
make menuconfig
Device Drivers --->
-- IOMMU Hardware Support --->
[] Shared Virtual Addressing support for the ARM SMMUv3
Misc devices --->
<> Accelerator Framework for User Land
-- Cryptographic API --->
[] Hardware crypto devices --->
<> Support for HiSilicon SEC2 crypto block cipher accelerator
<> Support for HiSilicon ZIP accelerator
<> Support for HISI HPRE accelerator
[*] Tracers --->

uadk: master version: update to 2.2.14

openssl-uadk: master: a96fbbf ras: work with nginx via both async and sync mode
https://github.com/Linaro/openssl-uadk
https://github.com/Linaro/openssl-uadk/blob/master/INSTALL.md

Test:
for((i=0; i<10; i++))
do
echo $i
openssl speed -elapsed -engine uadk -async_jobs 36 rsa2048

done

Hung since poll->recv can not get new data

Sometime report ras error first, then hung.
NOTICE: [RasEriInterrupt]:[141L]NodeTYP0Status = 0x0
NOTICE: [RasEriInterrupt]:[157L]NodeTYP1Status = 0x2
NOTICE: [NimbusHpreNodeType1]:[2405L]This is hpre, Base = 0x208000000
NOTICE: [NimbusHpreHandle]:[2348L] HpreHacIntSt = 0x33e7c8
NOTICE: [NimbusHpreHandle]:[2349L] HpreQmIntStatus = 0x22
NOTICE: [PrintSecurityType]:[389L] SecurityType is RECOVERABLE!
NOTICE: [RasErrorDataPcieOemProcessor]:[1748L]BDF[0x79:0x0:0x0]
NOTICE: [HestGhesV2SetGenericErrorData]:[188L] Fill in HEST TABLE ,AckRegister=44010000
NOTICE: [HestNotifiedOS]:[37L]
NOTICE: [RasEriInterrupt]:[173L]NodeTYP2Status = 0x0
[ 496.016498] {2}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 0
[ 496.024727] {2}[Hardware Error]: event severity: recoverable
[ 496.030361] {2}[Hardware Error]: Error 0, type: recoverable
[ 496.035994] {2}[Hardware Error]: section_type: PCIe error
[ 496.041539] {2}[Hardware Error]: version: 4.0
[ 496.046047] {2}[Hardware Error]: command: 0x0006, status: 0x0010
[ 496.052199] {2}[Hardware Error]: device_id: 0000:79:00.0
[ 496.057659] {2}[Hardware Error]: slot: 0
[ 496.061735] {2}[Hardware Error]: secondary_bus: 0x00
[ 496.066849] {2}[Hardware Error]: vendor_id: 0x19e5, device_id: 0xa258
[ 496.073431] {2}[Hardware Error]: class_code: 100000

root@ubuntu:/sys/kernel/debug/hisi_hpre/0000:79:00.0/qm# cat regs
QM_ECC_1BIT_CNT = 0x00000000
QM_ECC_MBIT_CNT = 0x00000000
QM_DFX_MB_CNT = 0x0000248e
QM_DFX_DB_CNT = 0x2ce1cada
QM_DFX_SQE_CNT = 0x08971cfc
QM_DFX_CQE_CNT = 0x08971cfc
QM_DFX_SEND_SQE_TO_ACC_CNT = 0x1670ef9d
QM_DFX_WB_SQE_FROM_ACC_CNT = 0x00000000
QM_DFX_ACC_FINISH_CNT = 0x1670ef9b
QM_DFX_CQE_ERR_CNT = 0x00000206
QM_DFX_FUNS_ACTIVE_ST = 0x00000002
QM_ECC_1BIT_INF = 0x00010045
QM_ECC_MBIT_INF = 0x000600c6
QM_DFX_ACC_RDY_VLD0 = 0x01000000
QM_DFX_ACC_RDY_VLD1 = 0x0000ff00
QM_DFX_AXI_RDY_VLD = 0x00001802
QM_DFX_FF_ST0 = 0x00000ff4
QM_DFX_FF_ST1 = 0x0208c000
QM_DFX_FF_ST2 = 0x01d73fff
QM_DFX_FF_ST3 = 0x00000000
QM_DFX_FF_ST4 = 0x0fffffff
QM_DFX_FF_ST5 = 0x00ff00ff
QM_DFX_FF_ST6 = 0x00078081
QM_IN_IDLE_ST = 0x00000000

uadk master can not reproduce since only support -t 2
test_hisi_hpre rsa-sgn --mode=crt --perf --trd_mode=async --seconds=10 -t 2

If -t 3, report "failed to send: retry exit!"

When test multiple blocks in block mode, case in "example.c" fails. It could be observed in both "master" and "hzhuang1-sva-3.3" branch.

/bin/bash ../../libtool --tag=CC --mode=link aarch64-linux-gnu-gcc -Wall -fno-strict-aliasing -I../../include -pthread -g -O2 -Wl,-rpath,'/usr/local/lib' -o test_hisi_hpre test_hisi_hpre.o ../../.libs/libwd.a ../../.libs/libwd_crypto.a ../../.libs/libhisi_hpre.a -ldl
libtool: link: aarch64-linux-gnu-gcc -Wall -fno-strict-aliasing -I../../include -pthread -g -O2 -Wl,-rpath -Wl,/usr/local/lib -o test_hisi_hpre test_hisi_hpre.o ../../.libs/libwd.a ../../.libs/libwd_crypto.a ../../.libs/libhisi_hpre.a -ldl -pthread
/usr/bin/ld: ../../.libs/libhisi_hpre.a(hisi_hpre.o): in function ecc_sign_out_transfer': /home/buildslave/uadk/drv/hisi_hpre.c:1225: undefined reference to wd_sm2_get_sign_out_params'
/usr/bin/ld: ../../.libs/libhisi_hpre.a(hisi_hpre.o): in function sm2_enc_out_transfer': /home/buildslave/uadk/drv/hisi_hpre.c:1279: undefined reference to wd_sm2_get_enc_out_params'
/usr/bin/ld: ../../.libs/libhisi_hpre.a(hisi_hpre.o): in function sm2_kg_out_transfer': /home/buildslave/uadk/drv/hisi_hpre.c:1259: undefined reference to wd_sm2_get_kg_out_params'
/usr/bin/ld: ../../.libs/libhisi_hpre.a(hisi_hpre.o): in function ecc_prepare_pubkey': /home/buildslave/uadk/drv/hisi_hpre.c:847: undefined reference to wd_ecc_get_pubkey_params'
/usr/bin/ld: ../../.libs/libhisi_hpre.a(hisi_hpre.o): in function ecc_prepare_prikey': /home/buildslave/uadk/drv/hisi_hpre.c:784: undefined reference to wd_ecc_get_prikey_params'
/usr/bin/ld: ../../.libs/libhisi_hpre.a(hisi_hpre.o): in function ecc_prepare_sign_in': /home/buildslave/uadk/drv/hisi_hpre.c:970: undefined reference to wd_ecc_get_prikey_params'
collect2: error: ld returned 1 exit status
make[3]: *** [Makefile:395: test_hisi_hpre] Error 1
make[3]: Leaving directory '/home/buildslave/uadk/test/hisi_hpre_test'
make[2]: *** [Makefile:352: all-recursive] Error 1
make[2]: Leaving directory '/home/buildslave/uadk/test'
make[1]: *** [Makefile:722: all-recursive] Error 1
make[1]: Leaving directory '/home/buildslave/uadk'
make: *** [Makefile:516: all] Error 2

prepare:
dd if=/dev/urandom of=origin bs=1K count=512
zip_sva_perf --in origin --dest

test:
zip_sva_perf -d --in dest --out origin2

result:
Missing IN list file!
Fail to run hw_ifl_perf() (-22)!

I think the error is caused by the 'load_ilist'. But I can't understand the use of 'ilist'. So I need you help.
@hzhuang1

Dear developers:
Thank you for your checking. In the method test_sec_cipher_async, the lock &test_sec_mutex maybe not be released correctly when the control flow goes to the else if branch and go to out, bypassing the unreleased lock.

static int test_sec_cipher_async(void *arg)
{
	...;
	pthread_mutex_lock(&test_sec_mutex);
	...;
	do {
try_do_again:
		ret = wd_do_cipher_async(h_sess, req);
		if (ret == -EBUSY) { // busy
			usleep(100);
			goto try_do_again;
		} else if (ret) {
			SEC_TST_PRT("test sec cipher send req is error!\n");
			goto out;   // bypass the unlock, leading to unreleased lock
		}
		cnt--;
	} while (cnt);
	pthread_mutex_unlock(&test_sec_mutex);
	...;
out:
	if (h_sess)
		wd_cipher_free_sess(h_sess);

	return ret;
}

Best,

UADK's scheduling for multiple queues will decrease performance after more than 16 queues.

In synchronous mode, 4threads，16 queues
algname: length: perf: iops: CPU_rate:
aes-128-cbc 1024Bytes 1026159.6KB/s 1002.0Kops 398.80%

In synchronous mode, 4threads， 24 queues
algname: length: perf: iops: CPU_rate:
aes-128-cbc 1024Bytes 1001670.6KB/s 978.0Kops 398.60%

In synchronous mode, 4threads， 32 queues
algname: length: perf: iops: CPU_rate:
aes-128-cbc 1024Bytes 821041.8KB/s 801.6Kops 386.40%

SM4 CTR/XTS/CBC/ECB/OFB modes, and AES ECB/CTR/XTS/CBC modes are supported by wd v1, should also all be supported by uadk

In guest system (openEuler), init2 fails since wd_get_usable_list failed to get usable devices(-19)!
Host system init2 works.

Guest:
cat /sys/class/uacce/hisi_sec2-0/device/numa_node
-1

uadk_tool test --m sec --digest 0 --sync --optype 0 --pktlen 16 --keylen 16 --times 1 --multi 1 --init 2
driver_type=3
HW
numa_id=-1 p=0x3cdbd9e0
p=(nil)
failed to get usable devices(-19)!

Host:
HW
numa_id=2 p=0xaaaac695a030
numa_id=0 p=0xaaaac695a440

After appended rand case in "example.c", stream mode is failed in NOSVA scenario. It could be observed in "hzhuang1-sva-3.2" branch.

if you try to build using make -jN (where N is >= 8), the build fails:
/usr/bin/ld: cannot find -lwd_crypto

https://github.com/Linaro/linux-kernel-uadk/tree/uacce-devel-5.12
make defconfig
make -j4

linaro@ubuntu:$ zip_sva_perf
Couldn't open file /sys/kernel/debug/tracing/events/iommu/dev_fault/id
IOPF statistic unavailable
NOTICE: RasIntBase = 0x140090000,ErrMapping = 0x29
NOTICE: [RasEriInterrupt]:[141L]NodeTYP0Status = 0x0
NOTICE: [RasEriInterrupt]:[157L]NodeTYP1Status = 0x1
NOTICE: [NimbusZipNodeType1]:[2593L]This is zip, Basbad status(ctx_st=0x0, s=0x41, t=3)
Compress bz=512000 nb=1×10, speed=184.3 MB/s (±0.0% N=1) overall=161.7 MB/s (±0.0%)
e = 0x141000000
NOTICE: [NimbusZipHandle]:[2542L]QM_PF_ABNORMAL_INT_SOURCE = 0x0
NOTICE: [NimbusZipHandle]:[2543L]ZIP_CORE_INT_SOURCE = 0xc
NOTICE: [PrintSecurityType]:[389L] SecurityType is RECOVERABLE!
NOTICE: [RasErrorDataPcieOemProcessor]:[1748L]BDF[0x75:0x0:0x0]
NOTICE: [HestGhesV2SetGenericErrorData]:[188L] Fill in HEST TABLE ,AckRegister=44010000
NOTICE: [Hestlinaro@ubuntu:$ NotifiedOS]:[37L]
NOTICE: [RasEriInterrupt]:[173L]NodeTYP2Status = 0x0
[ 849.220545] {7}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 0
[ 849.228776] {7}[Hardware Error]: event severity: recoverable
[ 849.234409] {7}[Hardware Error]: Error 0, type: recoverable
[ 849.240043] {7}[Hardware Error]: section_type: PCIe error
[ 849.245589] {7}[Hardware Error]: version: 4.0
[ 849.250097] {7}[Hardware Error]: command: 0x0006, status: 0x0010
[ 849.256248] {7}[Hardware Error]: device_id: 0000:75:00.0
[ 849.261707] {7}[Hardware Error]: slot: 0
[ 849.265784] {7}[Hardware Error]: secondary_bus: 0x00
[ 849.270897] {7}[Hardware Error]: vendor_id: 0x19e5, device_id: 0xa250
[ 849.277479] {7}[Hardware Error]: class_code: 120000

linaro@ubuntu:~$ test_hisi_sec
this is a hisi sec test.
Test sec Cipher parameter default, alg:ecb(aes), set_times:10,set_pktlen:16 bytes, set_keylen:128 bit.
hisi sec init Kunpeng920!
req src--------->:
\0x00\0x11\0x22\0x33\0x44\0x55\0x66\0x77
\0x88\0x99\0xaa\0xbb\0xcc\0xdd\0xee\0xff

cipher req key--------->:
SEC BD cipher fail! done=0x1, etype=0xd
SEC BD cipher fail! done=0x1, etype=0xd
SEC BD cipher fail! done=0x1, etype=0xd
SEC BD cipher fail! done=0x1, etype=0xd
SEC BD cipher fail! done=0x1, etype=0xd
SEC BD cipher fail! done=0x1, etype=0xd
SEC BD cipher fail! done=0x1, etype=0xd
SEC BD cipher fail! done=0x1, etype=0xd
SEC BD cipher fail! done=0x1, etype=0xd
SEC BD cipher fail! done=0x1, etype=0xd
time_used:275 us, send task num:10
Pro-5277, thread_id-5277, speed:36363.637 ops, Perf: 568 KB/s
Test cipher sync function: output dst-->
\0x00\0x00\0x00\0x00\0x00\0x00\0x00\0x00
\0x00\0x00\0x00\0x00\0x00\0x00\0x00\0x00

linaro@ubuntu:~$ test_hisi_hpre

test start run rsa-gen :
key_bits = 1024
trd_mode = sync
trd_num = 2
core_mask = [0x0][0x0]
msg_type = 0
msg_len = 0xffffffff
id_len = 0xffffffff
hash_type = 1
rand_type = 1
rand_len = 0xffffffff
data_from = 0
perf_test = 0
check = 0
cycles = 100
seconds = 0
HPRE do rsa fail!done=0x2, etype=0x4
Proc-5278, T-5279:hpre rsa-gen 0th status=0 fail!
HPRE do rsa fail!done=0x2, etype=0x4
Proc-5278, T-5280:hpre rsa-gen 0th status=0 fail!
<< test finish!
NOTICE: RasIntBase = 0x2010a0000,ErrMapping = 0x29
NOTICE: [RasEriInterrupt]:[141L]NodeTYP0Status = 0x0
NOTICE: [RasEriInterrupt]:[157L]NodeTYP1Status = 0x2
NOTICE: [NimbusHpreNodeType1]:[2405L]This is hpre, Base = 0x208000000
NOTICE: [NimbusHpreHandle]:[2348L] HpreHacIntSt = 0xe408
NOTICE: [NimbusHpreHandle]:[2349L] HpreQmIntStatus = 0x0
NOTICE: [PrintSecurityType]:[389L] SecurityType is RECOVERABLE!
NOTICE: [RasElinaro@ubuntu:~$ rrorDataPcieOemProcessor]:[1748L]BDF[0x79:0x0:0x0]
NOTICE: [HestGhesV2SetGenericErrorData]:[188L] Fill in HEST TABLE ,AckRegister=44010000
NOTICE: [HestNotifiedOS]:[37L]
NOTICE: [RasEriInterrupt]:[173L]NodeTYP2Status = 0x0
[ 1090.772613] {8}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 0
[ 1090.780842] {8}[Hardware Error]: event severity: recoverable
[ 1090.786474] {8}[Hardware Error]: Error 0, type: recoverable
[ 1090.792107] {8}[Hardware Error]: section_type: PCIe error
[ 1090.797653] {8}[Hardware Error]: version: 4.0
[ 1090.802161] {8}[Hardware Error]: command: 0x0006, status: 0x0010
[ 1090.808311] {8}[Hardware Error]: device_id: 0000:79:00.0
[ 1090.813771] {8}[Hardware Error]: slot: 0
[ 1090.817848] {8}[Hardware Error]: secondary_bus: 0x00
[ 1090.822961] {8}[Hardware Error]: vendor_id: 0x19e5, device_id: 0xa258
[ 1090.829543] {8}[Hardware Error]: class_code: 100000

nginx can not support since cipher and digest can not re-entry.
prevented by
if (wd_cipher_setting.config.ctx_num) {
return -WD_EEXIST;

Nginx master process will init cipher and digest, doing negotiation
then fork work process, init cipher and digest, doing process, here failed.

How to setup nginx, see https://docs.qq.com/doc/DRVdaZ2tqTmdzc1hI

Currently, we found the version number in the ChangeLog, however, it it not convenient to DFX. For example, we can easily obtain the version number in the interface.

https://github.com/Linaro/uadk/tree/nginx-fork

test_hisi_sec --digest 0 --optype 0 --pktlen 16 --keylen 16 --times 1 --sync --multi 1
SEC BD digest fail! done=0x1, etype=0x11

In the middle of sec_digest_sync_once, fork a process start sec_digest_sync_once again,
report error: SEC BD digest fail! done=0x1, etype=0x11

GCC 12 emits this error:

wd.c: In function 'wd_is_isolate':
wd.c:193:21: error: the comparison will always evaluate as 'true' for the address of 'dev_root' will never be NULL [-Werror=address]
  193 |         if (!dev || !dev->dev_root)
      |                     ^
In file included from wd.c:21:
./include/wd.h:131:14: note: 'dev_root' declared here
  131 |         char dev_root[PATH_STR_SIZE];
      |              ^~~~~~~~

dev_root being an initialized array, it cannot be NULL.
It should be fixed with this patch:

--- a/wd.c
+++ b/wd.c
@@ -190,7 +190,7 @@ int wd_is_isolate(struct uacce_dev *dev)
        int value = 0;
        int ret;
 
-       if (!dev || !dev->dev_root)
+       if (!dev)
                return -WD_EINVAL;
 
        ret = access_attr(dev->dev_root, "isolate", F_OK);

https://github.com/Linaro/linux-kernel-warpdrive/tree/uacce-devel-5.12
make defconfig
make menuconifg
-- Cryptographic API --->
[] Hardware crypto devices --->
Support for HiSilicon SEC2 crypto block cipher accelerator
Support for HiSilicon ZIP accelerator
Support for HISI HPRE accelerator

sudo insmod hisi_qm.ko
sudo insmod hisi_sec2.ko uacce_mode=2

$ cat /sys/class/uacce/hisi_sec2-0/flags
128

./cleanup.sh;./autogen.sh;
./configure
--host aarch64-linux-gnu
--target aarch64-linux-gnu
--includedir=/usr/local/include/uadk
--with-openssl_dir=~/work/openssl
--enable-static --with-static_drv --disable-shared

make

$ ./v1/test/hisi_sec_test/test_hisi_sec
Segmentation fault (core dumped)

test aes-cbc, use sync mode. Treads == queues. packet size = 1K. The lengths of other packets are similar. maybe worse.
NO-SVA SVA
Threads = 1, 502.4 MB/s 555.7 MB/s
Threads = 2, 989.3 MB/s 1067.6 MB/s
Threads = 4, 1963.9 MB/s 1910.8 MB/s
Threads = 8, 13604.3 MB/s 3189.9 MB/s
Threads = 16, 6571.5 MB/s 1911.6 MB/s
Threads = 32, 9403.3 MB/s 966.9 MB/s

It could be observed in "master" branch.

test zip using cmd:
zip_sva_perf -b 4096 -l 1 -v -m 0
host sva and guest vsva is enabled.
host: success
guest: fail, both host and guest report errors

detail steps:

insmod zip driver in host:
insmod uacce.ko
insmod hisi_qm.ko
insmod hisi_zip.ko uacce_mode=1 vfs_num=2

start vm, using 75:00.1:
echo 0000:75:00.1 > /sys/bus/pci/drivers/hisi_zip/unbind
echo vfio-pci > /sys/devices/pci0000:74/0000:74:00.0/0000:75:00.1/driver_override
echo 19e5 a251 > /sys/bus/pci/drivers/vfio-pci/new_id

./qemu-system-aarch64
-enable-kvm
-kernel ../Image
-smp 4
-m 4G
-machine virt,kernel_irqchip=on,gic-version=3,iommu=smmuv3
-device pcie-root-port,port=0x8,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x1
-net none
-device virtio-net-pci,netdev=net0,mac="52:54:11:12:34:55"
-netdev type=tap,id=net0,script=no,downscript=no,ifname=tap0
-device vfio-pci,host=75:00.1,bus=pci.1,addr=0x0,id=acc2
-initrd ../v688fs-guest
-cpu host
-nographic
-append "rdinit=init console=ttyAMA0 earlycon=pl011,0x9000000 ip=192.168.100.3"

insmod zip driver in guest:
insmod /uacce.ko
insmod /hisi_qm.ko
insmod /hisi_zip.ko uacce_mode=1

test zip using cmd:
zip_sva_perf -b 4096 -l 1 -v -m 0

[ 262.127989] hisi_zip 0000:75:00.0: qm_db_random_invalid [error status=0x1000] found
[ 262.127989] hisi_zip 0000:75:00.0: qm_db_random_invalid [error status=0x1000] found

guest zip driver(hisi_zip 0000:01:00.0 ) reports errors and ras error:

[ 23.974799] hisi_zip 0000:01:00.0: uacce release queue's qp_id = 0, BD sqe head = 0, tail = 0

NOTICE: RasIntBase = 0x140090000,ErrMapping = 0x29
NOTICE: [RasEriInterrupt]:[140L]NodeTYP0Status = 0x0
NOTICE: [RasEriInterrupt]:[156L]NodeTYP1Status = 0x1
NOTICE: [NimbusZipNodeType1]:[2254L]This is zip, Base = 0x141000000

NOTICE: [NimbusZipHandle]:[2188L]QM_PF_ABNORMAL_INT_SOURCE = 0x40

NOTICE: [NimbusZipHandle]:[2189L]ZIP_CORE_INT_SOURCE = 0x0

NOTICE: [PrintSecurityType]:[349L] SecurityType is RECOVERABLE!

NOTICE: [RasErrorDataPcieOemProcessor]:[1400L]BDF[0x75:0x0:0x0]

NOTICE: [HestGhesV2SetGenericErrorData]:[121L]
NOTICE: [HestGhesV2SetGenericErrorData]:[171L] Fill in HEST TABLE ,AckRegister=44010000
NOTICE: [ErrorBlockUpdateStatusStructure]:[38L] memset_s() run failed!
NOTICE: [HestNotifiedOS]:[18L]
NOTICE: [RasEriInterrupt]:[172L]NodeTYP2Status = 0x0
ERROR: [FiqInterruptHandler]:[115L] GicInterrupt1 : 0x1d3!!!

[ 1129.637064] {1}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 0
[ 1129.637064] {1}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 0
[ 1129.656343] {1}[Hardware Error]: event severity: recoverable
[ 1129.656343] {1}[Hardware Error]: event severity: recoverable
[ 1129.667667] {1}[Hardware Error]: Error 0, type: recoverable
[ 1129.667667] {1}[Hardware Error]: Error 0, type: recoverable
[ 1129.678991] {1}[Hardware Error]: section_type: PCIe error
[ 1129.678991] {1}[Hardware Error]: section_type: PCIe error
[ 1129.690139] {1}[Hardware Error]: version: 4.0
[ 1129.690139] {1}[Hardware Error]: version: 4.0
[ 1129.699198] {1}[Hardware Error]: command: 0x0006, status: 0x0010
[ 1129.699198] {1}[Hardware Error]: command: 0x0006, status: 0x0010
[ 1129.711566] {1}[Hardware Error]: device_id: 0000:75:00.0
[ 1129.711566] {1}[Hardware Error]: device_id: 0000:75:00.0
[ 1129.722540] {1}[Hardware Error]: slot: 0
[ 1129.722540] {1}[Hardware Error]: slot: 0
[ 1129.730728] {1}[Hardware Error]: secondary_bus: 0x00
[ 1129.730728] {1}[Hardware Error]: secondary_bus: 0x00
[ 1129.741006] {1}[Hardware Error]: vendor_id: 0x19e5, device_id: 0xa250
[ 1129.741006] {1}[Hardware Error]: vendor_id: 0x19e5, device_id: 0xa250
[ 1129.754244] {1}[Hardware Error]: class_code: 120000
[ 1129.754244] {1}[Hardware Error]: class_code: 120000
[ 30.004443] hisi_zip 0000:01:00.0: QM mailbox operation timeout!

[ 30.004875] hisi_zip 0000:01:00.0: Failed to dump sqc!

[ 30.005242] hisi_zip 0000:01:0[0.0: Failed to drain out data for stopping!

[ 30.005853] hisi_zip 0000:01:00.0: uacce release queue's qp_id = 1, BD sqe head = 0, tail = 0

[ 30.007466] hisi_zip 0000:01:00.0: QM mailbox is busy to start!

[ 30.007880] hisi_zip 0000:01:00.0: Failed to dump sqc!

[ 30.008236] hisi_zip 0000:01:00.0: Failed to drain out data for stopping!

[ 30.008715] hisi_zip 0000:01:00.0: uacce release queue's qp_id = 2, BD sqe head = 0, tail = 0

[ 30.010319] hisi_zip 0000:01:00.0: QM mailbox is busy to start!

[ 30.010733] hisi_zip 0000:01:00.0: Failed to dump sqc!

[ 30.011093] hisi_zip 0000:01:00.0: Failed to drain out data for stopping!

[ 30.011563] hisi_zip 0000:01:00.0: uacce release queue's qp_id = 3, BD sqe head = 0, tail = 0

[ 30.013155] hisi_zip 0000:01:00.0: QM mailbox is busy to start!

[ 30.013639] hisi_zip 0000:01:00.0: Failed to dump sqc!

[ 30.013998] hisi_zip 0000:01:00.0: Failed to drain out data for stopping!

After numa source code make install, apt-get install libnuma-dev can not work again, both on x86 and ARM.
So numa has to install from source code

1. apt-get install libnuma-dev
   success

2,
$git clone https://github.com/numactl/numactl.git
$cd numactl
$./autogen.sh
./configure, make; sudo make install

Can remove library in /use/lib, or sudo make uninstall

3. apt-get install libnuma-dev not work again

On x86
apt-get remove libnuma-dev
apt-get install libnuma-dev

ls /usr/lib/x86_64-linux-gnu/libnuma.*
/usr/lib/x86_64-linux-gnu/libnuma.a /usr/lib/x86_64-linux-gnu/libnuma.so

Only has link libnuma.so, but no libnuma.so.1.0.0

$ ll /usr/lib/x86_64-linux-gnu/libnuma.so
lrwxrwxrwx 1 root root 16 10月 20 2017 /usr/lib/x86_64-linux-gnu/libnuma.so -> libnuma.so.1.0.0
$ ls /usr/lib/x86_64-linux-gnu/libnuma.so.1.0.0
ls: cannot access '/usr/lib/x86_64-linux-gnu/libnuma.so.1.0.0': No such file or directory

On ARM
$ sudo apt-get remove libnuma-dev
$ sudo apt-get install libnuma-dev
$ numactl
numactl: error while loading shared libraries: libnuma.so.1: cannot open shared object file: No such file or directory
$ ll /usr/lib/aarch64-linux-gnu/libnuma*
-rw-r--r-- 1 root root 77642 Sep 8 2018 /usr/lib/aarch64-linux-gnu/libnuma.a
lrwxrwxrwx 1 root root 16 Sep 8 2018 /usr/lib/aarch64-linux-gnu/libnuma.so -> libnuma.so.1.0.0
Only link, no real library

wd_util.c
num = fmin(config_numa->async_poll_num, config_numa->async_ctx_num);

depend on -lm
core lib should not depend too much library

//async
test_hisi_hpre rsa-sgn --mode=crt --perf --trd_mode=async -t 1 --key_bits=2048

//sync
test_hisi_hpre rsa-sgn --mode=crt --perf --trd_mode=sync --key_bits=2048

sync:
23:<< sync 2 thread rsa-gen crt mode 2048 key_bits at 20899.988 ops!
45:<< sync 2 thread rsa-gen crt mode 2048 key_bits at 18341.109 ops!
67:<< sync 2 thread rsa-gen crt mode 2048 key_bits at 21087.043 ops!
89:<< sync 2 thread rsa-gen crt mode 2048 key_bits at 21136.064 ops!
111:<< sync 2 thread rsa-gen crt mode 2048 key_bits at 20772.770 ops!

async:
22:<< async 1 thread rsa-gen crt mode 2048 key_bits at 3570.233 ops!
43:<< async 1 thread rsa-gen crt mode 2048 key_bits at 58572.301 ops!
64:<< async 1 thread rsa-gen crt mode 2048 key_bits at 156542.062 ops!
85:<< async 1 thread rsa-gen crt mode 2048 key_bits at 4922.516 ops!
106:<< async 1 thread rsa-gen crt mode 2048 key_bits at 27717.783 ops!

async+bind:
numactl --cpubind=0 --membind=0 test_hisi_hpre rsa-sgn --mode=crt --perf --trd_mode=async -t 1 --key_bits=2048
linaro@ubuntu:~/test$ grep -rn "async 1 thread rsa-gen crt" log.2
22:<< async 1 thread rsa-gen crt mode 2048 key_bits at 6391.080 ops!
43:<< async 1 thread rsa-gen crt mode 2048 key_bits at 10102.447 ops!
64:<< async 1 thread rsa-gen crt mode 2048 key_bits at 18771.873 ops!
85:<< async 1 thread rsa-gen crt mode 2048 key_bits at 11188.066 ops!
106:<< async 1 thread rsa-gen crt mode 2048 key_bits at 8320.621 ops!
127:<< async 1 thread rsa-gen crt mode 2048 key_bits at 19878.707 ops!
148:<< async 1 thread rsa-gen crt mode 2048 key_bits at 6042.568 ops!
169:<< async 1 thread rsa-gen crt mode 2048 key_bits at 7707.399 ops!
190:<< async 1 thread rsa-gen crt mode 2048 key_bits at 1772.593 ops!
211:<< async 1 thread rsa-gen crt mode 2048 key_bits at 5408.329 ops!
232:<< async 1 thread rsa-gen crt mode 2048 key_bits at 4921.700 ops!
253:<< async 1 thread rsa-gen crt mode 2048 key_bits at 19461.562 ops!
274:<< async 1 thread rsa-gen crt mode 2048 key_bits at 17051.154 ops!
295:<< async 1 thread rsa-gen crt mode 2048 key_bits at 17449.238 ops!
316:<< async 1 thread rsa-gen crt mode 2048 key_bits at 4229.448 ops!
337:<< async 1 thread rsa-gen crt mode 2048 key_bits at 17965.992 ops!
358:<< async 1 thread rsa-gen crt mode 2048 key_bits at 19314.020 ops!
379:<< async 1 thread rsa-gen crt mode 2048 key_bits at 18230.029 ops!
400:<< async 1 thread rsa-gen crt mode 2048 key_bits at 4987.757 ops!
421:<< async 1 thread rsa-gen crt mode 2048 key_bits at 17950.393 ops!
442:<< async 1 thread rsa-gen crt mode 2048 key_bits at 18874.283 ops!
463:<< async 1 thread rsa-gen crt mode 2048 key_bits at 3388.029 ops!
484:<< async 1 thread rsa-gen crt mode 2048 key_bits at 120603.016 ops!
505:<< async 1 thread rsa-gen crt mode 2048 key_bits at 16211.062 ops!
526:<< async 1 thread rsa-gen crt mode 2048 key_bits at 21487.039 ops!
547:<< async 1 thread rsa-gen crt mode 2048 key_bits at 58381.984 ops!
568:<< async 1 thread rsa-gen crt mode 2048 key_bits at 10157.367 ops!
589:<< async 1 thread rsa-gen crt mode 2048 key_bits at 15453.960 ops!
610:<< async 1 thread rsa-gen crt mode 2048 key_bits at 5289.558 ops!
631:<< async 1 thread rsa-gen crt mode 2048 key_bits at 20218.229 ops!

commit b3db20a
Author: Haojian Zhuang [email protected]
Date: Wed Dec 30 20:38:35 2020 +0800

wd: replace mutex lock by spin lock in ctx structure

UADK multi-threaded performance cannot be increased to the hardware speed limit
8 threads：
algname: length: perf: iops: CPU_rate:
aes-128-cbc 1024Bytes 1254809.0KB/s 1225.2Kops 785.40%

16 threads：
algname: length: perf: iops: CPU_rate:
aes-128-cbc 1024Bytes 1420551.0KB/s 1387.2Kops 1564.60%

32 threads：
algname: length: perf: iops: CPU_rate:
aes-128-cbc 1024Bytes 539262.2KB/s 526.6Kops 3138.20%

In contrast, in the same synchronization mode, warpdrive can reach the hardware speed limit by adding threads:
8 threads：
algname: length: perf: iops: CPU_rate:
aes-128-cbc 1024Bytes 2970516.8KB/s 2900.8Kops 799.20%

16 threads：
algname: length: perf: iops: CPU_rate:
aes-128-cbc 1024Bytes 5664155.0KB/s 5531.4Kops 1597.80%

32 threads：
algname: length: perf: iops: CPU_rate:
aes-128-cbc 1024Bytes 8293766.6KB/s 8099.2Kops 3194.80%