Code Monkey home page Code Monkey logo

ocf's Introduction

Open CAS Framework

Build Status Tests Status Coverity status codecov License

Open CAS Framework (OCF) is high performance block storage caching meta-library written in C. It's entirely platform and system independent, accessing system API through user provided environment wrappers layer. OCF tightly integrates with the rest of software stack, providing flawless, high performance, low latency caching utility.

In this readme:

Documentation

OCF documentation is available on GitHub Pages. Doxygen API documentation is available here.

Source Code

Source code is available in the official OCF GitHub repository:

git clone https://github.com/Open-CAS/ocf.git
cd ocf

Deployment

OCF doesn't compile as separate library. It's designed to be included into another software stack. For this purpose OCF provides Makefile with two useful targets for deploying its source into target directories. Assuming OCFDIR is OCF directory, and SRCDIR and INCDIR are respectively your source and include directories, use following commands to deploy OCF into your project:

make -C $OCFDIR src O=$SRCDIR
make -C $OCFDIR inc O=$INCDIR

By default this will not copy OCF source files but create symbolic links to them, to avoid source duplication and allow for easy OCF code modification. If you prefer to copy OCF source files (e.g. you don't want to distribute whole OCF repository as your submodule) you can use following commands:

make -C $OCFDIR src O=$SRCDIR CMD=cp
make -C $OCFDIR inc O=$INCDIR CMD=cp

Examples

OCF is shipped with examples, which are complete, compillable and working programs, containing lot of comments that explain basics of caching. They are great starting point for everyone who wants to start working with OCF.

Examples can be found in directory example/.

Each example contains Makefile which can be used to compile it.

Unit Tests

OCF is shipped with dedicated unit test framework based on Cmocka.
To run unit tests you need to install following packages:

  • Cmake (>= 3.8.1)
  • Cmocka (>= 1.1.1)
  • ctags (>= 5.8)

To run unit tests use following command:

./tests/unit/framework/run_unit_tests.py

Build Test

OCF repository contains basic build test. It uses default POSIX environment. To run this test, use following commands:

cd tests/build/
make

Functional Tests

OCF repository contains dedicated functional test framework written in python and executed via pytest. With the use of ctypes it is possible to call, wrap ocf functions and use C compatible data types.
To run functional tests you need to install the following:

  • python3 (>=3.6.7)
  • pytest (Install with pip3 install pytest)

To run all functional tests (in compliance with the configuration file) compile using makefile located in ./tests/functional/Makefile and then use the following command:

pytest

Contributing

Feel like making OCF better? Don't hesitate to submit a pull request!
You can find more information about our contribution process here.
In case of any questions feel free to contact maintainer.

ocf's People

Contributors

amirharoush-huawei avatar andrzejjakowski avatar arutk avatar beef9999 avatar deixx avatar docentszachista avatar felipe-barajas avatar josehu07 avatar kamillepek avatar katlapinka avatar kmajzero avatar live4thee avatar loyou avatar michalwy avatar micrakow avatar minneyjohn avatar mmichal10 avatar mmkaypl avatar ostrokrzew avatar pdebski21 avatar rafalste avatar robertbaldyga avatar svelar avatar wolszews avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

ocf's Issues

Kernel warning during nhit promotion policy operations

Kernel warning during nhit promotion policy operations:

[21563.552417] WARNING: CPU: 15 PID: 0 at block/blk-core.c:3255 blk_end_bidi_request+0x68/0x70
[21563.561406] Modules linked in: cas_cache(O) cas_disk(O) xt_CHECKSUM iptable_mangle ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink bpfilter nls_utf8 rpcsec_gss_krb5 auth_rpcgss oid_registry nfsv4 nfs lockd grace fscache sunrpc intel_rapl iosf_mbi iTCO_wdt ioatdma mxm_wmi iTCO_vendor_support sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp crct10dif_pclmul crc32_pclmul ghash_clmulni_intel kvm_intel i2c_i801 lpc_ich intel_pch_thermal ipmi_si ipmi_devintf ipmi_msghandler wmi pcc_cpufreq acpi_pad ixgbe igb mdio i2c_algo_bit ptp crc32c_intel nvme i2c_core pps_core nvme_core dca xhci_pci xhci_hcd
[21563.629400] CPU: 15 PID: 0 Comm: swapper/15 Tainted: G O 4.19.56-t2.el7.twitter.x86_64 #1
[21563.639501] Hardware name: Supermicro SYS-5018D-FN4T/X10SDV-8C-TLN4F, BIOS 1.1c 10/03/2016
[21563.648390] RIP: 0010:blk_end_bidi_request+0x68/0x70
[21563.653675] Code: 43 00 89 de 48 89 ef 49 89 c5 e8 63 fe ff ff 49 8b bc 24 c0 01 00 00 4c 89 ee e8 33 38 43 00 31 d2 5b 89 d0 5d 41 5c 41 5d c3 <0f> 0b eb b0 0f 1f 40 00 0f 1f 44 00 00 48 8b 07 48 83 b8 00 01 00
[21563.673317] RSP: 0018:ffff9c95e7bc3d68 EFLAGS: 00010286
[21563.678859] RAX: 0000000000000000 RBX: ffff9c95e44892d8 RCX: 0000000000000000
[21563.686341] RDX: 0000000000001000 RSI: 0000000000000000 RDI: ffff9c95a1b417c0
[21563.693795] RBP: ffff9c95a1b417c0 R08: ffff9c95e1316a00 R09: 0000000000001000
[21563.701293] R10: ffff9c95e051ff98 R11: ffff9c95e1316450 R12: ffff9c95e3935658
[21563.708745] R13: ffff9c95e13163f8 R14: 0000000000000000 R15: 0000000000000000
[21563.716217] FS: 0000000000000000(0000) GS:ffff9c95e7bc0000(0000) knlGS:0000000000000000
[21563.724885] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[21563.730946] CR2: 00007fc2c8894630 CR3: 000000015a20a004 CR4: 00000000003606e0
[21563.738426] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[21563.745874] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[21563.753359] Call Trace:
[21563.756135]
[21563.758498] blk_end_request_all+0x1f/0x40
[21563.762919] block_dev_complete_rq+0x6f/0x90 [cas_cache]
[21563.768592] ? ocf_req_complete+0xeb/0x150 [cas_cache]
[21563.774070] ? _ocf_read_generic_miss_complete+0x75/0xd0 [cas_cache]
[21563.780744] ? ocf_submit_volume_req_cmpl+0x16/0x20 [cas_cache]
[21563.787000] ? cas_bd_io_end+0x37/0x50 [cas_cache]
[21563.792128] ? cas_bd_io_end_callback+0x5e/0xb0 [cas_cache]
[21563.798037] ? blk_update_request+0x78/0x2c0
[21563.802626] ? scsi_end_request+0x2c/0x240
[21563.807054] ? scsi_io_completion+0x88/0x610
[21563.811640] ? blk_done_softirq+0x84/0xa0
[21563.815980] ? __do_softirq+0xe2/0x2dd
[21563.820064] ? irq_exit+0xc5/0xd0
[21563.823691] ? do_IRQ+0x85/0xd0
[21563.827169] ? common_interrupt+0xf/0xf
[21563.831350]
[21563.833764] ? cpuidle_enter_state+0xa2/0x310
[21563.838468] ? cpuidle_enter_state+0x93/0x310
[21563.843163] ? do_idle+0x1b6/0x210
[21563.846885] ? cpu_startup_entry+0x5f/0x70
[21563.851339] ? start_secondary+0x182/0x1c0
[21563.855751] ? secondary_startup_64+0xa4/0xb0
[21563.860455] ---[ end trace 650fe3afc3dad17c ]---
[21563.865452] WARNING: CPU: 15 PID: 0 at block/blk-core.c:3200 blk_finish_request+0x11c/0x150
[21563.874422] Modules linked in: cas_cache(O) cas_disk(O) xt_CHECKSUM iptable_mangle ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink bpfilter nls_utf8 rpcsec_gss_krb5 auth_rpcgss oid_registry nfsv4 nfs lockd grace fscache sunrpc intel_rapl iosf_mbi iTCO_wdt ioatdma mxm_wmi iTCO_vendor_support sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp crct10dif_pclmul crc32_pclmul ghash_clmulni_intel kvm_intel i2c_i801 lpc_ich intel_pch_thermal ipmi_si ipmi_devintf ipmi_msghandler wmi pcc_cpufreq acpi_pad ixgbe igb mdio i2c_algo_bit ptp crc32c_intel nvme i2c_core pps_core nvme_core dca xhci_pci xhci_hcd
[21563.942349] CPU: 15 PID: 0 Comm: swapper/15 Tainted: G W O 4.19.56-t2.el7.twitter.x86_64 #1
[21563.952448] Hardware name: Supermicro SYS-5018D-FN4T/X10SDV-8C-TLN4F, BIOS 1.1c 10/03/2016
[21563.961327] RIP: 0010:blk_finish_request+0x11c/0x150
[21563.966608] Code: 6a 1a 00 00 48 8b 43 40 48 8d 53 40 48 39 c2 0f 84 4a ff ff ff 0f 0b 4c 89 ee 48 89 df e8 7c eb 00 00 8b 43 18 e9 1c ff ff ff <0f> 0b e9 07 ff ff ff 0f b6 43 14 83 e8 20 83 f8 03 0f 86 2c ff ff
[21563.986225] RSP: 0018:ffff9c95e7bc3d40 EFLAGS: 00010086
[21563.991768] RAX: 0000139c70051421 RBX: ffff9c95a1b417c0 RCX: 0000000000000018
[21563.999235] RDX: 000008a5cc704076 RSI: 002327a315fb43d7 RDI: ffffffff85234040
[21564.006687] RBP: ffff9c95e3935658 R08: 0000000000000001 R09: 0000000000000000
[21564.014157] R10: ffff9c95e051ea00 R11: 0000000000006500 R12: 0000000000000000
[21564.021610] R13: 0000139c70051421 R14: 0000000000000000 R15: 0000000000000000
[21564.029081] FS: 0000000000000000(0000) GS:ffff9c95e7bc0000(0000) knlGS:0000000000000000
[21564.037776] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[21564.043837] CR2: 00007fc2c8894630 CR3: 000000015a20a004 CR4: 00000000003606e0
[21564.051319] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[21564.058773] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[21564.066244] Call Trace:
[21564.069042]
[21564.071388] blk_end_bidi_request+0x4d/0x70
[21564.075884] blk_end_request_all+0x1f/0x40
[21564.080336] block_dev_complete_rq+0x6f/0x90 [cas_cache]
[21564.085990] ? ocf_req_complete+0xeb/0x150 [cas_cache]
[21564.091464] ? _ocf_read_generic_miss_complete+0x75/0xd0 [cas_cache]
[21564.098159] ? ocf_submit_volume_req_cmpl+0x16/0x20 [cas_cache]
[21564.104427] ? cas_bd_io_end+0x37/0x50 [cas_cache]
[21564.109573] ? cas_bd_io_end_callback+0x5e/0xb0 [cas_cache]
[21564.115491] ? blk_update_request+0x78/0x2c0
[21564.120092] ? scsi_end_request+0x2c/0x240
[21564.124535] ? scsi_io_completion+0x88/0x610
[21564.129142] ? blk_done_softirq+0x84/0xa0
[21564.133479] ? __do_softirq+0xe2/0x2dd
[21564.137560] ? irq_exit+0xc5/0xd0
[21564.141212] ? do_IRQ+0x85/0xd0
[21564.144673] ? common_interrupt+0xf/0xf
[21564.148818]
[21564.151271] ? cpuidle_enter_state+0xa2/0x310
[21564.155980] ? cpuidle_enter_state+0x93/0x310
[21564.160654] ? do_idle+0x1b6/0x210
[21564.164400] ? cpu_startup_entry+0x5f/0x70
[21564.168816] ? start_secondary+0x182/0x1c0
[21564.173244] ? secondary_startup_64+0xa4/0xb0
[21564.177915] ---[ end trace 650fe3afc3dad17d ]---

Metadata invalid checksum error after power failure during management operation

When power failure occurs during management operation like adding/removing core or changing I/O class configuration, it's possible that on next cache load invalid checksum error would happen.

It's possible that ocf_medatata_hash_flush_segment() step in ocf_metadata_hash_flush_sb_pipeline_props pipeline would flush metadata_segment_sb_config with checksums calculated based on new metadata content, but won't be able to flush actual segments content because of power failure. In this case on disk metadata would contain new checksums for all segments from this pipeline, but old content for one or more of them.

Loading cache with force set to true fails if cache was shutdown without flush

Description

When trying to load dirty shutdown wb cache it is possible to use force flag, which bypasses the dirty_shutdown bit check in attach and then fails the CRC check for collision segment.

Suggested solution

Reject attempt to load the cache with force flag set, or silently ignore it - it doesn't make sense in this context.

watchdog: BUG: soft lockup

We are consistently facing soft lockups in two machines that are near identical (one is v4, another is v2, one is LRDIM another RDIM); the third machine is doing ok with the very same cache device. All have a private LAN with Infiniband, Ubuntu 18, Intel 3605 1.5TB as cache;

The main diference is: the 2 machines with issue are x10qbi 4 x cpus, 1TB RAM DDR3; the one that is fine is x9dri (2 x cpus) 386 GB RAM DDR3, all on Ubuntu 18, another difference, the machine that is ok is running raid0 (4 disks x 2 TB), the 2 with issues, faced problems with RAID10 and RAID5, all adaptec raid cards (8 disks x 3 TB);

watchdog: BUG: soft lockup - CPU#51 stuck for 22s! [pacd:140354]
Modules linked in: macvlan cfg80211 ceph libceph fscache aufs overlay rdma_ucm(OE) ib_ucm(OE) ib_ipoib(OE) ib_umad(OE) esp6_offload esp6 esp4_offload esp4 xfrm_algo mlx5_fpga_
mdev vfio_iommu_type1 vfio mdev(OE) mlx4_en(OE) bonding xfs nls_iso8859_1 intel_rapl sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass ipmi_ssif i
k(OE) intel_rapl_perf mei_me lpc_ich mei shpchp ioatdma ipmi_si mac_hid ipmi_devintf ipmi_msghandler acpi_pad sch_fq_codel nf_nat_pptp nf_nat_proto_gre nf_conntrack_pptp nf_co
dma_cm(OE) iw_cm(OE) ib_cm(OE) iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi knem(OE) ip_tables x_tables autofs4
 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear mlx4_ib(OE) ib_uverbs(OE) ib
f_pclmul crc32_pclmul ghash_clmulni_intel ast hid_generic i2c_algo_bit pcbc ttm usbhid aesni_intel hid drm_kms_helper aes_x86_64 crypto_simd syscopyarea glue_helper ixgbe sysf
sys_fops ptp nvme_core devlink aacraid ahci pps_core drm mlx_compat(OE) libahci mdio wmi
CPU: 51 PID: 140354 Comm: pacd Tainted: G           OE    4.15.0-29-generic #31-Ubuntu
Hardware name: Supermicro PIO-848B-TRF4T-ST031/X10QBI, BIOS 3.2a 08/08/2019
RIP: 0010:_raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff8d09bf2c3d68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff11
RAX: 00000000ff5e4a5d RBX: ffff8d49ab049b68 RCX: ffff8d09bf2c3d98
RDX: ffff8d49ab049b70 RSI: 0000000000000202 RDI: 0000000000000202
RBP: ffff8d09bf2c3d68 R08: 000000000000000e R09: 0000000000000000
R10: ffff8d09bf2c3c70 R11: 0000000000000073 R12: 00000000ff5e4a5d
R13: 0000000000000202 R14: 0000000000000000 R15: 0000000000000000
FS:  00007ff6c3e9e700(0000) GS:ffff8d09bf2c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000152b4038 CR3: 0000003dc5766001 CR4: 00000000003606e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <IRQ>
 __wake_up_common_lock+0x8e/0xc0
 __wake_up+0x13/0x20
 rwb_wake_all+0x2c/0x60
 scale_up.part.20+0x28/0x40
 wb_timer_fn+0x21b/0x3d0
 ? blk_mq_tag_update_depth+0x110/0x110
 blk_stat_timer_fn+0x147/0x150
 call_timer_fn+0x30/0x130
 run_timer_softirq+0x3fb/0x450
 ? ktime_get+0x43/0xa0
 ? lapic_next_deadline+0x26/0x30
 __do_softirq+0xdf/0x2b2
 irq_exit+0xb6/0xc0
 smp_apic_timer_interrupt+0x71/0x130
 apic_timer_interrupt+0x84/0x90
 </IRQ>
RIP: 0010:_raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff9a6f5aa7fa68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff11
RAX: 0000000000000202 RBX: ffff9a6f5aa7fac0 RCX: ffff8c8805690000
RDX: 0000000000000000 RSI: 0000000000000202 RDI: 0000000000000202
RBP: ffff9a6f5aa7fa68 R08: 0000000000d00000 R09: ffff8d09bf663440
R10: 0000000000000000 R11: 0000000000000082 R12: ffff8d49ab049b68
R13: 0000000000000002 R14: 0000000000000060 R15: ffff8d49ab049b00
 prepare_to_wait_exclusive+0x72/0x80
 wbt_wait+0x137/0x350
 ? wait_woken+0x80/0x80
 blk_mq_make_request+0xe0/0x570
 generic_make_request+0x124/0x300
 submit_bio+0x73/0x150
 ? submit_bio+0x73/0x150
 ? xfs_setfilesize_trans_alloc.isra.13+0x3e/0x90 [xfs]
 xfs_submit_ioend+0x87/0x1c0 [xfs]
 xfs_vm_writepages+0xd1/0xf0 [xfs]
 do_writepages+0x4b/0xe0
 ? iomap_write_begin.constprop.18+0x140/0x140
 ? iomap_file_buffered_write+0x6e/0xa0
 ? iomap_write_begin.constprop.18+0x140/0x140
 ? xfs_iunlock+0xf8/0x100 [xfs]
 __filemap_fdatawrite_range+0xc1/0x100
 ? __filemap_fdatawrite_range+0xc1/0x100
 file_write_and_wait_range+0x5a/0xb0
 xfs_file_fsync+0x5f/0x230 [xfs]
 vfs_fsync_range+0x51/0xb0
 do_fsync+0x3d/0x70
 SyS_fdatasync+0x13/0x20
 do_syscall_64+0x73/0x130
 entry_SYSCALL_64_after_hwframe+0x3d/0xa2
RIP: 0033:0x7ff729d273e7
RSP: 002b:00007ff6c3d9ec50 EFLAGS: 00000293 ORIG_RAX: 000000000000004b
RAX: ffffffffffffffda RBX: 0000000000000027 RCX: 00007ff729d273e7
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000027
RBP: 00007ff6c3d9ed30 R08: 0000000000000000 R09: 000000000000002c
R10: 00007ff6bc0008d0 R11: 0000000000000293 R12: 00007ff6c3d9ece0
R13: 00007ff6bc010965 R14: 00007ff6c3d9ed60 R15: 00007ff6bc00e3f0
Code: 47 76 ff ff ff 7f 89 d0 5d c3 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f

Requests not completed when using nhit promotion policy

When using nhit promotion policy some requests are never completed (hung requests).
When using default promotion policy everything works fine.

Steps to reproduce:

  1. Start CAS (with default settings) and add core
  2. Run fio with read-only JESD219 workload (fio config below)

Fio config:

[JESD219]
ioengine=libaio
direct=1
rw=randread
norandommap
randrepeat=0
iodepth=256
numjobs=4
bssplit=512/4:1024/1:1536/1:2048/1:2560/1:3072/1:3584/1:4k/67:8k/10:16k/7:32k/3:64k/3
blockalign=4k
random_distribution=zoned:50/5:30/15:20/80
filename=${FILE}
group_reporting=1

iostat output at the end of workload:
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util cas1-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 622.00 0.00 0.00 0.00 0.00 100.00

root@localhost:~# uname -a
Linux localhost 4.4.0-28-generic #47-Ubuntu SMP Fri Jun 24 10:09:13 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
root@localhost:~# cat /etc/os-release
NAME="Ubuntu"
VERSION="16.04 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
UBUNTU_CODENAME=xenial
root@localhost:~# casadm --version
+=========================+===================+
| Name                    |      Version      |
+=========================+===================+
| CAS Cache Kernel Module | 19.09.00.00001216 |
| CAS Disk Kernel Module  | 19.09.00.00001216 |
| CAS CLI Utility         | 19.09.00.00001216 |
+=========================+===================+
root@localhost:~/open-cas-linux# git show
commit 40eaa3bd78121db778fc8dfae7b1a71072d52b10
root@localhost:~/open-cas-linux/ocf# git show
commit 4e5530851b0e7d15a4c5883c5d96c4065f0aa34f
root@localhost:~# casadm -P -i 1 -f conf
Cache Id                  1
Cache Size                477657 [4KiB Blocks] / 1.82 [GiB]
Cache Device              /dev/nvme0n1
Core Devices              1
Inactive Core Devices     0
Write Policy              wt
Eviction Policy           lru
Cleaning Policy           alru
Promotion Policy          nhit
Cache line size           4 [KiB]
Metadata Memory Footprint 42.5 [MiB]
Dirty for                 0 [s] / Cache clean
Metadata Mode             normal
Status                    Running
root@localhost:~# casadm -P -i 1 -j 1 -f conf
Core Id              1
Core Device          /dev/sdc1
Exported Object      /dev/cas1-1
Core Size            1952768 [4KiB Blocks] / 7.45 [GiB]
Dirty for            0 [s] / Cache clean
Status               Active
Seq cutoff threshold 1024 [KiB]
Seq cutoff policy    full

Fail to get a clean reboot while using more than 1 cache device

The initial start after configuring the opencas.conf file is ok; all devices are created as expected;
We are using raid0 via hardware, and the we identify the device with a consistent name across reboots;
We defined 1 raid device per cache; and we cannot get a clean reboot;
Also we tried to add core and cache devices to lvm global filter rules;
checking the journal, there are mentions that the core device was busy and will be attached as inactive; but it remains also as detached
The interesting part is that the failed initialization conducts to an emergency boot, in which we can see that some cache/core are assembled;

checking with casadm -L, shows that the same core device that failed is observed twice, once as detached, and again under its respective cache device as inactive;
using casctl stop; casctl init --force solve the issue;
also removing the detached cores, doing casctl stop, casctl init, also fix the issue;
but, casctl stop and casctl init without removing hte detached cores, prevents those to initiate correctly;

The fastest route is rename the opencas.conf file to avoid the initialization, let the machine boot, rename back, we can start the cache without any issue, but it has to be manually performed, and do not allow to map the drives at /etc/fstab

another good option is to type the root password to get into the emergency mode; casctl stop; casadm -L; casadm --remove-detached -d /dev/xxxx; casctl start;

abnornal IO when direct=1 and fdatasync=1 in fio test

I test the IO pattern for direct sync IO in fio, the writeback cache device seems to be very slow. can anyone help to explain why? and how to optimaze it?

fio configuration is below

[global]
name=fio-rand-write
filename=/dev/cas1-12:/dev/cas1-1:/dev/cas1-2:/dev/cas1-3:/dev/cas1-4:/dev/cas1-5:/dev/cas1-6:/dev/cas1-7:/dev/cas1-8:/dev/cas1-9:/dev/cas1-10:/dev/cas1-11
rw=randwrite
#rate_iops=40000
bs=8K
direct=1
fdatasync=1
numjobs=1
time_based=1
runtime=1000000000
norandommap
random_distribution=zipf:1.2

[file1]
ioengine=libaio
iodepth=32

Question on Performance of OCF

Hi there,

I have a question related to the OCF codebase performance/ scalability. (not related to cache behaviors)

Specifically, Im using some simple code to evaluate OCF's performance. The context logic is same as the example context, except:

  1. we do nothing in volume's submit_io function: the example is doing memcpy, while we directly call io->end()
  2. we use multiple threads to do ocf_core_submit_io: we change the example's main.c/perform_workload() to be a multi-threaded version
  3. miss ratio = 0%

However when we experiment with different number of threads, OCF looks like not scalable, e.g.:
1 thread: 700K requests/s
2 threads: 500K requests/s
4 threads: 360K requests/s

Im wondering:

  1. is this expected?
  2. is there any suggestion on configuring a scalable cache? (e.g. should there be some partitions?)

Thanks,
Kan

ACP Cleaning Policy for Chunk with small dirty cache lines

For ACP cleaning policy, there are two parameters which are "-w" and "-b", it means:

  • For each cycle, CAS will try to flush max "-b" cache blocks.
[-w, --wake-up <NUM>]: Period of time between awakenings of flushing thread in milliseconds. MIN: 0, MAX: 10000
(inclusive), DEFAULT: 10.
[-b, --flush-max-buffers <NUM>]: Number of dirty cache blocks to be flushed in one cleaning cycle. MIN: 1, MAX:
10000 (inclusive), DEFAULT: 128.

But seems there is another factor which may determine the flush speed of ACP:

  • The dirty cache blocks in the chunk
    Here is how the ACP works for every cycle:
  • First, choose one chunk (with most percentage of dirty blocks)
  • Then try to find max "-b" dirty cache blocks from the chunk and issue flush IO to core device
    • If the dirty cache blocks in the chunk is small (eg. smaller than "-b"), only a small number of dirty cache blocks will be flushed to core device
  • Wait for "-w" and repeat the process
void cleaning_policy_acp_perform_cleaning(ocf_cache_t cache,
		ocf_cleaner_end_t cmpl)
{
	struct acp_cleaning_policy_config *config;
	struct acp_context *acp = _acp_get_ctx_from_cache(cache);
	struct acp_state *state = &acp->state;

	acp->cmpl = cmpl;

	if (!state->in_progress) {
		/* get next chunk to clean */
		state->chunk = _acp_get_cleaning_candidate(cache);

		if (!state->chunk) {
			/* nothing co clean */
			cmpl(&cache->cleaner, ACP_BACKOFF_TIME_MS);
			return;
		}

		/* new cleaning cycle - reset state */
		state->iter = 0;
		state->in_progress = true;
	}

	ACP_DEBUG_INIT(acp);

	config = (void *)&cache->conf_meta->cleaning[ocf_cleaning_acp].data;

	if (_acp_prepare_flush_data(acp, config->flush_max_buffers))
		_acp_flush(acp);
	else
		_acp_flush_end(acp, 0);
}

Or in other words, with our current coding, here is how ACP works:

  • For each "-w" cycle, the number of dirty cache blocks flushed to core device is the min of "-b" and the dirty cache blocks in the chunks.

Metadata invalid checksum error after power failure during cache stop sequence

When power failure occurs during cache stopping, it's possible that there would be invalid metadata checksum on next cache load.

If in cache shutdown sequence power failure occurs after sb_runtime and/or part_runtime segments are flushed, but before final sb_config flush, there might be metadata checksum error for those segments on next cache load.
Checksums for those segments are verified in load sequence regardless of shutdown status - dirty or clean. They should be checked only after clean shutdown. For dirty shutdown there is no guarantee that checksums for those segments are up to date.

Lacks documents about how to manage worker threads, or best practice.

The ocf example only demonstrate the IO path by a synchronize call of submit_io and complete callback function. But it still remains unclear that how we should manager the synchronization in a real scenario. How many threads should there be? Do we need to create a new thread(coroutine) in the complete callback to achieve synchronization? Which worker thread is responsible for executing the volume-defined ocf_volume_properties.ops.submit_io interface?

The CAS itself is indeed a good example, however it requires the developers to have basic knowledge of the kernel. Otherwise you won't realize how and when each function is being called by specific kernel threads, just by reading the code.

So please add some documents in terms of worker threads management.

manual eviction

In some scenarios, we can know ahead that a high usage of the cache will be required.
Hence, in this situation, would be very interesting the capability to trigger the eviction in the same way the flush can be triggered.
With that, high performance can be achieved. We suggest a command to evict up a cache percentage defined by while casting the command.

Segmentation fault during send IO after initialization

I am getting NULL in io->volume->cache during ocf_core_validate_io() when sending reads immediately after initialization (in callback of ocf_mngt_cache_add_core_finish())

Full backtrace:

(gdb) bt
#0  0x00005555555e1d95 in ocf_core_validate_io (io=0x200027228190) at src/ocf/ocf_core.c:202
#1  0x00005555555e1ec6 in ocf_core_submit_io_mode (io=0x200027228190, cache_mode=ocf_cache_mode_none) at src/ocf/ocf_core.c:233
#2  0x00005555555e2515 in ocf_core_volume_submit_io (io=0x200027228190) at src/ocf/ocf_core.c:395
#3  0x00005555555e0a5d in ocf_volume_submit_io (io=0x200027228190) at src/ocf/ocf_volume.c:237
#4  0x00005555555b701b in ocf_core_submit_io (io=0x200027228190) at /home/vmysak/samba_share0/cas/hyb/lib/bdev/ocf/env/include/ocf/ocf_core.h:156
#5  0x00005555555b7e7c in io_submit_to_ocf (bdev_io=0x200007043c00, io=0x200027228190) at vbdev_ocf.c:576
#6  0x00005555555b8026 in io_handle (ch=0x555555dc6cd0, bdev_io=0x200007043c00) at vbdev_ocf.c:625
#7  0x00005555555b80ba in vbdev_ocf_get_buf_cb (ch=0x555555dc6cd0, bdev_io=0x200007043c00, success=true) at vbdev_ocf.c:653
#8  0x0000555555621c36 in spdk_bdev_io_get_buf (bdev_io=0x200007043c00, cb=0x5555555b8074 <vbdev_ocf_get_buf_cb>, len=32768) at bdev.c:596
#9  0x00005555555b8116 in vbdev_ocf_submit_request (ch=0x555555dc6cd0, bdev_io=0x200007043c00) at vbdev_ocf.c:664
#10 0x00005555556242e3 in _spdk_bdev_io_submit (ctx=0x200007043c00) at bdev.c:1693
#11 0x000055555562462e in spdk_bdev_io_submit (bdev_io=0x200007043c00) at bdev.c:1741
#12 0x000055555562646b in spdk_bdev_read_blocks (desc=0x555555dc6bc0, ch=0x555555dc6c00, buf=0x200027002b80, offset_blocks=0, num_blocks=64, cb=0x5555555ae364 <spdk_gpt_bdev_complete>, cb_arg=0x555555dc6af0) at bdev.c:2595
#13 0x0000555555626326 in spdk_bdev_read (desc=0x555555dc6bc0, ch=0x555555dc6c00, buf=0x200027002b80, offset=0, nbytes=32768, cb=0x5555555ae364 <spdk_gpt_bdev_complete>, cb_arg=0x555555dc6af0) at bdev.c:2563
#14 0x00005555555ae699 in vbdev_gpt_read_gpt (bdev=0x555555dc3ea0) at vbdev_gpt.c:477
#15 0x00005555555ae810 in vbdev_gpt_examine (bdev=0x555555dc3ea0) at vbdev_gpt.c:528
#16 0x0000555555629192 in spdk_bdev_start (bdev=0x555555dc3ea0) at bdev.c:3740
#17 0x00005555556291d7 in spdk_bdev_register (bdev=0x555555dc3ea0) at bdev.c:3751
#18 0x00005555555b8ce9 in register_ocf_bdev (vbdev=0x555555dc3d20) at vbdev_ocf.c:1051
#19 0x00005555555b6e94 in vbdev_ocf_mngt_continue (vbdev=0x555555dc3d20, status=0) at utils.c:128
#20 0x00005555555b8905 in add_core_cmpl (cache=0x2000142273c0, core=0x200014227b80, priv=0x555555dc3d20, error=0) at vbdev_ocf.c:913
#21 0x00005555555d7e26 in ocf_mngt_cache_add_core_finish (pipeline=0x20000fc1fa00, priv=0x20000fc1fa28, error=0) at src/ocf/mngt/ocf_mngt_core.c:525
#22 0x00005555555f8512 in _ocf_pipeline_run_step (req=0x200014628180) at src/ocf/utils/utils_pipeline.c:47
#23 0x00005555555e2f3d in ocf_io_handle (io=0x0, opaque=0x200014628180) at src/ocf/ocf_queue.c:74
#24 0x00005555555e303b in ocf_queue_run_single (q=0x20001c2001c0) at src/ocf/ocf_queue.c:94
#25 0x00005555555b84a8 in mngt_queue_poll (opaque=0x20001c2001c0) at vbdev_ocf.c:796
#26 0x0000555555657b9e in spdk_thread_poll (thread=0x555555dc2800, max_msgs=0, now=6274227700624692) at thread.c:456
#27 0x00005555556151d4 in _spdk_reactor_run (arg=0x555555dc25c0) at reactor.c:244
#28 0x00005555556154f6 in spdk_reactors_start () at reactor.c:333
#29 0x0000555555613d79 in spdk_app_start (opts=0x7fffffffe270, start_fn=0x555555564450 <spdk_startup>, arg1=0x0) at app.c:677
#30 0x00005555555645e5 in main (argc=3, argv=0x7fffffffe408) at iscsi_tgt.c:112

Possible cause:
Address of io->volume in ocf_core_validate_io() is greater by 1 byte than address of core->volume in ocf_mngt_cache_add_core_finish()

Also, removing check at #202 in ocf_core_validate_io() silenced this error and everything worked OK

Segmentation fault while executing pyocf tests

Seems to occur on low end instances - AWS EC2 t3.medium (2vCPU, 4GB RAM) in this case

Command executed: pytest --ignore=tests/security
Gdb output:
tests/basic/test_pyocf.py::test_ctx_fixture PASSED [ 0%]
tests/basic/test_pyocf.py::test_simple_wt_write [New Thread 0x7fffe6113700 (LWP 2327)]
[New Thread 0x7fffe57ec700 (LWP 2328)]
[New Thread 0x7fffe4feb700 (LWP 2329)]

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffe4feb700 (LWP 2329)]
0x00007fffea5b0605 in env_atomic64_add (i=17, a=0x19)
at /home/ec2-user/jenkins/workspace/OCF/pyocf/Execute_tests/tests/functional/pyocf/ocf/src/ocf/env/ocf_env.h:453
453 __sync_add_and_fetch(&a->counter, i);

Missing FLUSH request after metadata flush

OCF does not issue FLUSH request after flushing metadata to cache device. This could lead to lost of recent metadata state on drives that don't have PLI protection and do have write buffering enabled.

How to control the speed of flushing in cleaning-alru policy?

I wanted to use cleaning-alru policy and achieve the following goals:

  • The cleaning thread should wake up every 30s
  • Every time the cleaning thread wakes up, it should flush maximum of 40MiB data

The cache line size was 8KB so I thought the flush_max_buffers should be 40 * 1024 / 8 which is 5120
so I set the cleaning-alru policy with the following command:

casadm -X -n cleaning-alru -i 1 -w 30 -b 5120 -s 120

and then I dd-ed about 1GB data into /dev/cas1-1

But it turned out that after the cleaning thread woke up, instead of flushing 40MiB every 30 seconds, it flushed the whole 1GB data in nearly 14 seconds, so:

  1. How does this option work?
  2. How can I control the speed of flushing?

make simple example will fail if I enable the metadata cache hash debug

===if I change OCF_METADATA_HASH_DEBUG as 1.
#define OCF_METADATA_HASH_DEBUG 0

#if 1 == OCF_METADATA_HASH_DEBUG
#define OCF_DEBUG_TRACE(cache)
ocf_cache_log(cache, log_info, "[Metadata][Hash] %s\n", func)

#define OCF_DEBUG_PARAM(cache, format, ...)
ocf_cache_log(cache, log_info, "[Metadata][Hash] %s - "format"\n",
func, ##VA_ARGS)
#else

===== then I suffer below build error ==== could you please help?
===== or can you show me to how to understand and debug metadata hash map easily?
and also I see there is RB-Tree in the code, what is it for?
gcc -g -Wall -Iinclude/ -Isrc//ocf/env/ -c -o src/ocf/eviction/eviction.o src/ocf/eviction/eviction.c
gcc -g -Wall -Iinclude/ -Isrc//ocf/env/ -c -o src/ocf/eviction/lru.o src/ocf/eviction/lru.c
gcc -g -Wall -Iinclude/ -Isrc//ocf/env/ -c -o src/ocf/metadata/metadata.o src/ocf/metadata/metadata.c
gcc -g -Wall -Iinclude/ -Isrc//ocf/env/ -c -o src/ocf/metadata/metadata_collision.o src/ocf/metadata/metadata_collision.c
gcc -g -Wall -Iinclude/ -Isrc//ocf/env/ -c -o src/ocf/metadata/metadata_hash.o src/ocf/metadata/metadata_hash.c
src/ocf/metadata/metadata_hash.c:2:8: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘attribute’ before ‘-’ token

  • SPDX-License-Identifier: BSD-3-Clause-Clear
    ^
    In file included from src/ocf/metadata/../ocf_cache_priv.h:13:0,
    from src/ocf/metadata/metadata.h:10,
    from src/ocf/metadata/metadata_hash.c:5:
    src/ocf/metadata/../metadata/metadata_structs.h:193:4: error: unknown type name ‘ocf_metadata_end_t’
    ocf_metadata_end_t cmpl, void *priv);
    ^
    src/ocf/metadata/../metadata/metadata_structs.h:203:4: error: unknown type name ‘ocf_metadata_end_t’
    ocf_metadata_end_t cmpl, void *priv);
    ^
    src/ocf/metadata/../metadata/metadata_structs.h:213:4: error: unknown type name ‘ocf_metadata_end_t’
    ocf_metadata_end_t cmpl, void *priv);
    ^
    src/ocf/metadata/../metadata/metadata_structs.h:245:4: error: unknown type name ‘ocf_metadata_end_t’
    ocf_metadata_end_t cmpl, void *priv);
    ^
    src/ocf/metadata/../metadata/metadata_structs.h:248:4: error: unknown type name ‘ocf_metadata_end_t’
    ocf_metadata_end_t cmpl, void *priv);
    ^
    src/ocf/metadata/../metadata/metadata_structs.h:251:4: error: unknown type name ‘ocf_metadata_end_t’
    ocf_metadata_end_t cmpl, void *priv);
    ^
    In file included from src/ocf/metadata/metadata.h:16:0,
    from src/ocf/metadata/metadata_hash.c:5:
    src/ocf/metadata/metadata_superblock.h:68:3: error: unknown type name ‘ocf_metadata_end_t’
    ocf_metadata_end_t cmpl, void *priv)
    ^
    src/ocf/metadata/metadata_superblock.h:75:3: error: unknown type name ‘ocf_metadata_end_t’
    ocf_metadata_end_t cmpl, void *priv)
    ^
    src/ocf/metadata/metadata_superblock.h:81:3: error: unknown type name ‘ocf_metadata_end_t’
    ocf_metadata_end_t cmpl, void *priv)
    ^
    In file included from src/ocf/metadata/metadata_hash.c:5:0:
    src/ocf/metadata/metadata.h:124:3: error: unknown type name ‘ocf_metadata_end_t’
    ocf_metadata_end_t cmpl, void *priv);
    ^
    src/ocf/metadata/metadata.h:155:3: error: unknown type name ‘ocf_metadata_end_t’
    ocf_metadata_end_t cmpl, void *priv);
    ^
    src/ocf/metadata/metadata.h:165:3: error: unknown type name ‘ocf_metadata_end_t’
    ocf_metadata_end_t cmpl, void *priv);
    ^
    In file included from src/ocf/metadata/metadata_hash.c:7:0:
    src/ocf/metadata/metadata_raw.h:130:4: error: unknown type name ‘ocf_metadata_end_t’
    ocf_metadata_end_t cmpl, void *priv);
    ^
    src/ocf/metadata/metadata_raw.h:133:4: error: unknown type name ‘ocf_metadata_end_t’
    ocf_metadata_end_t cmpl, void *priv);
    ^
    src/ocf/metadata/metadata_raw.h:287:3: error: unknown type name ‘ocf_metadata_end_t’
    ocf_metadata_end_t cmpl, void *priv)
    ^
    src/ocf/metadata/metadata_raw.h:302:3: error: unknown type name ‘ocf_metadata_end_t’
    ocf_metadata_end_t cmpl, void *priv)
    ^
    src/ocf/metadata/metadata_hash.c: In function ‘ocf_metadata_hash_calculate_metadata_size’:
    src/ocf/metadata/metadata_hash.c:246:2: warning: format ‘%lld’ expects argument of type ‘long long int’, but argument 6 has type ‘int64_t’ [-Wformat=]
    OCF_DEBUG_PARAM(cache, "Cache lines = %lld", cache_lines);
    ^
    src/ocf/metadata/metadata_hash.c:314:3: warning: format ‘%lld’ expects argument of type ‘long long int’, but argument 6 has type ‘int64_t’ [-Wformat=]
    OCF_DEBUG_PARAM(cache, "Diff pages = %lld", diff_lines);
    ^
    src/ocf/metadata/metadata_hash.c:315:3: warning: format ‘%lld’ expects argument of type ‘long long int’, but argument 6 has type ‘int64_t’ [-Wformat=]
    OCF_DEBUG_PARAM(cache, "Cache lines = %lld", cache_lines);
    ^
    src/ocf/metadata/metadata_hash.c: In function ‘ocf_metadata_hash_raw_info’:
    src/ocf/metadata/metadata_hash.c:368:3: warning: format ‘%llu’ expects argument of type ‘long long unsigned int’, but argument 6 has type ‘uint64_t’ [-Wformat=]
    OCF_DEBUG_PARAM(cache, " : entries = %llu",
    ^
    src/ocf/metadata/metadata_hash.c:372:3: warning: format ‘%llu’ expects argument of type ‘long long unsigned int’, but argument 6 has type ‘uint64_t’ [-Wformat=]
    OCF_DEBUG_PARAM(cache, " : page offset = %llu",
    ^
    src/ocf/metadata/metadata_hash.c:374:3: warning: format ‘%llu’ expects argument of type ‘long long unsigned int’, but argument 6 has type ‘uint64_t’ [-Wformat=]
    OCF_DEBUG_PARAM(cache, " : pages = %llu",
    ^
    src/ocf/metadata/metadata_hash.c:394:3: warning: format ‘%llu’ expects argument of type ‘long long unsigned int’, but argument 7 has type ‘uint64_t’ [-Wformat=]
    OCF_DEBUG_PARAM(cache, "%s capacity %llu %s",
    ^
    src/ocf/metadata/metadata_hash.c: In function ‘ocf_metadata_config_init’:
    src/ocf/metadata/metadata_hash.c:440:2: warning: format ‘%llu’ expects argument of type ‘long long unsigned int’, but argument 7 has type ‘uint64_t’ [-Wformat=]
    OCF_DEBUG_PARAM(cache, "Cache line size = %lu, bits count = %llu, "
    ^
    src/ocf/metadata/metadata_hash.c: At top level:
    src/ocf/metadata/metadata_hash.c:1192:2: error: unknown type name ‘ocf_metadata_end_t’
    ocf_metadata_end_t cmpl;
    ^
    src/ocf/metadata/metadata_hash.c: In function ‘ocf_medatata_hash_load_segment’:
    src/ocf/metadata/metadata_hash.c:1216:2: warning: implicit declaration of function ‘ocf_metadata_raw_load_all’ [-Wimplicit-function-declaration]
    ocf_metadata_raw_load_all(cache, &ctrl->raw_desc[segment],
    ^
    src/ocf/metadata/metadata_hash.c: In function ‘ocf_metadata_hash_load_superblock_finish’:
    src/ocf/metadata/metadata_hash.c:1387:15: error: called object is not a function or function pointer
    context->cmpl(context->priv, error);
    ^
    src/ocf/metadata/metadata_hash.c: At top level:
    src/ocf/metadata/metadata_hash.c:1439:3: error: unknown type name ‘ocf_metadata_end_t’
    ocf_metadata_end_t cmpl, void *priv)
    ^
    src/ocf/metadata/metadata_hash.c: In function ‘ocf_medatata_hash_flush_segment’:
    src/ocf/metadata/metadata_hash.c:1537:2: warning: implicit declaration of function ‘ocf_metadata_raw_flush_all’ [-Wimplicit-function-declaration]
    ocf_metadata_raw_flush_all(cache, &ctrl->raw_desc[segment],
    ^
    src/ocf/metadata/metadata_hash.c: In function ‘ocf_metadata_hash_flush_superblock_finish’:
    src/ocf/metadata/metadata_hash.c:1550:15: error: called object is not a function or function pointer
    context->cmpl(context->priv, error);
    ^
    src/ocf/metadata/metadata_hash.c: At top level:
    src/ocf/metadata/metadata_hash.c:1587:3: error: unknown type name ‘ocf_metadata_end_t’
    ocf_metadata_end_t cmpl, void *priv)
    ^
    src/ocf/metadata/metadata_hash.c:1620:3: error: unknown type name ‘ocf_metadata_end_t’
    ocf_metadata_end_t cmpl, void *priv)
    ^
    src/ocf/metadata/metadata_hash.c: In function ‘ocf_medatata_hash_flush_all_set_status’:
    src/ocf/metadata/metadata_hash.c:1681:2: warning: implicit declaration of function ‘ocf_metadata_hash_set_shutdown_status’ [-Wimplicit-function-declaration]
    ocf_metadata_hash_set_shutdown_status(cache, shutdown_status,
    ^
    src/ocf/metadata/metadata_hash.c: In function ‘ocf_metadata_hash_flush_all_finish’:
    src/ocf/metadata/metadata_hash.c:1701:15: error: called object is not a function or function pointer
    context->cmpl(context->priv, error);
    ^
    src/ocf/metadata/metadata_hash.c: At top level:
    src/ocf/metadata/metadata_hash.c:1737:3: error: unknown type name ‘ocf_metadata_end_t’
    ocf_metadata_end_t cmpl, void *priv)
    ^
    src/ocf/metadata/metadata_hash.c: In function ‘ocf_metadata_hash_load_all_finish’:
    src/ocf/metadata/metadata_hash.c:1829:15: error: called object is not a function or function pointer
    context->cmpl(context->priv, error);
    ^
    src/ocf/metadata/metadata_hash.c: At top level:
    src/ocf/metadata/metadata_hash.c:1859:3: error: unknown type name ‘ocf_metadata_end_t’
    ocf_metadata_end_t cmpl, void *priv)
    ^
    src/ocf/metadata/metadata_hash.c: In function ‘ocf_metadata_hash_load_recovery_legacy_finish’:
    src/ocf/metadata/metadata_hash.c:2001:15: error: called object is not a function or function pointer
    context->cmpl(context->priv, error);
    ^
    src/ocf/metadata/metadata_hash.c: At top level:
    src/ocf/metadata/metadata_hash.c:2018:3: error: unknown type name ‘ocf_metadata_end_t’
    ocf_metadata_end_t cmpl, void *priv)
    ^
    src/ocf/metadata/metadata_hash.c: In function ‘ocf_metadata_hash_load_recovery_atomic_finish’:
    src/ocf/metadata/metadata_hash.c:2143:15: error: called object is not a function or function pointer
    context->cmpl(context->priv, error);
    ^
    src/ocf/metadata/metadata_hash.c: At top level:
    src/ocf/metadata/metadata_hash.c:2162:3: error: unknown type name ‘ocf_metadata_end_t’
    ocf_metadata_end_t cmpl, void *priv)
    ^
    src/ocf/metadata/metadata_hash.c:2189:3: error: unknown type name ‘ocf_metadata_end_t’
    ocf_metadata_end_t cmpl, void *priv)
    ^
    src/ocf/metadata/metadata_hash.c:2724:2: error: unknown field ‘flush_all’ specified in initializer
    .flush_all = ocf_metadata_hash_flush_all,
    ^
    src/ocf/metadata/metadata_hash.c:2724:15: error: ‘ocf_metadata_hash_flush_all’ undeclared here (not in a function)
    .flush_all = ocf_metadata_hash_flush_all,
    ^
    src/ocf/metadata/metadata_hash.c:2727:2: error: unknown field ‘load_all’ specified in initializer
    .load_all = ocf_metadata_hash_load_all,
    ^
    src/ocf/metadata/metadata_hash.c:2727:14: error: ‘ocf_metadata_hash_load_all’ undeclared here (not in a function)
    .load_all = ocf_metadata_hash_load_all,
    ^
    src/ocf/metadata/metadata_hash.c:2728:2: error: unknown field ‘load_recovery’ specified in initializer
    .load_recovery = ocf_metadata_hash_load_recovery,
    ^
    src/ocf/metadata/metadata_hash.c:2728:19: error: ‘ocf_metadata_hash_load_recovery’ undeclared here (not in a function)
    .load_recovery = ocf_metadata_hash_load_recovery,
    ^
    src/ocf/metadata/metadata_hash.c:2733:2: error: unknown field ‘set_shutdown_status’ specified in initializer
    .set_shutdown_status = ocf_metadata_hash_set_shutdown_status,
    ^
    src/ocf/metadata/metadata_hash.c:2733:25: error: ‘ocf_metadata_hash_set_shutdown_status’ undeclared here (not in a function)
    .set_shutdown_status = ocf_metadata_hash_set_shutdown_status,
    ^
    src/ocf/metadata/metadata_hash.c:2734:2: error: unknown field ‘flush_superblock’ specified in initializer
    .flush_superblock = ocf_metadata_hash_flush_superblock,
    ^
    src/ocf/metadata/metadata_hash.c:2734:22: error: ‘ocf_metadata_hash_flush_superblock’ undeclared here (not in a function)
    .flush_superblock = ocf_metadata_hash_flush_superblock,
    ^
    src/ocf/metadata/metadata_hash.c:2735:2: error: unknown field ‘load_superblock’ specified in initializer
    .load_superblock = ocf_metadata_hash_load_superblock,
    ^
    src/ocf/metadata/metadata_hash.c:2735:21: error: ‘ocf_metadata_hash_load_superblock’ undeclared here (not in a function)
    .load_superblock = ocf_metadata_hash_load_superblock,
    ^
    make[1]: *** [src/ocf/metadata/metadata_hash.o] Error 1
    make[1]: Leaving directory `/home/wayne/open-cas-linux/ocf/example/simple'
    make: *** [all] Error 2
    [root@localhost simple]#
    Socket error Event: 32 Error: 10053.
    Connection closing...Socket close.

Connection closed by foreign host.

Disconnected from remote host(10.239.70.152-lab) at 20:40:05.

Type `help' to learn how to use Xshell prompt.
[C:~]$

Invalid logic in cache async lock reference counting

ocf_mngt_cache_lock() increments cache reference counter. When cache is being stopped, outstanding waiters are woken up if cache reference count drops to 0. This will actually never happen as the waiters are holding reference to cache object.

To fix this it should probably be enough to wake up outstanding waiters unconditionally during cache stop.

Need for extra documents

Where can I find docs about the detail design? For example, metadata, IO engine, cache-line, etc.

It's very difficult to understand the code barely by comments.

env_strncpy misbehaves when strlen(src) == strlen

From code inspection, I found that env/postix/ocf_env.h isn't quite right.

#include <stdio.h>
#include <string.h>

#define min(a, b) ((a < b) ? (a) : (b))
#define env_strncpy(dest, dmax, src, slen) ({ \
                strncpy(dest, src, min(dmax - 1, slen)); \
                dest[dmax - 1] = '\0'; \
                0; \
        })

int
main(void)
{
        char buf[16] = "0123456789abcde";
        char *s = "###";

        (void) printf(" before: '%s'\n", buf);
        env_strncpy(buf, sizeof (buf), s, strlen(s));
        (void) printf("  after: '%s'\n", buf);
        return (0);
}
$ ./t
 before: '0123456789abcde'
  after: '###3456789abcde'

My example program is fixed with the following:

@@ -4,7 +4,7 @@
 #define min(a, b) ((a < b) ? (a) : (b))
 #define env_strncpy(dest, dmax, src, slen) ({ \
                 strncpy(dest, src, min(dmax - 1, slen)); \
-                dest[dmax - 1] = '\0'; \
+                dest[min(dmax - 1, slen)] = '\0'; \
                 0; \
         })

I believe a similar issue exists with env_strncmp(), but I have not analyzed callers to understand their intent.

Not all objects deallocated after submitting discard IOs

Hi,
I have an issue that using ocf_submit_discard ends up in some allocator objects not being deallocated

After doing ocf_cache_stop -> ocf_ctx_exit I see in env_allocator_destroy that allocated object counter is > 0.
Same is not the case for reads, writes and flushes - all objects are getting freed by the end of applictation as it expected

I use Write-Through mode with one cache device and one core device
My environment for OCF is this implementation in SPDK: https://review.gerrithub.io/#/c/spdk/spdk/+/435708/

Please help describe the IO procedures of CAS

ocf_core_submit_io
    ocf_volume_submit_io
        volume->type->properties->ops.submit_io
            ocf_core_volume_submit_io
                ocf_engine_hndl_req
                    ocf_queue_run
                        ocf_io_handle
                            ocf_write_wt  // req->io_if->write
                                _ocf_write_wt_do
                                    _ocf_write_wt_submit    
                                        ocf_submit_cache_reqs
                                            ocf_volume_submit_io    

Looks like the final ocf_volume_submit_io traverses back to the beginning.
So I guess the CAS IO is a run-to-completion design? And where should it block if there is no incoming IO?

Not able to compile spdk with ocf

Tried compiling SPDK with ocf and it failed with below:

[root@Target spdk]# ./configure --with-rdma --with-ocf=/home/ocf
Using default SPDK env in /home/spdk/lib/env_dpdk
Using default DPDK in /home/spdk/dpdk/build
Configuring ISA-L (logfile: /tmp/spdk-isal.log)...done.
configuring OCF...
INSTALL /home/spdk/lib/bdev/ocf/env/include/ocf/ocf_stats.h
INSTALL /home/spdk/lib/bdev/ocf/env/include/ocf/ocf_utilities.h
INSTALL /home/spdk/lib/bdev/ocf/env/include/ocf/ocf_io_class.h
INSTALL /home/spdk/lib/bdev/ocf/env/include/ocf/ocf_def.h
INSTALL /home/spdk/lib/bdev/ocf/env/include/ocf/ocf_cfg.h
INSTALL /home/spdk/lib/bdev/ocf/env/include/ocf/ocf_logger.h
INSTALL /home/spdk/lib/bdev/ocf/env/include/ocf/ocf_metadata_updater.h
INSTALL /home/spdk/lib/bdev/ocf/env/include/ocf/ocf_io.h
INSTALL /home/spdk/lib/bdev/ocf/env/include/ocf/ocf_queue.h
INSTALL /home/spdk/lib/bdev/ocf/env/include/ocf/ocf_ctx.h
INSTALL /home/spdk/lib/bdev/ocf/env/include/ocf/ocf_trace.h
INSTALL /home/spdk/lib/bdev/ocf/env/include/ocf/ocf_cleaner.h
INSTALL /home/spdk/lib/bdev/ocf/env/include/ocf/ocf_cache.h
INSTALL /home/spdk/lib/bdev/ocf/env/include/ocf/ocf_stats_builder.h
INSTALL /home/spdk/lib/bdev/ocf/env/include/ocf/ocf_core.h
INSTALL /home/spdk/lib/bdev/ocf/env/include/ocf/cleaning/alru.h
INSTALL /home/spdk/lib/bdev/ocf/env/include/ocf/cleaning/acp.h
INSTALL /home/spdk/lib/bdev/ocf/env/include/ocf/ocf_metadata.h
INSTALL /home/spdk/lib/bdev/ocf/env/include/ocf/ocf_types.h
INSTALL /home/spdk/lib/bdev/ocf/env/include/ocf/ocf_err.h
INSTALL /home/spdk/lib/bdev/ocf/env/include/ocf/ocf.h
INSTALL /home/spdk/lib/bdev/ocf/env/include/ocf/ocf_data_obj.h
INSTALL /home/spdk/lib/bdev/ocf/env/include/ocf/ocf_mngt.h
make[1]: Entering directory '/home/ocf'
make[1]: Nothing to be done for 'distcleandir'.
make[1]: Leaving directory '/home/ocf'
done configuring OCF
Creating mk/config.mk...done.
Type 'make' to build.

[root@Target spdk]# make
Configuration done using x86_64-native-linuxapp-gcc
make[3]: Entering directory '/home/spdk/dpdk'
== Build lib
== Build lib/librte_compat
SYMLINK-FILE include/rte_compat.h
== Build lib/librte_kvargs
SYMLINK-FILE include/rte_kvargs.h
CC rte_kvargs.o
AR librte_kvargs.a
INSTALL-LIB librte_kvargs.a
== Build lib/librte_eal
== Build lib/librte_eal/common
SYMLINK-FILE include/rte_branch_prediction.h
SYMLINK-FILE include/rte_common.h
SYMLINK-FILE include/rte_debug.h
SYMLINK-FILE include/rte_eal.h
SYMLINK-FILE include/rte_eal_interrupts.h
SYMLINK-FILE include/rte_errno.h
SYMLINK-FILE include/rte_launch.h
SYMLINK-FILE include/rte_lcore.h
SYMLINK-FILE include/rte_log.h
SYMLINK-FILE include/rte_memory.h
SYMLINK-FILE include/rte_memzone.h
SYMLINK-FILE include/rte_per_lcore.h
SYMLINK-FILE include/rte_random.h
SYMLINK-FILE include/rte_tailq.h
SYMLINK-FILE include/rte_interrupts.h
SYMLINK-FILE include/rte_alarm.h
SYMLINK-FILE include/rte_string_fns.h
SYMLINK-FILE include/rte_version.h
SYMLINK-FILE include/rte_eal_memconfig.h
SYMLINK-FILE include/rte_malloc_heap.h
SYMLINK-FILE include/rte_hexdump.h
SYMLINK-FILE include/rte_devargs.h
SYMLINK-FILE include/rte_bus.h
SYMLINK-FILE include/rte_dev.h
SYMLINK-FILE include/rte_class.h
SYMLINK-FILE include/rte_option.h
SYMLINK-FILE include/rte_pci_dev_feature_defs.h
SYMLINK-FILE include/rte_pci_dev_features.h
SYMLINK-FILE include/rte_malloc.h
SYMLINK-FILE include/rte_keepalive.h
SYMLINK-FILE include/rte_time.h
SYMLINK-FILE include/rte_service_component.h
SYMLINK-FILE include/rte_service.h
SYMLINK-FILE include/rte_bitmap.h
SYMLINK-FILE include/rte_vfio.h
SYMLINK-FILE include/rte_hypervisor.h
SYMLINK-FILE include/rte_test.h
SYMLINK-FILE include/rte_reciprocal.h
SYMLINK-FILE include/rte_fbarray.h
SYMLINK-FILE include/rte_uuid.h
SYMLINK-FILE include/rte_atomic.h
SYMLINK-FILE include/rte_atomic_32.h
SYMLINK-FILE include/rte_atomic_64.h
SYMLINK-FILE include/rte_byteorder.h
SYMLINK-FILE include/rte_byteorder_32.h
SYMLINK-FILE include/rte_byteorder_64.h
SYMLINK-FILE include/rte_cpuflags.h
SYMLINK-FILE include/rte_cycles.h
SYMLINK-FILE include/rte_io.h
SYMLINK-FILE include/rte_memcpy.h
SYMLINK-FILE include/rte_pause.h
SYMLINK-FILE include/rte_prefetch.h
SYMLINK-FILE include/rte_rtm.h
SYMLINK-FILE include/rte_rwlock.h
SYMLINK-FILE include/rte_spinlock.h
SYMLINK-FILE include/rte_vect.h
SYMLINK-FILE include/generic/rte_atomic.h
SYMLINK-FILE include/generic/rte_byteorder.h
SYMLINK-FILE include/generic/rte_cycles.h
SYMLINK-FILE include/generic/rte_prefetch.h
SYMLINK-FILE include/generic/rte_spinlock.h
SYMLINK-FILE include/generic/rte_memcpy.h
SYMLINK-FILE include/generic/rte_cpuflags.h
SYMLINK-FILE include/generic/rte_rwlock.h
SYMLINK-FILE include/generic/rte_vect.h
SYMLINK-FILE include/generic/rte_io.h
SYMLINK-FILE include/generic/rte_pause.h
== Build lib/librte_eal/linuxapp
== Build lib/librte_eal/linuxapp/eal
SYMLINK-FILE include/exec-env/rte_kni_common.h
CC eal.o
CC eal_cpuflags.o
CC eal_hugepage_info.o
CC eal_memory.o
CC eal_thread.o
CC eal_log.o
CC eal_vfio.o
CC eal_vfio_mp_sync.o
CC eal_memalloc.o
CC eal_lcore.o
CC eal_debug.o
CC eal_timer.o
CC eal_interrupts.o
CC eal_alarm.o
CC eal_dev.o
CC eal_common_lcore.o
CC eal_common_timer.o
CC eal_common_memzone.o
CC eal_common_log.o
CC eal_common_launch.o
CC eal_common_memalloc.o
CC eal_common_memory.o
CC eal_common_tailqs.o
CC eal_common_errno.o
CC eal_common_cpuflags.o
CC eal_common_hypervisor.o
CC eal_common_string_fns.o
CC eal_common_hexdump.o
CC eal_common_devargs.o
CC eal_common_class.o
CC eal_common_bus.o
CC eal_common_dev.o
CC eal_common_options.o
CC eal_common_thread.o
CC eal_common_proc.o
CC eal_common_fbarray.o
CC eal_common_uuid.o
CC rte_malloc.o
CC hotplug_mp.o
CC malloc_elem.o
CC malloc_heap.o
CC malloc_mp.o
CC rte_keepalive.o
CC rte_option.o
CC rte_service.o
CC rte_reciprocal.o
CC rte_cpuflags.o
CC rte_hypervisor.o
CC rte_spinlock.o
CC rte_cycles.o
AR librte_eal.a
INSTALL-LIB librte_eal.a
== Build lib/librte_pci
== Build lib/librte_ring
== Build lib/librte_cmdline
SYMLINK-FILE include/rte_pci.h
CC rte_pci.o
SYMLINK-FILE include/rte_ring.h
SYMLINK-FILE include/rte_ring_generic.h
SYMLINK-FILE include/rte_ring_c11_mem.h
CC rte_ring.o
SYMLINK-FILE include/cmdline.h
SYMLINK-FILE include/cmdline_parse_num.h
SYMLINK-FILE include/cmdline_parse.h
SYMLINK-FILE include/cmdline_parse_ipaddr.h
SYMLINK-FILE include/cmdline_parse_string.h
SYMLINK-FILE include/cmdline_parse_etheraddr.h
SYMLINK-FILE include/cmdline_rdline.h
SYMLINK-FILE include/cmdline_vt100.h
SYMLINK-FILE include/cmdline_socket.h
SYMLINK-FILE include/cmdline_cirbuf.h
SYMLINK-FILE include/cmdline_parse_portlist.h
CC cmdline.o
CC cmdline_cirbuf.o
CC cmdline_parse.o
CC cmdline_parse_etheraddr.o
CC cmdline_parse_ipaddr.o
CC cmdline_parse_num.o
CC cmdline_parse_string.o
CC cmdline_rdline.o
CC cmdline_vt100.o
CC cmdline_socket.o
CC cmdline_parse_portlist.o
AR librte_pci.a
INSTALL-LIB librte_pci.a
AR librte_ring.a
INSTALL-LIB librte_ring.a
== Build lib/librte_mempool
AR librte_cmdline.a
INSTALL-LIB librte_cmdline.a
SYMLINK-FILE include/rte_mempool.h
CC rte_mempool.o
CC rte_mempool_ops.o
CC rte_mempool_ops_default.o
AR librte_mempool.a
INSTALL-LIB librte_mempool.a
== Build lib/librte_mbuf
SYMLINK-FILE include/rte_mbuf.h
SYMLINK-FILE include/rte_mbuf_ptype.h
SYMLINK-FILE include/rte_mbuf_pool_ops.h
CC rte_mbuf.o
CC rte_mbuf_pool_ops.o
CC rte_mbuf_ptype.o
AR librte_mbuf.a
INSTALL-LIB librte_mbuf.a
== Build lib/librte_net
== Build lib/librte_cryptodev
== Build lib/librte_compressdev
SYMLINK-FILE include/rte_ip.h
SYMLINK-FILE include/rte_udp.h
SYMLINK-FILE include/rte_tcp.h
SYMLINK-FILE include/rte_sctp.h
SYMLINK-FILE include/rte_esp.h
SYMLINK-FILE include/rte_crypto_sym.h
SYMLINK-FILE include/rte_icmp.h
SYMLINK-FILE include/rte_cryptodev_pmd.h
SYMLINK-FILE include/rte_arp.h
SYMLINK-FILE include/rte_crypto.h
SYMLINK-FILE include/rte_gre.h
SYMLINK-FILE include/rte_crypto_asym.h
SYMLINK-FILE include/rte_cryptodev.h
SYMLINK-FILE include/rte_ether.h
SYMLINK-FILE include/rte_net.h
SYMLINK-FILE include/rte_comp.h
SYMLINK-FILE include/rte_net_crc.h
SYMLINK-FILE include/rte_compressdev.h
SYMLINK-FILE include/rte_compressdev_pmd.h
SYMLINK-FILE include/rte_compressdev_internal.h
CC rte_cryptodev.o
SYMLINK-FILE include/rte_mpls.h
CC rte_cryptodev_pmd.o
CC rte_net.o
CC rte_net_crc.o
CC rte_compressdev.o
CC rte_arp.o
CC rte_compressdev_pmd.o
CC rte_comp.o
AR librte_net.a
INSTALL-LIB librte_net.a
== Build lib/librte_ethdev
AR librte_compressdev.a
SYMLINK-FILE include/rte_ethdev_core.h
SYMLINK-FILE include/rte_ethdev.h
SYMLINK-FILE include/rte_ethdev_driver.h
SYMLINK-FILE include/rte_ethdev_pci.h
SYMLINK-FILE include/rte_ethdev_vdev.h
SYMLINK-FILE include/rte_eth_ctrl.h
SYMLINK-FILE include/rte_tm.h
SYMLINK-FILE include/rte_flow_driver.h
SYMLINK-FILE include/rte_flow.h
SYMLINK-FILE include/rte_dev_info.h
SYMLINK-FILE include/rte_tm_driver.h
SYMLINK-FILE include/rte_mtr.h
INSTALL-LIB librte_compressdev.a
SYMLINK-FILE include/rte_mtr_driver.h
CC ethdev_private.o
CC rte_ethdev.o
CC rte_class_eth.o
CC rte_flow.o
CC rte_tm.o
CC rte_mtr.o
CC ethdev_profile.o
AR librte_cryptodev.a
INSTALL-LIB librte_cryptodev.a
AR librte_ethdev.a
INSTALL-LIB librte_ethdev.a
== Build buildtools
== Build kernel
== Build buildtools/pmdinfogen
== Build kernel/linux
HOSTCC pmdinfogen.o
HOSTLD dpdk-pmdinfogen
INSTALL-HOSTAPP dpdk-pmdinfogen
== Build drivers
== Build drivers/common
== Build drivers/bus
== Build drivers/bus/pci
== Build drivers/bus/vdev
SYMLINK-FILE include/rte_bus_vdev.h
CC vdev.o
CC vdev_params.o
SYMLINK-FILE include/rte_bus_pci.h
CC pci_params.o
CC linux/pci.o
CC pci_common_uio.o
CC pci_common.o
CC linux/pci_uio.o
CC linux/pci_vfio.o
AR librte_bus_vdev.a
INSTALL-LIB librte_bus_vdev.a
AR librte_bus_pci.a
INSTALL-LIB librte_bus_pci.a
== Build drivers/mempool
== Build drivers/mempool/bucket
== Build drivers/mempool/ring
CC rte_mempool_bucket.o
CC rte_mempool_ring.o
AR librte_mempool_ring.a
INSTALL-LIB librte_mempool_ring.a
AR librte_mempool_bucket.a
INSTALL-LIB librte_mempool_bucket.a
== Build drivers/net
== Build drivers/crypto
== Build drivers/compress
== Build app
Build complete [x86_64-native-linuxapp-gcc]
make[3]: Leaving directory '/home/spdk/dpdk'
CC lib/bdev/bdev.o
CC lib/bdev/part.o
CC lib/bdev/scsi_nvme.o
LIB libspdk_bdev.a
CC lib/bdev/error/vbdev_error.o
CC lib/bdev/error/vbdev_error_rpc.o
LIB libspdk_bdev_error.a
CC lib/bdev/gpt/gpt.o
CC lib/bdev/gpt/vbdev_gpt.o
LIB libspdk_bdev_gpt.a
CC lib/bdev/lvol/vbdev_lvol.o
CC lib/bdev/lvol/vbdev_lvol_rpc.o
LIB libspdk_bdev_lvol.a
CC lib/bdev/malloc/bdev_malloc.o
CC lib/bdev/malloc/bdev_malloc_rpc.o
LIB libspdk_bdev_malloc.a
CC lib/bdev/null/bdev_null.o
CC lib/bdev/null/bdev_null_rpc.o
LIB libspdk_bdev_null.a
CC lib/bdev/nvme/bdev_nvme.o
CC lib/bdev/nvme/bdev_nvme_rpc.o
CC lib/bdev/nvme/nvme_rpc.o
LIB libspdk_bdev_nvme.a
CC lib/bdev/passthru/vbdev_passthru.o
CC lib/bdev/passthru/vbdev_passthru_rpc.o
LIB libspdk_bdev_passthru.a
CC lib/bdev/raid/bdev_raid.o
CC lib/bdev/raid/bdev_raid_rpc.o
LIB libspdk_bdev_raid.a
CC lib/bdev/rpc/bdev_rpc.o
LIB libspdk_bdev_rpc.a
CC lib/bdev/split/vbdev_split.o
CC lib/bdev/split/vbdev_split_rpc.o
LIB libspdk_bdev_split.a
CC lib/bdev/ocf/env/src/ocf/ocf_queue.o
CC lib/bdev/ocf/env/src/ocf/engine/engine_wi.o
CC lib/bdev/ocf/env/src/ocf/engine/engine_wt.o
CC lib/bdev/ocf/env/src/ocf/engine/engine_wb.o
CC lib/bdev/ocf/env/src/ocf/engine/engine_fast.o
CC lib/bdev/ocf/env/src/ocf/engine/engine_common.o
CC lib/bdev/ocf/env/src/ocf/engine/engine_rd.o
CC lib/bdev/ocf/env/src/ocf/engine/engine_bf.o
CC lib/bdev/ocf/env/src/ocf/engine/engine_pt.o
CC lib/bdev/ocf/env/src/ocf/engine/engine_discard.o
CC lib/bdev/ocf/env/src/ocf/engine/engine_d2c.o
CC lib/bdev/ocf/env/src/ocf/engine/engine_wa.o
CC lib/bdev/ocf/env/src/ocf/engine/engine_zero.o
CC lib/bdev/ocf/env/src/ocf/engine/engine_inv.o
CC lib/bdev/ocf/env/src/ocf/engine/engine_ops.o
CC lib/bdev/ocf/env/src/ocf/engine/cache_engine.o
CC lib/bdev/ocf/env/src/ocf/ocf_utils.o
CC lib/bdev/ocf/env/src/ocf/metadata/metadata_hash.o
CC lib/bdev/ocf/env/src/ocf/metadata/metadata_partition.o
CC lib/bdev/ocf/env/src/ocf/metadata/metadata_misc.o
CC lib/bdev/ocf/env/src/ocf/metadata/metadata_raw_dynamic.o
CC lib/bdev/ocf/env/src/ocf/metadata/metadata.o
CC lib/bdev/ocf/env/src/ocf/metadata/metadata_io.o
CC lib/bdev/ocf/env/src/ocf/metadata/metadata_updater.o
CC lib/bdev/ocf/env/src/ocf/metadata/metadata_raw.o
CC lib/bdev/ocf/env/src/ocf/metadata/metadata_raw_volatile.o
CC lib/bdev/ocf/env/src/ocf/metadata/metadata_raw_atomic.o
CC lib/bdev/ocf/env/src/ocf/metadata/metadata_collision.o
CC lib/bdev/ocf/env/src/ocf/ocf_ctx.o
CC lib/bdev/ocf/env/src/ocf/ocf_stats.o
CC lib/bdev/ocf/env/src/ocf/concurrency/ocf_concurrency.o
CC lib/bdev/ocf/env/src/ocf/concurrency/ocf_cache_concurrency.o
CC lib/bdev/ocf/env/src/ocf/mngt/ocf_mngt_misc.o
CC lib/bdev/ocf/env/src/ocf/mngt/ocf_mngt_core_pool.o
CC lib/bdev/ocf/env/src/ocf/mngt/ocf_mngt_cache.o
CC lib/bdev/ocf/env/src/ocf/mngt/ocf_mngt_core.o
CC lib/bdev/ocf/env/src/ocf/mngt/ocf_mngt_flush.o
CC lib/bdev/ocf/env/src/ocf/mngt/ocf_mngt_io_class.o
CC lib/bdev/ocf/env/src/ocf/mngt/ocf_mngt_common.o
CC lib/bdev/ocf/env/src/ocf/ocf_core.o
CC lib/bdev/ocf/env/src/ocf/ocf_data_obj.o
CC lib/bdev/ocf/env/src/ocf/ocf_cache.o
CC lib/bdev/ocf/env/src/ocf/eviction/eviction.o
CC lib/bdev/ocf/env/src/ocf/eviction/lru.o
CC lib/bdev/ocf/env/src/ocf/cleaning/alru.o
CC lib/bdev/ocf/env/src/ocf/cleaning/cleaning.o
CC lib/bdev/ocf/env/src/ocf/cleaning/acp.o
CC lib/bdev/ocf/env/src/ocf/ocf_logger.o
CC lib/bdev/ocf/env/src/ocf/ocf_trace.o
CC lib/bdev/ocf/env/src/ocf/ocf_stats_builder.o
CC lib/bdev/ocf/env/src/ocf/ocf_io.o
CC lib/bdev/ocf/env/src/ocf/ocf_io_class.o
CC lib/bdev/ocf/env/src/ocf/utils/utils_io.o
CC lib/bdev/ocf/env/src/ocf/utils/utils_cache_line.o
CC lib/bdev/ocf/env/src/ocf/utils/utils_list.o
CC lib/bdev/ocf/env/src/ocf/utils/utils_allocator.o
CC lib/bdev/ocf/env/src/ocf/utils/utils_cleaner.o
CC lib/bdev/ocf/env/src/ocf/utils/utils_part.o
CC lib/bdev/ocf/env/src/ocf/utils/utils_req.o
CC lib/bdev/ocf/env/src/ocf/ocf_metadata.o
CC lib/bdev/ocf/env/ocf_env.o
LIB libspdk_ocfenv.a
CC lib/bdev/ocf/ctx.o
ctx.c:340:3: error: ‘const struct ocf_ctx_ops’ has no member named ‘name’
.name = "OCF SPDK",
^~~~
ctx.c:340:10: warning: initialization of ‘ctx_data_t * ()(uint32_t)’ {aka ‘void * ()(unsigned int)’} from incompatible pointer type ‘char ’ [-Wincompatible-pointer-types]
.name = "OCF SPDK",
^~~~~~~~~~
ctx.c:340:10: note: (near initialization for ‘vbdev_ocf_ctx_ops.data.alloc’)
ctx.c:342:3: error: ‘const struct ocf_ctx_ops’ has no member named ‘data_alloc’
.data_alloc = vbdev_ocf_ctx_data_alloc,
^~~~~~~~~~
ctx.c:342:16: warning: initialization of ‘int (
)(struct ocf_queue )’ from incompatible pointer type ‘ctx_data_t * ()(uint32_t)’ {aka ‘void * ()(unsigned int)’} [-Wincompatible-pointer-types]
.data_alloc = vbdev_ocf_ctx_data_alloc,
^~~~~~~~~~~~~~~~~~~~~~~~
ctx.c:342:16: note: (near initialization for ‘vbdev_ocf_ctx_ops.queue.init’)
ctx.c:343:3: error: ‘const struct ocf_ctx_ops’ has no member named ‘data_free’
.data_free = vbdev_ocf_ctx_data_free,
^~~~~~~~~
ctx.c:343:15: warning: initialization of ‘int (
)(struct ocf_cleaner )’ from incompatible pointer type ‘void ()(ctx_data_t )’ {aka ‘void ()(void )’} [-Wincompatible-pointer-types]
.data_free = vbdev_ocf_ctx_data_free,
^~~~~~~~~~~~~~~~~~~~~~~
ctx.c:343:15: note: (near initialization for ‘vbdev_ocf_ctx_ops.cleaner.init’)
ctx.c:344:3: error: ‘const struct ocf_ctx_ops’ has no member named ‘data_mlock’
.data_mlock = vbdev_ocf_ctx_data_mlock,
^~~~~~~~~~
ctx.c:344:16: warning: initialization of ‘int (
)(struct ocf_metadata_updater )’ from incompatible pointer type ‘int ()(ctx_data_t )’ {aka ‘int ()(void )’} [-Wincompatible-pointer-types]
.data_mlock = vbdev_ocf_ctx_data_mlock,
^~~~~~~~~~~~~~~~~~~~~~~~
ctx.c:344:16: note: (near initialization for ‘vbdev_ocf_ctx_ops.metadata_updater.init’)
ctx.c:345:3: error: ‘const struct ocf_ctx_ops’ has no member named ‘data_munlock’
.data_munlock = vbdev_ocf_ctx_data_munlock,
^~~~~~~~~~~~
ctx.c:345:18: warning: initialization of ‘int (
)(struct ocf_logger )’ from incompatible pointer type ‘void ()(ctx_data_t )’ {aka ‘void ()(void *)’} [-Wincompatible-pointer-types]
.data_munlock = vbdev_ocf_ctx_data_munlock,
^~~~~~~~~~~~~~~~~~~~~~~~~~
ctx.c:345:18: note: (near initialization for ‘vbdev_ocf_ctx_ops.logger.open’)
ctx.c:346:3: error: ‘const struct ocf_ctx_ops’ has no member named ‘data_rd’; did you mean ‘data’?
.data_rd = vbdev_ocf_ctx_data_rd,
^~~~~~~
data
ctx.c:346:13: warning: excess elements in struct initializer
.data_rd = vbdev_ocf_ctx_data_rd,
^~~~~~~~~~~~~~~~~~~~~
ctx.c:346:13: note: (near initialization for ‘vbdev_ocf_ctx_ops’)
ctx.c:347:3: error: ‘const struct ocf_ctx_ops’ has no member named ‘data_wr’; did you mean ‘data’?
.data_wr = vbdev_ocf_ctx_data_wr,
^~~~~~~
data
ctx.c:347:13: warning: excess elements in struct initializer
.data_wr = vbdev_ocf_ctx_data_wr,
^~~~~~~~~~~~~~~~~~~~~
ctx.c:347:13: note: (near initialization for ‘vbdev_ocf_ctx_ops’)
ctx.c:348:3: error: ‘const struct ocf_ctx_ops’ has no member named ‘data_zero’
.data_zero = vbdev_ocf_ctx_data_zero,
^~~~~~~~~
ctx.c:348:15: warning: excess elements in struct initializer
.data_zero = vbdev_ocf_ctx_data_zero,
^~~~~~~~~~~~~~~~~~~~~~~
ctx.c:348:15: note: (near initialization for ‘vbdev_ocf_ctx_ops’)
ctx.c:349:3: error: ‘const struct ocf_ctx_ops’ has no member named ‘data_seek’
.data_seek = vbdev_ocf_ctx_data_seek,
^~~~~~~~~
ctx.c:349:15: warning: excess elements in struct initializer
.data_seek = vbdev_ocf_ctx_data_seek,
^~~~~~~~~~~~~~~~~~~~~~~
ctx.c:349:15: note: (near initialization for ‘vbdev_ocf_ctx_ops’)
ctx.c:350:3: error: ‘const struct ocf_ctx_ops’ has no member named ‘data_cpy’; did you mean ‘data’?
.data_cpy = vbdev_ocf_ctx_data_cpy,
^~~~~~~~
data
ctx.c:350:14: warning: excess elements in struct initializer
.data_cpy = vbdev_ocf_ctx_data_cpy,
^~~~~~~~~~~~~~~~~~~~~~
ctx.c:350:14: note: (near initialization for ‘vbdev_ocf_ctx_ops’)
ctx.c:351:3: error: ‘const struct ocf_ctx_ops’ has no member named ‘data_secure_erase’
.data_secure_erase = vbdev_ocf_ctx_data_secure_erase,
^~~~~~~~~~~~~~~~~
ctx.c:351:23: warning: excess elements in struct initializer
.data_secure_erase = vbdev_ocf_ctx_data_secure_erase,
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
ctx.c:351:23: note: (near initialization for ‘vbdev_ocf_ctx_ops’)
ctx.c:353:3: error: ‘const struct ocf_ctx_ops’ has no member named ‘queue_init’; did you mean ‘queue’?
.queue_init = vbdev_ocf_ctx_queue_init,
^~~~~~~~~~
queue
ctx.c:353:16: warning: excess elements in struct initializer
.queue_init = vbdev_ocf_ctx_queue_init,
^~~~~~~~~~~~~~~~~~~~~~~~
ctx.c:353:16: note: (near initialization for ‘vbdev_ocf_ctx_ops’)
ctx.c:354:3: error: ‘const struct ocf_ctx_ops’ has no member named ‘queue_kick’; did you mean ‘queue’?
.queue_kick = vbdev_ocf_ctx_queue_kick,
^~~~~~~~~~
queue
ctx.c:354:16: warning: excess elements in struct initializer
.queue_kick = vbdev_ocf_ctx_queue_kick,
^~~~~~~~~~~~~~~~~~~~~~~~
ctx.c:354:16: note: (near initialization for ‘vbdev_ocf_ctx_ops’)
ctx.c:355:3: error: ‘const struct ocf_ctx_ops’ has no member named ‘queue_stop’; did you mean ‘queue’?
.queue_stop = vbdev_ocf_ctx_queue_stop,
^~~~~~~~~~
queue
ctx.c:355:16: warning: excess elements in struct initializer
.queue_stop = vbdev_ocf_ctx_queue_stop,
^~~~~~~~~~~~~~~~~~~~~~~~
ctx.c:355:16: note: (near initialization for ‘vbdev_ocf_ctx_ops’)
ctx.c:357:3: error: ‘const struct ocf_ctx_ops’ has no member named ‘cleaner_init’; did you mean ‘cleaner’?
.cleaner_init = vbdev_ocf_ctx_cleaner_init,
^~~~~~~~~~~~
cleaner
ctx.c:357:18: warning: excess elements in struct initializer
.cleaner_init = vbdev_ocf_ctx_cleaner_init,
^~~~~~~~~~~~~~~~~~~~~~~~~~
ctx.c:357:18: note: (near initialization for ‘vbdev_ocf_ctx_ops’)
ctx.c:358:3: error: ‘const struct ocf_ctx_ops’ has no member named ‘cleaner_stop’; did you mean ‘cleaner’?
.cleaner_stop = vbdev_ocf_ctx_cleaner_stop,
^~~~~~~~~~~~
cleaner
ctx.c:358:18: warning: excess elements in struct initializer
.cleaner_stop = vbdev_ocf_ctx_cleaner_stop,
^~~~~~~~~~~~~~~~~~~~~~~~~~
ctx.c:358:18: note: (near initialization for ‘vbdev_ocf_ctx_ops’)
ctx.c:360:3: error: ‘const struct ocf_ctx_ops’ has no member named ‘metadata_updater_init’; did you mean ‘metadata_updater’?
.metadata_updater_init = vbdev_ocf_dobj_updater_init,
^~~~~~~~~~~~~~~~~~~~~
metadata_updater
ctx.c:360:27: warning: excess elements in struct initializer
.metadata_updater_init = vbdev_ocf_dobj_updater_init,
^~~~~~~~~~~~~~~~~~~~~~~~~~~
ctx.c:360:27: note: (near initialization for ‘vbdev_ocf_ctx_ops’)
ctx.c:361:3: error: ‘const struct ocf_ctx_ops’ has no member named ‘metadata_updater_stop’; did you mean ‘metadata_updater’?
.metadata_updater_stop = vbdev_ocf_dobj_updater_stop,
^~~~~~~~~~~~~~~~~~~~~
metadata_updater
ctx.c:361:27: warning: excess elements in struct initializer
.metadata_updater_stop = vbdev_ocf_dobj_updater_stop,
^~~~~~~~~~~~~~~~~~~~~~~~~~~
ctx.c:361:27: note: (near initialization for ‘vbdev_ocf_ctx_ops’)
ctx.c:362:3: error: ‘const struct ocf_ctx_ops’ has no member named ‘metadata_updater_kick’; did you mean ‘metadata_updater’?
.metadata_updater_kick = vbdev_ocf_dobj_updater_kick,
^~~~~~~~~~~~~~~~~~~~~
metadata_updater
ctx.c:362:27: warning: excess elements in struct initializer
.metadata_updater_kick = vbdev_ocf_dobj_updater_kick,
^~~~~~~~~~~~~~~~~~~~~~~~~~~
ctx.c:362:27: note: (near initialization for ‘vbdev_ocf_ctx_ops’)
ctx.c:339:53: warning: missing braces around initializer [-Wmissing-braces]
static const struct ocf_ctx_ops vbdev_ocf_ctx_ops = {
^
.name = "OCF SPDK",
{ }
ctx.c:342:16:
.data_alloc = vbdev_ocf_ctx_data_alloc,
{ }
.data_free = vbdev_ocf_ctx_data_free,
{ }
.data_mlock = vbdev_ocf_ctx_data_mlock,
{ }
.data_munlock = vbdev_ocf_ctx_data_munlock,
{ }
ctx.c:386:21: error: variable ‘logger’ has initializer but incomplete type
static const struct ocf_logger logger = {
^~~~~~~~~~
ctx.c:387:3: error: ‘const struct ocf_logger’ has no member named ‘printf’
.printf = vbdev_ocf_ctx_log_printf,
^~~~~~
ctx.c:387:12: warning: excess elements in struct initializer
.printf = vbdev_ocf_ctx_log_printf,
^~~~~~~~~~~~~~~~~~~~~~~~
ctx.c:387:12: note: (near initialization for ‘logger’)
ctx.c:388:3: error: ‘const struct ocf_logger’ has no member named ‘dump_stack’
.dump_stack = NULL,
^~~~~~~~~~
ctx.c:388:16: warning: excess elements in struct initializer
.dump_stack = NULL,
^~~~
ctx.c:388:16: note: (near initialization for ‘logger’)
ctx.c: In function ‘vbdev_ocf_ctx_init’:
ctx.c:396:37: warning: passing argument 2 of ‘ocf_ctx_init’ from incompatible pointer type [-Wincompatible-pointer-types]
ret = ocf_ctx_init(&vbdev_ocf_ctx, &vbdev_ocf_ctx_ops);
^~~~~~~~~~~~~~~~~~
In file included from /home/spdk/lib/bdev/ocf/env/include/ocf/ocf_cache.h:17,
from /home/spdk/lib/bdev/ocf/env/include/ocf/ocf.h:22,
from ctx.c:33:
/home/spdk/lib/bdev/ocf/env/include/ocf/ocf_ctx.h:339:63: note: expected ‘const struct ocf_ctx_config *’ but argument is of type ‘const struct ocf_ctx_ops *’
int ocf_ctx_init(ocf_ctx_t *ctx, const struct ocf_ctx_config *cfg);
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~
ctx.c:401:2: warning: implicit declaration of function ‘ocf_ctx_set_logger’; did you mean ‘ocf_core_set_name’? [-Wimplicit-function-declaration]
ocf_ctx_set_logger(vbdev_ocf_ctx, &logger);
^~~~~~~~~~~~~~~~~~
ocf_core_set_name
ctx.c: At top level:
ctx.c:386:32: error: storage size of ‘logger’ isn’t known
static const struct ocf_logger logger = {
^~~~~~
make[3]: *** [/home/spdk/mk/spdk.common.mk:320: ctx.o] Error 1
make[2]: *** [/home/spdk/mk/spdk.subdirs.mk:44: ocf] Error 2
make[1]: *** [/home/spdk/mk/spdk.subdirs.mk:44: bdev] Error 2
make: *** [/home/spdk/mk/spdk.subdirs.mk:44: lib] Error 2
[root@Target spdk]#

Kernel BUG during cache stopping

Kernel bug occured during cache stop:

[ 4799.934072] Cache cache1 successfully stopped
[ 4799.939197] [Open-CAS] [Classifier] Deinitialized IO classifier
[ 4799.946112] Thread cas_io_cache1_-1 stopped
[ 4799.951269] Thread ocf_metadata_updater_cache1 stopped
[ 4799.957307] ------------[ cut here ]------------
[ 4799.962688] kernel BUG at /root/open-cas-linux/modules/cas_cache/src/ocf/utils/utils_refcnt.c:19!
[ 4799.973045] invalid opcode: 0000 [#1] SMP
[ 4799.977873] Modules linked in: cas_cache(OE) cas_disk(OE) brd nls_utf8 isofs xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache loop sunrpc dm_mirror dm_region_hash dm_log dm_mod sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper iTCO_wdt cryptd iTCO_vendor_support mei_me joydev sg pcspkr mei lpc_ich wmi i2c_i801 ioatdma ipmi_si ipmi_devintf ipmi_msghandler ip_tables ext4 mbcache jbd2 sd_mod
[ 4800.060336] crc_t10dif crct10dif_generic drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops crct10dif_pclmul crct10dif_common ttm isci crc32c_intel igb ahci libsas nvme drm scsi_transport_sas libahci ptp nvme_core libata pps_core drm_panel_orientation_quirks dca i2c_algo_bit [last unloaded: cas_disk]
[ 4800.095851] CPU: 0 PID: 103074 Comm: casadm Kdump: loaded Tainted: G W OE ------------ 3.10.0-957.el7.x86_64 #1
[ 4800.108634] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013
[ 4800.121629] task: ffff9608cffa1040 ti: ffff960760804000 task.ti: ffff960760804000
[ 4800.130406] RIP: 0010:[] [] ocf_refcnt_dec+0x45/0x50 [cas_cache]
[ 4800.141184] RSP: 0018:ffff960760807c18 EFLAGS: 00010297
[ 4800.147350] RAX: 00000000ffffffff RBX: ffffa4acc49dc000 RCX: dead000000000200
[ 4800.155552] RDX: ffff960760807ca0 RSI: 00000000fff0bdbb RDI: ffffa4acc49dc5d0
[ 4800.163748] RBP: ffff960760807c30 R08: ffffa4accc420000 R09: 00000001802a0016
[ 4800.171958] R10: 00000000f6158001 R11: ffffe732c0d85600 R12: 00000000fff0bdbb
[ 4800.180156] R13: ffff960760807ca0 R14: ffffa4accc420000 R15: ffffa4acc4ae4b18
[ 4800.188374] FS: 00007f3a2e465740(0000) GS:ffff960aee000000(0000) knlGS:0000000000000000
[ 4800.197843] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 4800.204475] CR2: 00007f37133dc0a0 CR3: 00000000ae0c6000 CR4: 00000000000607f0
[ 4800.212663] Call Trace:
[ 4800.215618] [] ? ocf_mngt_cache_put+0x20/0x60 [cas_cache]
[ 4800.223716] [] _ocf_mngt_cache_lock_complete+0x85/0x90 [cas_cache]
[ 4800.232887] [] _ocf_async_lock_run_waiters+0x4b/0x70 [cas_cache]
[ 4800.241865] [] ocf_async_lock_deinit+0x8d/0xb0 [cas_cache]
[ 4800.250060] [] ocf_mngt_cache_put+0x3b/0x60 [cas_cache]
[ 4800.257974] [] ocf_queue_put+0x49/0x60 [cas_cache]
[ 4800.265398] [] cache_mngt_exit_instance+0x177/0x260 [cas_cache]
[ 4800.274274] [] cas_service_ioctl_ctrl+0x1039/0x1e50 [cas_cache]
[ 4800.283166] [] ? sched_slice.isra.58+0x66/0xc0
[ 4800.290205] [] ? __enqueue_entity+0x78/0x80
[ 4800.296945] [] ? enqueue_entity+0x2ef/0xbe0
[ 4800.303690] [] ? cpumask_next_and+0x35/0x50
[ 4800.310423] [] ? enqueue_task_fair+0x208/0x6c0
[ 4800.317455] [] ? sched_clock_cpu+0x85/0xc0
[ 4800.324106] [] do_vfs_ioctl+0x3a0/0x5a0
[ 4800.330455] [] ? wake_up_new_task+0x131/0x1a0
[ 4800.337389] [] ? do_fork+0x107/0x320
[ 4800.343463] [] SyS_ioctl+0xa1/0xc0
[ 4800.349348] [] ? system_call_after_swapgs+0xa2/0x146
[ 4800.356957] [] system_call_fastpath+0x22/0x27
[ 4800.363882] [] ? system_call_after_swapgs+0xae/0x146
[ 4800.371487] Code: 75 0e b8 01 00 00 00 f0 0f b1 5f 08 85 c0 75 06 89 d8 5b 5d c3 90 48 8b 57 18 48 8b 47 10 48 89 d7 e8 d0 50 6e ca 89 d8 5b 5d c3 <0f> 0b 66 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 53
[ 4800.394090] RIP [] ocf_refcnt_dec+0x45/0x50 [cas_cache]
[ 4800.402005] RSP

OCF should throttle queue depth during flushing metadata

In current implementation of metadata_io_i_asynch function there is no mechanism to throttle queue depth. This mechanism is needed in environments based on model without interruptions - like SPDK, now in the worth case scenario ocf can double memory consumption (can consume double size of metadata stored on cache device) during flushing metadata to cache device.

In case when we have enough memory to handle situation described above, starting cache still can fail due to io submit fail caused by running out of space in submission queue.

Question about alru cleaning

I have one question about alru cleaning while reading codes about alru cleaning recently. strcut alru_cleaning_policy_meta could be accessed concurrently when doing alru cleaning and when also a write request hits cache and ocf_cleaning_set_hot_cache_line is called to update alru list. So it seems possible that when cleaning thread traverse alru list, the lru_prev or lru_next could be changed in case of cache line hit. I think there should be a lock or something to handle this race condition. But I could not find it in ofc codes. Am I missing anything? Could you please help to explain that?

wildcard support on IO Classification

There is a clear mention in the documentation of how control evection for folders and file extension.
Is it possible to use wildcards and files too? so, we can effectively create exclusion/inclusions in the best fashion as possible?

Need some clarification for promotion feature

In Open CAS there is a feature called "promotion" which didn't existed in Intel CAS, I've went through the documentation and it said "This parameter specifies the desired promotion policy of core lines", but still I don't know what it is and how this parameter would affect core line. Maybe Adding some little extra description of such concept to the "Basic concepts" section of this page : [https://open-cas.github.io/ocf_intro.html](OpenCAS Introduction) would be very helpful?

How can I disable currently used io class file without having to restart the cache?

I specified a io class file for a cache but when I decided to disable it there was not option that would allow me to do so, and I had to stop/start the cache to achieve it, so how can I disable currently used io class file, without having to restart the cache?

root@node195:/etc/opencas# casadm -C -H
Usage: casadm --io-class {--load-config|--list}

Manage IO classes

Loads configuration for IO classes:
Usage: casadm --io-class --load-config --cache-id --file
Options that are valid with --load-config (-C) are:
-i --cache-id Identifier of cache instance <1-16384>
-f --file Configuration file containing IO class definition

Lists currently configured IO classes:
Usage: casadm --io-class --list --cache-id [option...]
Options that are valid with --list (-L) are:
-i --cache-id Identifier of cache instance <1-16384>
-o --output-format Output format: {table|csv}

IO Class Configuration - Allocation

Allocation is an option about the IO Class Configuration in the enterprise edition Intel CAS.
image
But I can't find the same configure option in the open-cas.

/**
 * @brief IO class configuration
 */
struct ocf_mngt_io_class_config {
	/**
	 * @brief IO class ID
	 */
	uint32_t class_id;

	/**
	 * @brief IO class name
	 */
	const char *name;

	/**
	 * @brief IO class eviction priority
	 */
	int16_t prio;

	/**
	 * @brief IO class cache mode
	 */
	ocf_cache_mode_t cache_mode;

	/**
	 * @brief IO class minimum size
	 */
	uint32_t min_size;

	/**
	 * @brief IO class maximum size
	 */
	uint32_t max_size;
};
  1. Does open-cas support the allocation option?
  2. What means about the allocation options of the IO class ?

Errors during test_negative_io test suite execution

Multiple problems occurred. High reproduction rate

  • test_neg_write_too_long_data - SIGABRT
  • test_neg_read_too_long_data - SIGSEGV
  • test_neg_write_too_far - SIGABRT
  • test_neg_read_too_far - SIGSEGV
  • test_neg_write_offset_outside_of_device - fails for any input (TypeError)
  • test_neg_read_offset_outside_of_device - fails for any input (TypeError)

Pass-through write and read miss race condition

There's a possible race condition between pass-through write and cache insert. When PT write and read insert are submitted to the same LBA at the same time it's possible all further read hits to given LBA would return old data instead of data written by PT write request.

Race condition scenario:

Time     Pass-through Write                  Cache insert read
  |
  |      Traversation (->miss)
  |           :                              Traversation (->miss)
  |           :                              Mapping
  |           :                              Read from core
  |           :                                   :
  |      Data written to the device               :
  |
  |
  v

Key conditions:

  • Read mapping must happen after write traversation
  • Read from core device must happen before write to the core device

It's easily reproducible by artificially slowing down writes to core device in bottom adapter.

Test script (for open-cas-linux):

#!/bin/bash

set -ex

CACHE=/dev/nvme0n1p1
CORE=/dev/sdb1
S=50000

dd if=/dev/urandom of=base1.bin bs=4k count=1
dd if=/dev/urandom of=base2.bin bs=4k count=1

# 1. Write base pattern to core (base1.bin)
dd if=base1.bin of=${CORE} seek=${S} bs=4k count=1 oflag=direct

casadm -S -i 1 -d ${CACHE} -c wa -f
casadm -A -i 1 -d ${CORE}

# 2. Overwrite pattern on cas dev. Goes in PT mode. (base2.bin)
dd if=base2.bin of=/dev/cas1-1 seek=${S} bs=4k count=1 oflag=direct &

# 3. Read the same LBA in parallel while Open-CAS/open-cas-linux#2 is in progress (out1.bin)
sleep 3
dd if=/dev/cas1-1 of=out1.bin skip=${S} bs=4k count=1 iflag=direct

wait

# 4. Reread the same LBA after write is finished (out2.bin)
dd if=/dev/cas1-1 of=out2.bin skip=${S} bs=4k count=1 iflag=direct


casadm -T -i 1 -n

# 5. Reread the same LBA after cache stop, directly from core (out3.bin)
dd if=${CORE} of=out3.bin skip=${S} bs=4k count=1 iflag=direct

md5sum base1.bin base2.bin out1.bin out2.bin out3.bin

# Expected checksums:
# base1 - A
# base2 - B
# out1  - A or B
# out2  - B
# out3  - B

# Actual checksums:
# base1 - A
# base2 - B
# out1  - A
# out2  - A -> wrong
# out3  - B

Metadata initialization fails in SPDK (volatile = true)

Starting cache with configure option metadata_volatile = true fails with:

MalCache: Metadata Error
MalCache: Metadata Flush ERROR
MalCache: ERROR: Cannot save cache state
MalCache: Attaching cache device failed

Full log

Inserting cache MalCache
MalCache: Successfully added
MalCache: Cache mode : wb
MalCache: Super block config offset : 0 kiB
MalCache: Super block config size : 3544 B
MalCache: Super block runtime offset : 0 kiB
MalCache: Super block runtime size : 1512 B
MalCache: Reserved offset : 0 kiB
MalCache: Reserved size : 128 kiB
MalCache: Core config offset : 0 kiB
MalCache: Core config size : 360 kiB
MalCache: Core runtime offset : 0 kiB
MalCache: Core runtime size : 1172 kiB
MalCache: Core UUID offset : 0 kiB
MalCache: Core UUID size : 16384 kiB
MalCache: Cleaning offset : 0 kiB
MalCache: Cleaning size : 604 kiB
MalCache: Eviction offset : 0 kiB
MalCache: Eviction size : 400 kiB
MalCache: Collision offset : 0 kiB
MalCache: Collision size : 604 kiB
MalCache: List info offset : 0 kiB
MalCache: List info size : 856 kiB
MalCache: Hash offset : 0 kiB
MalCache: Hash size : 68 kiB
MalCache: Cache line size: 4 kiB
MalCache: Metadata capacity: 36 MiB
MalCache: OCF metadata self-test PASSED
MalCache: Metadata Error
MalCache: Metadata Flush ERROR
MalCache: ERROR: Cannot save cache state
MalCache: Attaching cache device failed
vbdev_ocf.c: 767:start_cache_cmpl: *ERROR*: Failed to attach cache device: -1000017
MalCache: Metadata Error
MalCache: Adding core core1 failed
vbdev_ocf.c: 906:add_core_cmpl: *ERROR*: Failed to add core device to cache instance

Messages printed during ocf_metadata_hash_load_superblock operation
pipeline->error gets set in ocf_metadata_hash_generic_complete to -95
Backtrace:

(gdb) bt
#0  ocf_pipeline_finish (pipeline=0x200027c0b780, error=-95) at src/ocf/utils/utils_pipeline.c:123
#1  0x00005555555e9c37 in ocf_metadata_hash_generic_complete (priv=0x200027c0b7a8, error=-95) at src/ocf/metadata/metadata_hash.c:892
#2  0x00005555555f25d0 in raw_volatile_flush_all (cache=0x2000142273c0, raw=0x20001c2002d8, cmpl=0x5555555e9c05 <ocf_metadata_hash_generic_complete>, priv=0x200027c0b7a8) at src/ocf/metadata/metadata_raw_volatile.c:45
#3  0x00005555555e81e7 in ocf_metadata_raw_flush_all (cache=0x2000142273c0, raw=0x20001c2002d8, cmpl=0x5555555e9c05 <ocf_metadata_hash_generic_complete>, priv=0x200027c0b7a8) at src/ocf/metadata/metadata_raw.h:293
#4  0x00005555555ea5b2 in ocf_medatata_hash_flush_segment (pipeline=0x200027c0b780, priv=0x200027c0b7a8, arg=0x55555594b620 <ocf_metadata_hash_flush_sb_flush_segment_args>) at src/ocf/metadata/metadata_hash.c:1161
#5  0x00005555555f8227 in _ocf_pipeline_run_step (req=0x200014627f00) at src/ocf/utils/utils_pipeline.c:44
#6  0x00005555555e2c7f in ocf_io_handle (io=0x0, opaque=0x200014627f00) at src/ocf/ocf_queue.c:74
#7  0x00005555555e2d7d in ocf_queue_run_single (q=0x20001c2001c0) at src/ocf/ocf_queue.c:94
#8  0x00005555555b84a8 in mngt_queue_poll (opaque=0x20001c2001c0) at vbdev_ocf.c:796
#9  0x0000555555657520 in spdk_thread_poll (thread=0x555555dc2800, max_msgs=0, now=6263853433141732) at thread.c:456
#10 0x0000555555614b56 in _spdk_reactor_run (arg=0x555555dc25c0) at reactor.c:244
#11 0x0000555555614e78 in spdk_reactors_start () at reactor.c:333
#12 0x00005555556136fb in spdk_app_start (opts=0x7fffffffe270, start_fn=0x555555564450 <spdk_startup>, arg1=0x0) at app.c:677
#13 0x00005555555645e5 in main (argc=3, argv=0x7fffffffe408) at iscsi_tgt.c:112

Tested on d88a4ac and previus,
the issue reproduces precisely after 6cd8447

SPDK version (devel): https://review.gerrithub.io/c/spdk/spdk/+/448537/9

Bottom devices are RAM regions (malloc bdevs)

Unknown type error when building ocf spdk bdev

Hello
I am getting the following error when I try to build SPDK with ocf (--with-ocf option). I have tried to build with the ocf module in SPDK and also downloading the framework from the ocf git repository and I got the same error for both. I also did a build with ocf 18.12 as suggested in one of the other issues posted here and got the same error.

In file included from src/ocf/cleaning/nop.c:7:0:
src/ocf/cleaning/nop.h:12:55: error: unknown type name ‘ocf_cleaner_end_t’; did you mean ‘ocf_cleaner_t’?
void cleaning_nop_perform_cleaning(ocf_cache_t cache, ocf_cleaner_end_t cmpl);
^~~~~~~~~~~~~~~~~
ocf_cleaner_t
src/ocf/cleaning/nop.c:10:55: error: unknown type name ‘ocf_cleaner_end_t’; did you mean ‘ocf_cleaner_t’?
void cleaning_nop_perform_cleaning(ocf_cache_t cache, ocf_cleaner_end_t cmpl)
^~~~~~~~~~~~~~~~~
ocf_cleaner_t
/home/john/spdk/mk/spdk.common.mk:330: recipe for target 'src/ocf/cleaning/nop.o' failed
make[3]: *** [src/ocf/cleaning/nop.o] Error 1
/home/john/spdk/mk/spdk.subdirs.mk:44: recipe for target 'ocf/env' failed
make[2]: *** [ocf/env] Error 2
/home/john/spdk/mk/spdk.subdirs.mk:44: recipe for target 'bdev' failed
make[1]: *** [bdev] Error 2
/home/john/spdk/mk/spdk.subdirs.mk:44: recipe for target 'lib' failed
make: *** [lib] Error 2

CC lib/bdev/ocf/env/src/ocf/cleaning/nop.o In file included from src/ocf/cleaning/nop.c:7:0: src/ocf/cleaning/nop.h:12:55: error: unknown type name ‘ocf_cleaner_end_t’; did you mean ‘ocf_cleaner_t’? void cleaning_nop_perform_cleaning(ocf_cache_t cache, ocf_cleaner_end_t cmpl); ^~~~~~~~~~~~~~~~~ ocf_cleaner_t src/ocf/cleaning/nop.c:10:55: error: unknown type name ‘ocf_cleaner_end_t’; did you mean ‘ocf_cleaner_t’? void cleaning_nop_perform_cleaning(ocf_cache_t cache, ocf_cleaner_end_t cmpl) ^~~~~~~~~~~~~~~~~ ocf_cleaner_t /home/john/spdk/mk/spdk.common.mk:330: recipe for target 'src/ocf/cleaning/nop.o' failed make[3]: *** [src/ocf/cleaning/nop.o] Error 1 /home/john/spdk/mk/spdk.subdirs.mk:44: recipe for target 'ocf/env' failed make[2]: *** [ocf/env] Error 2 /home/john/spdk/mk/spdk.subdirs.mk:44: recipe for target 'bdev' failed make[1]: *** [bdev] Error 2 /home/john/spdk/mk/spdk.subdirs.mk:44: recipe for target 'lib' failed make: *** [lib] Error 2

How to make data in a certain range permanently resident in cache?

Suppose I had a core device /dev/core that contains some metadata at 0-1GB, and I wanted this 1GB metadata to be always resident in the cache, how should I make it happen?

I've thought about using IO classification, but it seemed that the only available IO class name for this scenario was “lba”, so I don't know if it is the answer to my problem, and if it is, how should I write the classification configuration file?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.