How I know that write operation is success when I use RPMA_F_COMPLETION_ON_ERROR about rpma HOT 5 CLOSED

Hi @hualongfeng

There needs rpma_conn_completion_wait and rpma_conn_completion_get when I use rpma_write with RPMA_F_COMPLETION_ALWAYS. But When I use RPMA_F_COMPLETION_ON_ERROR, I don't need to wait the completion status. So how I know the operation is success or failed?

When you use RPMA_F_COMPLETION_ON_ERROR you indicate you do not want to have a completion of this particular operation. If you want to get a completion of this exact operation you should use RPMA_F_COMPLETION_ALWAYS.

But you can indicate the operation is completed successfully by collecting a completion of an operation which comes after the one you are wanting the completion work out (*). This is related to your other questions.

After I continuously perform the rma_write operation with RPMA_F_COMPLETION_ON_ERROR, and then use rma_read with RPMA_F_COMPLETION_ALWAYS, then if rma_read returns success, it means that the previous rma_write is also successful? And is the order of these operations strictly executed in the order in which I called them? In other words, the rpma_read operation must be executed after the rpma_write is completely completed?

(*) The operations are executed in order of posting but it does not imply the completion of operation B indicates the operation A I completed. This depends on the RDMA ordering rules in general. You can learn about it here: https://tools.ietf.org/html/rfc5040#page-59

RPMA_F_COMPLETION_ON_ERROR is useful when you want to send multiple operations and wait just for the completion of the last one which may indicate the completion of all of them. It is a way of optimizing the pipeline.

@grom72 Maybe it is a good idea to include this information in the librpma documentation.

from rpma.

@janekmi Now I imitate the code of 05-flush-to-persistent in the example, the rpma_write() operation uses RPMA_F_COMPLETION_ON_ERROR, and the rpma_flush() operation uses the RPMA_F_COMPLETION_ALWAYS operation, and then the rpma_conn_completion_wait and rpma_conn_completion_get operations confirm that the rpma_flush() operation is successful, then this Does success mean that the operation of rpma_write is successful? In other words, I must set the RPMA_F_COMPLETION_ALWAYS flag for these two operations(rpam_write and rpma_flush) to ensure that my write and flush are both successful?

from rpma.

@janekmi Now I imitate the code of 05-flush-to-persistent in the example, the rpma_write() operation uses RPMA_F_COMPLETION_ON_ERROR, and the rpma_flush() operation uses the RPMA_F_COMPLETION_ALWAYS operation, and then the rpma_conn_completion_wait and rpma_conn_completion_get operations confirm that the rpma_flush() operation is successful, then this Does success mean that the operation of rpma_write is successful? In other words, I must set the RPMA_F_COMPLETION_ALWAYS flag for these two operations(rpam_write and rpma_flush) to ensure that my write and flush are both successful?

Hi @janekmi, I looked at the implementation of rpma_flush, it used the rpma_mr_read operation and read 8bytes of data. How does this operation determine that the data has been flushed? Thanks.

from rpma.

@hualongfeng please see Whitepaper when read after write technology for data flushing is described.
https://software.intel.com/content/www/us/en/develop/articles/persistent-memory-replication-over-traditional-rdma-part-1-understanding-remote-persistent.html

from rpma.

@hualongfeng is there anything else to be clarified in this issue or we can close it?

from rpma.