Add Commit Hooks to write extra information within the same transaction about qbeast-spark HOT 2 CLOSED

If you could provide any other necessary info regarding this enhancement would be great @Jiaweihu08 @cugni

from qbeast-spark.

More details about the implementation in #319

The idea is to have an interface such as:

/**
 * A hook that can be run before a commit
 */
trait QbeastPreCommitHook {

  /**
   * The name of the hook
   */
  val name: String

  type PreCommitHookResponse = Map[String, String]

  /**
   * Run the hook with the given actions
   *
   * @param actions
   *   the actions to run the hook with
   */
  def run(actions: Seq[Action]): PreCommitHookResponse
}

Users can extend that interface and add their code to run pre-commit checks or actions. Those actions need to return a Map[String, String] of information they want to write on the Commit Log about the execution of the hook.

Registering a PreCommitHook

In terms of usage from the DataFrame API, one user may call the hook as:

df.write
.format("qbeast")
.option("qbeast.preCommitHook.my_hook_1", "my.hook.class.path")
.option("qbeast.preCommitHook.my_hook_1.arg1", "first_argument")
.save(...)

In that way, we will identify each hook by its name on the options.

Output of a PreCommitHook

Each implementation of a PreCommitHook must return a Map[String, String] with any useful information to register in the Commit Log. Qbeast will add the prefix registered in the options to the result map keys, and it will save it on the tags variable in the CommitInfo.

"commitInfo" {
    "tags" { 
          "qbeast.preCommitHook.my_hook_1.write_id":"id",
          "qbeast.preCommitHook.my_hook_1.cpu_time":"30ms"
     }
 }

from qbeast-spark.