When the chaincode pod is created by the k8s builder, it does not currently set the pod owner attribute.
https://kubernetes.io/docs/concepts/overview/working-with-objects/owners-dependents/

Update the owner reference for the child pod such that it is set as a dependent resource created by the peer. This will force k8s to run an automatic GC of the chaincode pod if/when the peer is terminated.

Some original context and thread was discussed over on discord at #fabric-kubernetes, reported by @alius.miles

Ideally the k8s pod should be terminated if the peer is terminated. This is related to #88 however it will not always be possible to add an owner reference.

From v2.5.0, Fabric switches base docker images from golang-alpine to ubuntu 20.
The k8s builder will need to follow this update.

Related Issue

hyperledger/fabric-samples#840 (comment)

test-network-k8s includes support for the k8s builder.
In the current implementation, test-network-k8s internally copies Go executables of the builder in k8s-fabric-peer to peer pods.
When using Fabric v2.5, the peer pods on Ubuntu try to run the Go executables created in Alpine, but due to this distribution mismatch, the executable cannot be run ('no such file or directory' error occurs).

If the k8s fabric peer image is updated based on Ubuntu, test-network-k8s will be able to support Fabric 2.5 with k8s builder with fewer modifications.

Chaincode pods are currently created without specifying a service account which means that they are automatically assigned the default service account. To pull chaincode images this service account may require an image pull secret to be defined - see #65 - which may not be desirable on the default service account.

Add a new FABRIC_K8S_BUILDER_SERVICE_ACCOUNT option similar to the existing FABRIC_K8S_BUILDER_NAMESPACE to allow a different service account to be specified if required.

There are currently some minimal unit tests for the detect, build, release and run commands but it would definitely be good it improve the unit test coverage!

The chaincode PODS don ´t have defined the resource limits.

Those limits can be configured in:

1. Chaincode Artifacts - image.json;
2. Environment Variables;
3. etc?

Hey, @jt-nti I explored this repo and find it interesting and want to contribute to this. I am DevOps and a full-stack web3 developer. Can u please suggest to me some First-time contributions?
Thank You !

Based on discussions with @y12studio in #98, it would be useful to support liveness probes for chaincode containers.

I don't think there is a simple way to implement this with current Fabric capabilities since the peer<->chaincode communication is via a long-lived grpc stream, and I'm not sure what chaincode language neutral alternatives might be possible instead.

One possibility is to update the Fabric protos to add a liveness service for chaincode implementations to (optionally?) support this requirement. The peer uses a separate operations service for health checks but that doesn't seem like a good approach for chaincode.

It may be desirable to deploy different chaincode in different namespaces and, with #102, with different resource requests and limits.

Currently the chaincode packages only contain information about what is being deployed, not how to deploy it, and I am keen to keep this distinction. For example, requiring a new package to go through the chaincode lifecycle to change resource limits is not ideal. It would also reduce the likelihood of different organisations being able to share the same chaincode package.

it can point to another one, if it's a copy of another

The chaincode pod imagePullSecret is not specified, so any credentials used to load the container need to be added to the service account running the peer.

It appears that the folders needed by the private data is not being copied during release:

https://github.com/hyperledger-labs/fabric-builder-k8s/blob/main/internal/util/copy.go#L16-L40

Private Data Indexes with private data:
https://hyperledger-fabric.readthedocs.io/en/latest/private_data_tutorial.html#using-indexes-with-private-data

Add a makefile with instructions for build, test and linting the project

The builder needs to use suitably unique names for the kubernetes "stuff" required for running chaincode

The run binary has access to the unique ID associated with the chaincode package, which is in the form of label:hash, the address of the peer, which is in the form of host:port, and the local mspid of the peer.

The builder can also use environment variables which are configured in core.yaml, for example CORE_PEER_ID can be propagated to the builder, and currently is required. Using the peer address would reduce the configuration required and may be preferable.

Important: whatever combination is used needs to be sanitised and truncated to give a valid kubernetes object name, and the naming rules differ between objects. For example the peer ID can be relatively long, e.g. http://peer0.org1.example.com/ in the nano test network, and the same for the chaincode label, plus the chaincode ID includes a full hash. All these will need to be truncated and have any invalid characters removed or replaced.

Investigate whether the //nolint:gochecknoglobals overrides in the cmd test suites can be avoided

Will require code of conduct, maintainers, etc.

See https://github.com/jt-nti/hyperledger-labs.github.io/blob/k8s-builder/labs/fabric-builder-k8s.md

The run command currently runs indefinitely which may cause problems if the chaincode terminates and the peer needs to restart it

Currently haven't found anything that supports private images. So when the chaincode pod is created it shows an error. It would be possible to implement support for private images using "imagePullSecrets" as a parameter so that we can inform the credential of the image's private repository.

Need to check whether the builder is running the chaincode the best way.

The prototype creates a deployment since the Kubernetes documentation states...

Usually you don't need to create Pods directly, even singleton Pods. Instead, create them using workload resources such as Deployment

This would almost certainly be the correct approach if the builder was able to start the chaincode as an external service, however the current Fabric builder implementation means that is not possible (see below).

Since the peer is still managing the chaincode workload in the traditional chaincode as a client case, a Pod may be the right option instead of a Deployment.

Note: the builder run executable should not exit while the chaincode is still running. The current prototype code just has a hack to stop the run executable exiting at all but it should be checking whether the chaincode is running and exiting if it has stopped.

It would be nice to fix the Fabric builder implementation to allow the k8s builder to run the chaincode as an external service. This is currently problematic because the builder is expected to provide the chaincode ID and TLS certificates in the release executable, which are actually provided to the builder's run executable... except this is not called for chaincode as an external service. The builder might be able to get the TLS certificates from the filesystem if it knew where to look, e.g. using a propagated environment variable, however the chaincode ID is not available. A possible fix might be to modify the Fabric builder implementation so that the run binary is optionally called even for chaincode as an external service.

Example: https://github.com/hyperledger-labs/solang/blob/main/MAINTAINERS.md Example of a pointer: https://github.com/hyperledger-labs/homebrew-solang/blob/main/MAINTAINERS.md

The .github/workflows for building, preparing, and publishing chaincode images and packages are awesome. Awesome. 💯 ❤️

The images created by the build could use a little bit of context. For instance, the contract-java and contract-node images are being published to the container registry as:

• ghcr.io/hyperledger-labs/contract-java@sha256:....
• ghcr.io/hyperledger-labs/contract-node@sha256:...

I don't think that is correct. The image output paths should include the application/repo/folder/etc. context in the URLs.

I was able to change the path for the published images by passing the full path, e.g.:

jobs:
  docker_build:
    name: Docker build
    uses: ./.github/workflows/docker-build.yaml
    with:
      imagename: full-stack-asset-transfer-guide/contracts/asset-tx-typescript
      path: contracts/asset-tx-typescript
      chaincode: true

With this update, publish steps upload the image correctly as ghcr.io/hyperledgendary/full-stack-asset-transfer-guide/contracts/asset-tx-typescript@sha256:.... This is good.

But when passing the full path in for the imagename attribute to the package-k8s-chaincode-action, the ${imagename}.tgz package output can not be found, as (I think) it's being generated up at the root folder in the hierarchy. Or the first path element is not available, or ... something. I think where this starts to go off the rails is where docker-build.yaml is passing ${{inputs.imagename}} in for chaincode-label to the package action.

There are limits on pod names which led to the current human unfriendly names in #16. While there are labels and annotations to help manage the chaincode pods, it would be good to improve the pod names as well.

Create a fabric network configured with the builder and check that k8s builder chaincode can be used successfully

The chaincode PODS don ´t have defined the resource limits.

Those limits can be configured in:

1. Chaincode Artifacts - image.json;
2. Environment Variables;
3. etc?

MSP and peer ID annotations look like they might cause problems. Also potentially related to #88, can the object labels be improved to make management easier?

Valid label values:

• must be 63 characters or less (can be empty),
• unless empty, must begin and end with an alphanumeric character ([a-z0-9A-Z]),
• could contain dashes (-), underscores (_), dots (.), and alphanumerics between.

Possible label changes:

The k8s-fabric-peer images are based on hyperledger/fabric-peer which do not publish arm versions, so things are very broken.

Options:

• wait until Fabric support arm
• build a completely custom peer image

Log details of the pod that was started to make it easier to find in case there are problems starting the chaincode container

There are currently a random selection of print statements which isn't great!

There are currently some docs in the form of a readme and markdown files in the docs folder, however it would be great to have a more user friendly doc site

For example the Hyperledger Besu project uses the MKDOCS_MATERIAL material docs insider:

• https://github.com/hyperledger/besu-docs/blob/d63f1467fa4acc4e807bb34fa34a03d560930b64/mkdocs-insider.yml
• hyperledger/besu-docs#1113
• https://squidfunk.github.io/mkdocs-material/

Hi everyone, I hope everyone is doing well. About the k8s-builder project I came across an important question regarding the arch that the environment was provisioned. In a scenario where we have 3 orgs with the following distribution Org1 was provisioned in an arch:AMD64 environment, Org2 also in an arch:AMD64 environment, while Org3 was provisioned in an arch:ARM64 environment. In this scenario using k8s-builder the digest of the chaincode image would be only 1. Since they are environments with different arch each image will have a different digest AMD64 images will have a digest while ARM64 images will have another digest purely because they are different architectures .
Knowing this difference an AMD64 image will not work in an ARM64 environment and vice versa, the version of k8s-builder currently does not support image parameter per arch, on this point is there any workaround for using k8s-builder in such a scenario Similar to the one mentioned above?
In our reviews we actually saw that this would have to be implemented for it to work correctly. So that all chaincode packages have the same ID when they are installed in any environment be it AMD64 or ARM64 or any other architecture.
Have you already checked something about and or is there any intention to support multiple architectures with this project, since FABRIC itself in its 2.5.x version started to support ARM images by default?

The contents of the chaincode package, i.e. the image.json file, must only contain details about the chaincode "inside" the package, and not any deployment configuration.

In the case of the k8s builder the chaincode is not actually inside the package, but in order not to break the Fabric chaincode lifecycle, the contents of the package must uniquely identify the chaincode that will eventually run.

An image digest does that, but a mutable tag does not.

Unfortunately that definitely causes problems for chaincode which hasn't been pushed to a repository, which potentially makes chaincode development more difficult. The chaincode as a server (CCaaS) builder is a better option for development environments, since it breaks the link between updates to the code and the chaincode lifecycle, and makes it easy to attach a debugger.

If there is feedback that there are significant usability problems with the CCaaS approach, then it may be worthwhile adding an unsafe development mode to the k8s builder to enable it to use unpublished chaincode images.

See #27 for an initial investigation.

• test-network-k8s
• test-network-nano-bash
• fabric-operator
• hlf-operator
• microfab
• hyperledger-fabric-operator
• hl-fabric-operator
• fabric-operator

It would be good to update now that Fabric 2.5 has been released, and hopefully this should make arm support easier.

Starting to think about what needs doing to get to a v1 release. I would like to get the following done, in vague priority order...

• Optionally set service account of chaincode pods #76
• Add end to end tests #48
• Improve test coverage #28
• New k8s-fabric-peer arm images do not work #74
• Document required permissions in readme #52
• Document pulling images from registries which require credentials #65
• Graduate from labs project hyperledger/fabric-rfcs#56
• Define desired kubernetes object names #16

Does that sound reasonable? Is there anything missing?

The save-state command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/

The set-output command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/

Node.js 12 actions are deprecated. For more information see: https://github.blog/changelog/2022-09-22-github-actions-all-actions-will-begin-running-on-node16-instead-of-node12/. Please update the following actions to use Node.js 16: softprops/action-gh-release@v1

The following is more than strictly necessary but a good starting point...

---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: fabric-builder-k8s-role
rules:
  - apiGroups:
      - ""
      - apps
    resources:
      - pods
      - deployments
      - configmaps
      - secrets
    verbs:
      - get
      - list
      - watch
      - create
      - delete
      - patch

Need to document the minimum verbs required for each resource and how to apply the permissions to the peer's service account.

Include example kubectl auth can-i commands to check for the required permissions.

It might help people migrate to the k8s builder if there was a tool to create a template project and seed it with the contents of an existing chaincode package.

Traditional Go, Java, and Node chaincode packages contain the smart contract source code which is compiled by a chaincode builder container. (Note: Java packages may only contain prebuilt jars in some cases.)

Packages for the new lifecycle are just tarballs, and packages for the old lifecycle are protobuf based- see cds-cli for an example of how to read those.

A new repo would be required for the new tool but opening this issue here for discussion.

Docker tags are mutable which can cause problems if the tag is changed accidentally or maliciously.

To get the digest...

docker inspect ghcr.io/OWNER/IMAGE_NAME

Using both the tag (which would be ignored) and digest in the current image.json would probably work already, e.g.

{
  "name": "ghcr.io/OWNER/IMAGE_NAME",
  "tag": "TAG@sha256:82jf9a84u29hiasldj289498uhois8498hjs29hkuhs"
}

Enforce digests in the future by updating the image.json file to include digest instead of tag, e.g.

{
  "name": "ghcr.io/OWNER/IMAGE_NAME",
  "digest": "82jf9a84u29hiasldj289498uhois8498hjs29hkuhs"
}

There shouldn't be any secrets in the chaincode package- these should be part of the Fabric deployment. It may be useful to provide the name of the secret to use in the chaincode package though- tbc.

Similarly, imagePullPolicy seems like an optional configuration option for the builder, rather than something which belongs in the chaincode package.

I previously wondered about a configuration option to provide a kubernetes .yaml file to allow more fine grained control rather than exposing individual options. Probably needs a bit more thought though.

It would be nice to publish (working) arm docker images. This is probably easiest to achieve with a move to Fabric 2.5 when it becomes the LTS release.