philippelt / microk8s_prune Goto Github PK

Containerd prune utility to cleanup loaded but unused images. This utility is targeting microk8s release but can easily be adapted to any containerd installation.

License: GNU General Public License v3.0

Python 100.00%

kubernetes containerd microk8s administration prune

microk8s_prune's Introduction

microk8s_prune

Official tool: For cleanup, the crictl tool from cri-tools have now a rmi --prune feature. Thanks to Franco Martin (https://discuss.kubernetes.io/t/microk8s-images-prune-utility-for-production-servers/15874/2) for the info

New: selected output results in JSON format for easy pipe reuse

As you can expect, this small utility is doing for a microk8s / containerd environment the same operation that docker system prune does : it search for images unused by container to list them and/or delete them to recover space

You will need a python3 running environnment and the containerd library.

Note: Protobuf project, used by containerd through grpc layer, introduced a breaking change between 3.x and 4.x version, mainly for performance gains. The containerd python library is not yet available in a compatible version with protobuf 4.x. Thus it is recommended to stay with 3.x (ideally the last release before 4.x, at time of writing, 3.20.3). Unless your are explicitely using protobuf 4.x in your applications that would create installation conflict, this will not have undesirable effects on the utility behavior. Would you need elsewhere protobuf v4.x, it is always possible to run the utility in a small dedicated container (mounting the sock, see below, as a volume).

if you chmod +x the utility, just run it with :

./microk8s_prune.py [ciuspfj]

Parameters:

c : List containers and associated images (line output prefixed with C:)
i : List all found images names (prefixed with I:)
u : List unused images names (prefixed with U:)
s : Print summary information (# containers, total images, unused images)
p : DELETE unused images (will require confirmation if run interactively)
f : Force delete without confirmation
j : All requested output in json format for pipe reuse

Without any parameter, it will do the job of identifying unused images but do not display anything nor delete anything. Without p and/or f parameter, the utility is strictly read-only.

The ctr utility to manage containerd do not provide such feature and considering that microk8s is especially usefull for small appliances where resources are limited I think it would be nice to have it.

In the meantime, this utility will solve the problem 😊

This utility could be used in non microk8s containerd environment, just replace the namespace which is harcoded to k8s.io and reference the appropriate sock in place of the microk8s one unix:///var/snap/microk8s/common/run/containerd.sock

microk8s_prune's People

Contributors

Stargazers

Watchers

Forkers

m8e tfmark wutiarn lezli01 klamas1 shinodark halloyusufnurfaisal koss822 raj-adroit sachajw

microk8s_prune's Issues

Received message larger than max (4554417 vs. 4194304) with large number of containers

Firstly: thanks for this script!

I ran it on my UAT box and everything was great, then tried against PROD (which has many more containers) and I got the following error:

root@prod:~# python3 /tmp/microk8s_prune.py s
Traceback (most recent call last):
  File "/tmp/microk8s_prune.py", line 62, in <module>
    containers = containersv1.List( containers_pb2.ListContainersRequest(),
  File "/usr/local/lib/python3.8/dist-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.8/dist-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.RESOURCE_EXHAUSTED
        details = "Received message larger than max (4554417 vs. 4194304)"
        debug_error_string = "{"created":"@1630400883.201938367","description":"Received message larger than max (4554417 vs. 4194304)","file":"src/core/ext/filters/message_size/message_size_filter.cc","file_line":206,"grpc_status":8}"
>

After some Googling I found that this can be remedied by passing the message size limit when creating the channel:

root@prod-jupyter-vm-1:~# python3 /tmp/microk8s_prune.py s
S: 409 containers
S: 30 total images
S: 4.6 GB (4889119653 bytes) total images size, 3.0 GB shared
S: 12 used images
S: 20 unused images

So I will raise a PR in a few minutes.

Protocol buffers error

Nice tool, just what I needed for my older microk8s installation

Initially I got error

$ pip install containerd
$ ./microk8s_prune.py
Traceback (most recent call last):
  File "/home/ubuntu/./microk8s_prune.py", line 7, in <module>
    from containerd.services.containers.v1 import containers_pb2_grpc, containers_pb2
  File "/home/ubuntu/.local/lib/python3.10/site-packages/containerd/services/containers/v1/containers_pb2_grpc.py", line 5, in <module>
    from containerd.services.containers.v1 import containers_pb2 as containerd_dot_services_dot_containers_dot_v1_dot_containers__pb2
  File "/home/ubuntu/.local/lib/python3.10/site-packages/containerd/services/containers/v1/containers_pb2.py", line 14, in <module>
    from containerd.vendor.gogoproto import gogo_pb2 as containerd_dot_vendor_dot_gogoproto_dot_gogo__pb2
  File "/home/ubuntu/.local/lib/python3.10/site-packages/containerd/vendor/gogoproto/gogo_pb2.py", line 29, in <module>
    goproto_enum_prefix = _descriptor.FieldDescriptor(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/google/protobuf/descriptor.py", line 560, in __new__
    _message.Message._CheckCalledFromGeneratedFile()
TypeError: Descriptors cannot not be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
 1. Downgrade the protobuf package to 3.20.x or lower.
 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).

pip install protobuf==3.20.* solved it. I don't know python really but maybe the script can be updated instead?

philippelt / microk8s_prune Goto Github PK

microk8s_prune's Introduction

microk8s_prune

microk8s_prune's People

Contributors

Stargazers

Watchers

Forkers

microk8s_prune's Issues

Received message larger than max (4554417 vs. 4194304) with large number of containers

Protocol buffers error

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent