Code Monkey home page Code Monkey logo

autopilot-log-collector's Introduction

Juju logo next to the text Canonical Juju

Juju is an open source application orchestration engine that enables any application operation (deployment, integration, lifecycle management) on any infrastructure (Kubernetes or otherwise) at any scale (development or production) in the same easy way (typically, one line of code), through special operators called β€˜charms’.

juju snap build

πŸ‘‰ Juju Learn how to quickly deploy, integrate, and manage charms on any cloud with Juju.
It's as simple as juju deploy foo, juju integrate foo bar, ..., on any cloud.
Charmhub Sample our existing charms on Charmhub.
A charm can be a cluster (OpenStack, Kubernetes), a data platform (PostgreSQL, MongoDB, etc.), an observability stack (Canonical Observability Stack), an MLOps solution (Kubeflow), and so much more.
Charm SDK Write your own charm!
Juju is written in Go, but our SDK supports easy charm development in Python.

Give it a try!

Let's use Juju to deploy, configure, and integrate some Kubernetes charms:

Set up

You will need a cloud and Juju. The quickest way is to use a Multipass VM launched with the charm-dev blueprint.

Install Multipass: Linux | macOS | Windows. On Linux:

sudo snap install multipass

Use Multipass to launch an Ubuntu VM with the charm-dev blueprint:

multipass launch --cpus 4 --memory 8G --disk 30G --name tutorial-vm charm-dev 

Open a shell into the VM:

multipass shell tutorial-vm

Verify that you have Juju and two localhost clouds:

juju clouds

Bootstrap a Juju controller into the MicroK8s cloud:

juju bootstrap microk8s tutorial-controller

Add a workspace, or 'model':

juju add-model tutorial-model

Deploy, configure, and integrate a few things

Deploy Mattermost:

juju deploy mattermost-k8s

See more: Charmhub | mattermost-k8s

Deploy PostgreSQL:

juju deploy postgresql-k8s --channel 14/stable --trust

See more: Charmhub | postgresql-k8s

Enable security in your PostgreSQL deployment:

juju deploy tls-certificates-operator
juju config tls-certificates-operator generate-self-signed-certificates="true" ca-common-name="Test CA"
juju integrate postgresql-k8s tls-certificates-operator

Integrate Mattermost with PostgreSQL:

juju integrate mattermost-k8s postgresql-k8s:db

Watch your deployment come to life:

juju status --watch 1s

(Press Ctrl-C to quit. Drop the --watch 1s flag to get the status statically. Use the --relations flag to view more information about your integrations.)

Test your deployment

When everything is in active or idle status, note the IP address and port of Mattermost and pass them to curl:

curl <IP address>:<port>/api/v4/system/ping

You should see the output below:

{"AndroidLatestVersion":"","AndroidMinVersion":"","IosLatestVersion":"","IosMinVersion":"","status":"OK"}

Congratulations!

You now have a Kubernetes deployment consisting of a Mattermost backed by PosgreSQL with TLS-encrypted traffic!

Clean up

Delete your Multipass VM:

multipass delete --purge tutorial-vm

Uninstall Multipass: Linux | macOS | Windows. On Linux:

snap remove multipass

Next steps

Learn more

Chat with us

Read our Code of conduct and:

File an issue

Make your mark

autopilot-log-collector's People

Contributors

blackboxsw avatar dpb1 avatar fginther avatar

Stargazers

 avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

autopilot-log-collector's Issues

Consider merging with juju-crashdump

There seems to be a fair amount of overlap with the juju-crashdump plugin. Could we merge these projects to reduce duplication of effort?

For reference, that project was originally part of the juju-plugins repo and was started there several months ago, and announced on the Juju mailing list in November, but it was recently split into its own project to better manage packaging, updates, and issue tracking.

log collector broken on recent versions of juju

12:44:29 + cat /tmp/jenkins/workspace/landscape-system-tests/build-area/artifacts/test-output.subunit
12:44:29 + subunit2junitxml --no-passthrough -o /tmp/jenkins/workspace/landscape-system-tests/build-area/artifacts/test-output.xml
12:44:30 + . /tmp/jenkins/workspace/landscape-system-tests/build-area/artifacts/run-info
12:44:30 ++ SERVER_PPA=trunk
12:44:30 + export LANDSCAPE_API_KEY LANDSCAPE_API_SECRET LANDSCAPE_API_URI LANDSCAPE_API_SSL_CA_FILE
12:44:30 + get_cloud_deploy_activity
12:44:30 ++ sort -n
12:44:30 ++ jq -r '.[].id'
12:44:30 ++ python /tmp/jenkins/workspace/landscape-system-tests/build-area/api-client/landscape_api/base.py --json get-activities --query type:DeployOpenstackRegionRequest
12:44:30 ++ tail -n 1
12:44:30 python: can't open file '/tmp/jenkins/workspace/landscape-system-tests/build-area/api-client/landscape_api/base.py': [Errno 2] No such file or directory
12:44:30 + parent_id=
12:44:30 + '[' 2 -ne 0 -o -z '' ']'
12:44:30 + echo 'Failed to fetch the cloud deployment activity'
12:44:30 Failed to fetch the cloud deployment activity
12:44:30 + return 1
12:44:30 + '[' 2 -ne 0 ']'
12:44:30 + extra_logs=
12:44:30 + '[' -f system-tests-console.log ']'
12:44:30 + extra_logs=system-tests-console.log
12:44:30 + '[' -f deploy-openstack-region-activity.json ']'
12:44:30 + collect_logs system-tests-console.log
12:44:30 + local extra_log_files=system-tests-console.log
12:44:30 + echo 'Collecting logs, might fail if LDS failed to bootstrap or deploy'
12:44:30 Collecting logs, might fail if LDS failed to bootstrap or deploy
12:44:30 + juju ssh landscape-server/0 sudo 'HOME=/tmp JUJU_DATA=/var/lib/landscape/juju-homes/`sudo ls -rt /var/lib/landscape/juju-homes/ | tail -1` JUJU_MODEL=controller sudo -u landscape -E juju status --format=json'
12:44:30 + grep -vE '^sudo:'
12:44:30 Connection to 10.96.37.101 closed.
12:44:30 + grep -q 'error: no current controller' /tmp/jenkins/workspace/landscape-system-tests/build-area/artifacts/juju-status.json
12:44:30 + cp /tmp/jenkins/workspace/landscape-system-tests/build-area/artifacts/juju-status.json .
12:44:30 + extra_log_files='system-tests-console.log juju-status.json'
12:44:30 + cat /tmp/jenkins/workspace/landscape-system-tests/build-area/artifacts/juju-status.json
12:44:30 + json2yaml
12:44:30 + python -c 'import json,yaml; import sys; print yaml.safe_dump(json.loads(sys.stdin.read()))'
12:44:31 Traceback (most recent call last):
12:44:31   File "<string>", line 1, in <module>
12:44:31   File "/usr/lib/python2.7/json/__init__.py", line 339, in loads
12:44:31     return _default_decoder.decode(s)
12:44:31   File "/usr/lib/python2.7/json/decoder.py", line 364, in decode
12:44:31     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
12:44:31   File "/usr/lib/python2.7/json/decoder.py", line 382, in raw_decode
12:44:31     raise ValueError("No JSON object could be decoded")
12:44:31 ValueError: No JSON object could be decoded
12:44:31 ++ cat /tmp/jenkins/workspace/landscape-system-tests/build-area/artifacts/juju-status.json
12:44:31 ++ get_pending_nodes
12:44:31 ++ python -c 'import json,sys; status = json.loads(sys.stdin.read()); print " ".join([status["machines"][machine]["dns-name"] for machine in status["machines"].keys() if status["machines"][machine]["agent-state"] != "started"])'
12:44:31 Traceback (most recent call last):
12:44:31   File "<string>", line 1, in <module>
12:44:31   File "/usr/lib/python2.7/json/__init__.py", line 339, in loads
12:44:31     return _default_decoder.decode(s)
12:44:31   File "/usr/lib/python2.7/json/decoder.py", line 364, in decode
12:44:31     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
12:44:31   File "/usr/lib/python2.7/json/decoder.py", line 382, in raw_decode
12:44:31     raise ValueError("No JSON object could be decoded")
12:44:31 ValueError: No JSON object could be decoded
12:44:31 + pending_nodes=
12:44:31 + '[' -n '' ']'
12:44:31 + git clone https://github.com/juju/autopilot-log-collector.git autopilot-log-collector
12:44:31 Cloning into 'autopilot-log-collector'...
12:44:31 + autopilot-log-collector/collect-logs /tmp/jenkins/workspace/landscape-system-tests/build-area/artifacts/all-logs.tar.gz system-tests-console.log juju-status.json
12:44:32 Traceback (most recent call last):
12:44:32   File "autopilot-log-collector/collect-logs", line 649, in <module>
12:44:32     main(tarfile, args.extrafiles, juju, args.inner_model, args.inner)
12:44:32   File "autopilot-log-collector/collect-logs", line 622, in main
12:44:32     collect_logs(juju)
12:44:32   File "autopilot-log-collector/collect-logs", line 368, in collect_logs
12:44:32     units = get_units(juju)
12:44:32   File "autopilot-log-collector/collect-logs", line 264, in get_units
12:44:32     juju_units.append(JujuUnit(name, unit["public-address"]))
12:44:32 KeyError: 'public-address'
12:44:32 + tar xzf /tmp/jenkins/workspace/landscape-system-tests/build-area/artifacts/all-logs.tar.gz landscape-server-0/var/log/landscape-server/job-handler.log
12:44:32 tar (child): /tmp/jenkins/workspace/landscape-system-tests/build-area/artifacts/all-logs.tar.gz: Cannot open: No such file or directory
12:44:32 tar (child): Error is not recoverable: exiting now
12:44:32 tar: Child returned status 2
12:44:32 tar: Error is not recoverable: exiting now
12:44:32 + mv landscape-server-0/var/log/landscape-server/job-handler.log /tmp/jenkins/workspace/landscape-system-tests/build-area/artifacts
12:44:32 mv: cannot stat 'landscape-server-0/var/log/landscape-server/job-handler.log': No such file or directory
12:44:32 + maas_login
12:44:32 + maas login system-tests http://10.96.0.10/MAAS/ Dc6fwNPvWKpWyntrgz:HGW3h8QghBfvWEp4QT:ZT6KaSTKL6QH4GB33HxgNyXBrGhDr6Um
12:44:33 

Need to add collection of juju config data

collect-logs should gather the juju config data for each juju service it logs. This would make our debugging of charm issues easy. Suggest storing the files to a /meta or /juju-config directory. Something like this:

haproxy-0/
postgresql-0/
juju-config/haproxy.config
juju-config/postgresql.config
juju-config/landscape-server.config
juju-config/...
landscape-server-0/
landscape-0-inner-logs/cinder-0
landscape-0-inner-logs/ceph-mon-0
landscape-0-inner-logs/juju-config/cinder.config
landscape-0-inner-logs/juju-config/ceph-mon.config
landscape-0-inner-logs/juju-config/openstack.config
landscape-0-inner-logs/juju-config/...
landscape-0-inner-logs/openstack-0
landscape-0-inner-logs/...
...

Additional files are dropped if original tar file can't be created

collect-logs allows additional files to specified that will be added to the generated tar file. However, if that file isn't created (because no juju model could be found) the additional files aren't added either and the result is no log file is created.

The desired outcome is to create a log file as requested with just the files specified on the command line.

This is what the failure looks like:

autopilot-log-collector/collect-logs /tmp/jenkins/workspace/landscape-system-tests/build-area/artifacts/all-logs.tar.gz system-tests-console.log juju-status.json
ERROR, no units found. Make sure the right juju environmentis set.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    πŸ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. πŸ“ŠπŸ“ˆπŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❀️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.