We're using journald-cloudwatch-logs, which is a Go application that sends our servers' Journald logs to Amazon Web Services' CloudWatch Logs service.

Up until very recently – perhaps the last couple of days – I was happily installing journald-cloudwatch-logs via go get as part of a Chef run. However, it's started bombing, and it seems to be failing while installing prerequisites, one of which is go-systemd-sdjournal. I tracked down the failure and was able to reproduce it while installing go-systemd-sdjournal on its own. Here's how the failure looks:

[root@server ~]# go get github.com/coreos/go-systemd/sdjournal
# github.com/coreos/go-systemd/sdjournal
/usr/local/src/go/src/github.com/coreos/go-systemd/sdjournal/read.go:164: multiple-value writer.Write() in single-value context

The system's running Red Hat Enterprise Linux 7.2.

I don't have any knowledge of Go and largely am just a monkey installing packages, so I'm not comfortable debugging this. I thought I'd bring this up here to see if anyone has any idea what might be going wrong, or if anyone else is experiencing the same problem.

I'm happy to provide any further details. Just let me know what you need.

Thanks!

#164 (comment) seems to have highlighted a race between Match/Send/Wait/Next, where Wait returns after seeing the matching entry but Next reports and EOF. Increasing the Wait timeout doesn't seem to help, so there may be a race somewhere.

After #178 and #179, this is the only remaining testsuite failure. It doesn't happen always, but seem very frequent on debian/ubuntu systems (I've seen it often locally and on containerized travis).

It looks like someone refactored the serialization of the unit directives to group them, but then failed to update the tests.

FYI. When I test sdjournal, I got this error:

systemd/sd-journal.h: No such file or directory
compilation terminated

@jonboulle

forgive me if there's an obvious solution, but the journal code looks unusable since one can't initialize the socket connection from outside the library.

Since the unit directives were grouped into sections using a map, when iterating over the map to serialize the data, the ordering is no longer deterministic due to the randomisation of map iteration that was added into golang.

The value of the ExecStart in the following unit file ...

[Unit]
Description=PING

[Service]
ExecStart=/bin/bash -c "while true; do echo \"ping\"; sleep 1; done"

...is being parsed /bin/bash -c "while true; do echo \"ping. I expect it to be /bin/bash -c "while true; do echo \"ping\"; sleep 1; done".

When attempting to compile an application that depends on go-systemd, I get error messages from pkg-config failing to find lsystemd. I've produced this on images derived from golang:1.5 and ubuntu:14.04 with dockerfiles along these lines:

FROM golang:1.5
RUN apt-get update
RUN apt-get -y install build-essentials pkg-config libsystemd-journal-dev
# compile go application here

I've gotten a range of different errors by playing with the combination of the many transitional libsystemd packages that I install:

# pkg-config --cflags libsystemd
Package libsystemd was not found in the pkg-config search path.
Perhaps you should add the directory containing `libsystemd.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libsystemd' found
pkg-config: exit status 1

# pkg-config --cflags libsystemd-journal
Package libsystemd-journal was not found in the pkg-config search path.
Perhaps you should add the directory containing `libsystemd-journal.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libsystemd-journal' found
pkg-config: exit status 1

# github.com/coreos-inc/quay-builder
/usr/local/go/pkg/tool/linux_amd64/link: running gcc failed: exit status 1
/usr/bin/ld: cannot find -lsystemd
collect2: error: ld returned 1 exit status

There should be a isSystemd helper function to help services determine if they are running under systemd. This will aid services in support more systemd features such as sdnotify, without breaking ignoring errors when not running on systemd systems.

~/.../gopath/src/github.com/coreos/go-systemd/dbus$ go test
# github.com/coreos/go-systemd/dbus
./dbus.go:67: undefined: dbus.BusObject
./dbus.go:71: undefined: dbus.BusObject
FAIL    github.com/coreos/go-systemd/dbus [build failed]

Not sure when it broke, but it broke.

Disclaimer: I'm not normally a Go developer, so some of my terms may not be quite right.

I'm trying to build a Docker plugin that use the go-plugins-helpers from Docker and I wanted the binary to run in Docker alpine, but alpine doesn't have the linked dynamic binaries or whatever, so I was reading up that I need to set CGO_ENABLED=0 when building. I've tried a few different build commands, including:

CGO_ENABLED=0 go build -a -installsuffix cgo -ldflags '-s' -o bin/linux/amd64/local-persist -v
CGO_ENABLED=0 go build -a -tags netgo -a -o bin/linux/amd64/local-persist -v

(Tried that last one with and without CGO_ENABLED flag. No error, but still won't run in alpine.)

I keep getting this error:

../github.com/docker/go-plugins-helpers/sdk/unix_listener.go:12:2: no buildable Go source files in /go/src/github.com/coreos/go-systemd/util
make: *** [binary-linux-amd64] Error 1

Steps to reproduce yourself:

git clone https://github.com/CWSpear/local-persist
cd local-persist
docker run -it -v `pwd`:/go/src/app  golang:onbuild bash
# now you'll be inside the docker container, next run:
export CGO_ENABLED=0
go get -v -d
go build [whatever flags] .

See docker/go-plugins-helpers#50

Repro steps

1. Create a new instance of go-systemd's dbus.Conn
2. Call GetUnitInfo with a dbus path containing unsupported characters

Result:
The process will hang

Expected:
Return an error telling me I've asked to do the impossible

Here's a script to illustrate the problem:

package main

import (
        systemdDbus "github.com/coreos/go-systemd/dbus"
        "github.com/guelfey/go.dbus"
        "os"
)

func main() {
        systemd := systemdDbus.New()
        path := dbus.ObjectPath(os.Args[1])
        info, _ := systemd.GetUnitInfo(path)
        println(info["ActiveState"].Value().(string))
}

Build this and run something like

./test ./test /org/freedesktop/systemd1/unit/ping.service

vs.

./test /org/freedesktop/systemd1/unit/ping_2eservice

This describes valid dbus paths: http://dbus.freedesktop.org/doc/dbus-specification.html#message-protocol-marshaling-object-path

This should illustrate what encoding systemd expects: http://cgit.freedesktop.org/systemd/systemd/tree/src/shared/util.c#n1360

Currently the util module is not covered by any tests. We should start adding at least some basic ones, which would help avoiding breaking things (e.g. #192) during refactoring.

Hello, I'm trying to connect to systemd --user, using dbus.NewUserConnection()

dbus.service and dbus.socket are both running

by default DBUS_SESSION_ADDRESS is set to
unix:abstract=/tmp/dbus-RdwKN0BXaL,guid=6b95481a972dfb9cbd5654b557024934

when trying to connect it shows this error
Process org.freedesktop.systemd1 exited with status 1

I tried changing it to the socket in the dbus.socket
unix:path=/run/user/10561/dbus/user_bus_socket

but then it fails with
Activation of org.freedesktop.systemd1 timed out

looking at the journald logs I see
dbus-daemon[910]: Failed to activate service 'org.freedesktop.systemd1': timed out

any ideas what might be wrong?

Thanks,
Kiril

Here in JournalReader.Read(), the returned length can be larger than what's actually been read when the buffer provided is not big enough.
Which will cause a panic in https://golang.org/src/bytes/buffer.go#L177. (go 1.6.2 linux/amd64)

I found this when doing kubectl logs on some pods.

Here is the panic trace from the rkt api service:

Jun 11 01:05:48 yifan-test-rkt-minion-group-fc7i rkt[1212]: panic: runtime error: slice bounds out of range [recovered]
Jun 11 01:05:48 yifan-test-rkt-minion-group-fc7i rkt[1212]:         panic: runtime error: slice bounds out of range
Jun 11 01:05:48 yifan-test-rkt-minion-group-fc7i rkt[1212]: goroutine 6580 [running]:
Jun 11 01:05:48 yifan-test-rkt-minion-group-fc7i rkt[1212]: panic(0xdbd0c0, 0xc82000c030)
Jun 11 01:05:48 yifan-test-rkt-minion-group-fc7i rkt[1212]:         /usr/lib/go-1.6/src/runtime/panic.go:481 +0x3e6
Jun 11 01:05:48 yifan-test-rkt-minion-group-fc7i rkt[1212]: io/ioutil.readAll.func1(0xc8201b9950)
Jun 11 01:05:48 yifan-test-rkt-minion-group-fc7i rkt[1212]:         /usr/lib/go-1.6/src/io/ioutil/ioutil.go:30 +0x11e
Jun 11 01:05:48 yifan-test-rkt-minion-group-fc7i rkt[1212]: panic(0xdbd0c0, 0xc82000c030)
Jun 11 01:05:48 yifan-test-rkt-minion-group-fc7i rkt[1212]:         /usr/lib/go-1.6/src/runtime/panic.go:443 +0x4e9
Jun 11 01:05:48 yifan-test-rkt-minion-group-fc7i rkt[1212]: bytes.(*Buffer).ReadFrom(0xc8201b98a8, 0x7f9d80c0b780, 0xc82012c428, 0x0, 0x0, 0x0)
Jun 11 01:05:48 yifan-test-rkt-minion-group-fc7i rkt[1212]:         /usr/lib/go-1.6/src/bytes/buffer.go:177 +0x322
Jun 11 01:05:48 yifan-test-rkt-minion-group-fc7i rkt[1212]: io/ioutil.readAll(0x7f9d80c0b780, 0xc82012c428, 0x200, 0x0, 0x0, 0x0, 0x0, 0x0)
Jun 11 01:05:48 yifan-test-rkt-minion-group-fc7i rkt[1212]:         /usr/lib/go-1.6/src/io/ioutil/ioutil.go:33 +0x156
Jun 11 01:05:48 yifan-test-rkt-minion-group-fc7i rkt[1212]: io/ioutil.ReadAll(0x7f9d80c0b780, 0xc82012c428, 0x0, 0x0, 0x0, 0x0, 0x0)
Jun 11 01:05:48 yifan-test-rkt-minion-group-fc7i rkt[1212]:         /usr/lib/go-1.6/src/io/ioutil/ioutil.go:42 +0x51
Jun 11 01:05:48 yifan-test-rkt-minion-group-fc7i rkt[1212]: main.(*v1AlphaAPIServer).GetLogs(0xc82012c418, 0xc8200485c0, 0x7f9d80c0b730, 0xc8203de760, 0x0, 0x0)
Jun 11 01:05:48 yifan-test-rkt-minion-group-fc7i rkt[1212]:         /opt/build-rkt/gopath/src/github.com/coreos/rkt/rkt/api_service.go:767 +0x734
Jun 11 01:05:48 yifan-test-rkt-minion-group-fc7i rkt[1212]: github.com/coreos/rkt/api/v1alpha._PublicAPI_GetLogs_Handler(0xe69940, 0xc82012c418, 0x7f9d80c0b698, 0xc82013c960, 0x0, 0x0)
Jun 11 01:05:48 yifan-test-rkt-minion-group-fc7i rkt[1212]:         /opt/build-rkt/gopath/src/github.com/coreos/rkt/api/v1alpha/api.pb.go:1088 +0x175
Jun 11 01:05:48 yifan-test-rkt-minion-group-fc7i rkt[1212]: google.golang.org/grpc.(*Server).processStreamingRPC(0xc8202d9740, 0x7f9d80bf6158, 0xc82019abd0, 0xc820372540, 0xc820312040, 0x1580720, 0xc820199230, 0x0, 0x0)
Jun 11 01:05:48 yifan-test-rkt-minion-group-fc7i rkt[1212]:         /opt/build-rkt/gopath/src/github.com/coreos/rkt/Godeps/_workspace/src/google.golang.org/grpc/server.go:421 +0x2dc
Jun 11 01:05:48 yifan-test-rkt-minion-group-fc7i rkt[1212]: google.golang.org/grpc.(*Server).handleStream(0xc8202d9740, 0x7f9d80bf6158, 0xc82019abd0, 0xc820372540, 0xc820199230)
Jun 11 01:05:48 yifan-test-rkt-minion-group-fc7i rkt[1212]:         /opt/build-rkt/gopath/src/github.com/coreos/rkt/Godeps/_workspace/src/google.golang.org/grpc/server.go:493 +0x114e
Jun 11 01:05:48 yifan-test-rkt-minion-group-fc7i rkt[1212]: google.golang.org/grpc.(*Server).Serve.func2.1.1(0xc8202d9740, 0x7f9d80bf6158, 0xc82019abd0, 0xc820372540, 0xc820199230, 0xc8202d2420)
Jun 11 01:05:48 yifan-test-rkt-minion-group-fc7i rkt[1212]:         /opt/build-rkt/gopath/src/github.com/coreos/rkt/Godeps/_workspace/src/google.golang.org/grpc/server.go:278 +0x49
Jun 11 01:05:48 yifan-test-rkt-minion-group-fc7i rkt[1212]: created by google.golang.org/grpc.(*Server).Serve.func2.1
Jun 11 01:05:48 yifan-test-rkt-minion-group-fc7i rkt[1212]:         /opt/build-rkt/gopath/src/github.com/coreos/rkt/Godeps/_workspace/src/google.golang.org/grpc/server.go:280 +0x5ac
Jun 11 01:05:48 yifan-test-rkt-minion-group-fc7i systemd[1]: rkt-api-service.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Jun 11 01:05:48 yifan-test-rkt-minion-group-fc7i systemd[1]: rkt-api-service.service: Unit entered failed state.
Jun 11 01:05:48 yifan-test-rkt-minion-group-fc7i systemd[1]: rkt-api-service.service: Failed with result 'exit-code'.

cc @jonboulle @iaguis @alban @euank

It will be better if provide API for read journal like python
http://www.freedesktop.org/software/systemd/python-systemd/journal.html#accessing-the-journal

Perhaps I should have clued into this more when c224373 went by but we need the ability to control jobs without waiting for them on all job control methods. This is required for coreos-cloudinit which should never call the blocking API since that can result in deadlocks.

What Happens:

Calling the ListUnitsByNames function on *dbus.Conn results in an unexpected error: Unknown method 'ListUnitsByNames' or interface 'org.freedesktop.systemd1.Manager'

What I expected to happen:

A slice of UnitStatus and no error, or a more descriptive error message.

How to reproduce:

Sample Code:

package main
import (
	"fmt"
	"github.com/coreos/go-systemd/dbus"
)

func main() {
	conn, err := dbus.New()
	if err != nil {
		fmt.Println("error getting connection: ", err)
	}
	res, err := conn.ListUnitsByNames([]string{"acpid.service"})
	if err != nil {
		fmt.Println("error with ListUnitsByNames: ", err)
		return
	}
	fmt.Printf("got %d results\n", len(res))
}

Test system:

uname -a; lsb_release -a
Linux rebecca-vboxdev 4.4.0-62-generic #83-Ubuntu SMP Wed Jan 18 14:10:15 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 16.04.2 LTS
Release:	16.04
Codename:	xenial

core@coreos-test ~ $ journalctl
Error was encountered while opening journal files: Invalid argument

core@coreos-test ~ $ cat /etc/os-release
NAME=CoreOS
ID=coreos
VERSION=723.3.0
VERSION_ID=723.3.0
BUILD_ID=
PRETTY_NAME="CoreOS 723.3.0"
ANSI_COLOR="1;32"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://github.com/coreos/bugs/issues"

see https://travis-ci.org/coreos/mayday/builds/57730602

dbus.Conn.startJob() adds the job to the joblisteners map:
c.jobListener.jobs[path] = ch
But while jobComplete reads this it doesn't actually remove it from the map.

Hi everyone. I was wondering if a tag could be made for a release of go-systemd? I've been working on updating the Debian packaging for go-systemd and would like to package a release instead of merely a snapshot.

I was going to go out on a limb and say that since there haven't been many commits recently maybe a release tag is in order but there have been a few fixes and merges since I last updated my local copy. Obviously it's up to you guys.

Thanks,

Tim.

This should be an extension of the existing GetUnitProperties(unit string) interface. Probably a prototype like GetUnitProperty(unit string, propertyName string) systemd.Property

https://github.com/coreos/go-systemd/blob/master/dbus/methods.go#L164

We originally discovered in fleet [0] that systemd silently truncates lines at 2000 characters. This has been confirmed upstream [1]. Unfortunately, I believe we need to add this check to our unit file parser so we can give a reasonable experience to users rather than making them chase down the same rabbit hole that I did.

[0] coreos/fleet#992
[1] https://bugs.freedesktop.org/show_bug.cgi?id=85308

Hi,

Currently github.com/coreos/pkg/capnslog/journald_formatter imports github.com/coreos/go-systemd/journal, but github.com/coreos/go-systemd/journal and github.com/coreos/go-systemd/sdjournal/journal import github.com/coreos/pkg/dlopen. This is a pain for distro developers: we want to package both github.com/coreos/go-systemd and github.com/coreos/pkg but cycles in the dependency graph are a pain. In this case it looks as if .../pkg/dlopen could move to ../go-systemd but I've no idea if that would really be appropriate.

Splitting things out into separate repos would also always be a fix for this.

Cheers,
mwh

My usual first step of looking at a go repo is to go install ./... from the repo root, to make sure everything compiles. But you can't do that with go-systemd, because you get this:

# github.com/coreos/go-systemd/examples/activation
examples/activation/listen.go:25: fixListenPid redeclared in this block
    previous declaration at examples/activation/activation.go:25
examples/activation/listen.go:34: main redeclared in this block
    previous declaration at examples/activation/activation.go:34
examples/activation/udpconn.go:26: fixListenPid redeclared in this block
    previous declaration at examples/activation/listen.go:25
examples/activation/udpconn.go:35: main redeclared in this block
    previous declaration at examples/activation/listen.go:34

Maybe it would be prudent to move each example into their own directory?

The methods are documented here:

• https://www.freedesktop.org/software/systemd/man/sd_notify.html
• https://www.freedesktop.org/software/systemd/man/sd_listen_fds.html

FYI I'm currently working on implementing these methods in the notify_fds branch of my fork, and I have a basic POC working (but without proper tests at the moment). I plan to contribute back my work under the CoreOS contribution guidelines. The use case is that we (Uber) have services where a single process needs to listen on multiple well-known ports, and we'd like to implement graceful restarts of these services.

sd_journal_wait appears from the man page to take a (relative) timeout value but the code passes in an absolute wall-clock time, which will presumably be massively larger than what we meant

In sdjournal.Wait():
to = uint64(time.Now().Add(timeout).Unix() / 1000)
should be
to = uint64(timeout / 1000)

It certainly works better with this change at my end.

Hi go-systemd team!

Many thanks for all the work you do!

I'm packaging go-systemd for Ubuntu (as part of the juju project).
We do license checks for all the code we package as part of our packaging pipeline.
It seems to me that one file in your repo violates requirements of the Apache-2.0 license.
Could you please help me to understand if it's true or not and (if so) fix the issue in your repo?

File in question: https://github.com/coreos/go-systemd/blob/master/daemon/sdnotify.go

File states that it is 'forked from Docker project' which is correct.
Original file from Docker project can be found here:
https://github.com/docker/docker/blob/master/pkg/systemd/sd_notify.go
In terms of Apache-2.0 license forked version of the file is called Derivative Work.

Docker project is licensed under Apache-2.0 license (as well as go-systemd project).
This license allows you to redistribute Derivative Works only if you obey the following rules:

(a) You must give any other recipients of the Work or Derivative Works a copy of this License;

While both projects are license under Apache-2.0 license, text is a bit different between the licenses.
Go-systemd license: https://github.com/coreos/go-systemd/blob/master/LICENSE
Docker license: https://github.com/docker/docker/blob/master/LICENSE
You may see that at least formatting of paragraph 4 is different. It means that you need to include Docker's license into your project (maybe put it inside daemon/ folder?)
and put a reference to it from sdnotify.go file.

(b) You must cause any modified files to carry prominent notices stating that You changed the files;

While 'forked from Docker project' looks okay in the sense of this requirement I'd recommend you to specify which exact file you've modified (link to Docker's github repo may work).

(c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices <...>

Docker's sd_notify.go file doesn't contain explicit copyright notices but they are available in Docker's LICENSE file:
https://github.com/docker/docker/blob/master/LICENSE#L179
You may find there: 'Copyright 2013-2015 Docker, Inc.'

This copyright information needs to be included into daemon/sdnotify.go.
You may add your own copyright notice (for your changes) but you shouldn't remove Docker's copyright.

(d) If the Work includes a "NOTICE" text file <...> then any Derivative Works <...> must include a readable copy of the attribution notices contained within such NOTICE file <...>

Docker project has such NOTICE file: https://github.com/docker/docker/blob/master/NOTICE
This file contains some valuable information and I'd recomment you to put it into daemon/ folder as well.
It may be linked from daemon/sd_notify.go as well as the LICENSE file.

Thanks A LOT for helping!
Oleg

Unfortunately the ListUnits method does not fill my needs. An enabled (but inactive) unit may not show up under some conditions. I've run into this consistently.

The implementation should be virtually identical to ListUnits. ListUnitFiles is a dbus method on the manager object (per the docs).

ParseProperties(props []Property, unitType string) ([]_unit.UnitOption, error)
ParseOptions(opts []_unit.UnitOption) ([]Property, string, error)

From what I can tell, all the unit options [1] map into dbus properties [2](though the reverse is not true, of course). I've done something like this for a limited sub-set, but it would be helpful for a number of reasons if this functionality existed upstream (here).

[1] http://www.freedesktop.org/software/systemd/man/systemd.directives.html
[2] http://www.freedesktop.org/wiki/Software/systemd/dbus/

go-systemd currently relies on an existing dbus system bus in order to invoke systemd methods. However, this may be not strictly needed as systemd itself exposes a UNIX socket for one-to-one dbus calling at /var/run/systemd/private.

It may be possible to implement a fallback mechanism, in order to retry connecting to the p2p endpoint if the system bus doesn't exist. However this needs a bit more investigation to determine if this works in a transparent way, and the same set of methods/properties are available.

Looks like an upstream dependency changed:

$ go get github.com/coreos/go-systemd/dbus

github.com/coreos/go-systemd/dbus

src/github.com/coreos/go-systemd/dbus/dbus.go:186: cannot use conn.Object("org.freedesktop.systemd1", dbus.ObjectPath("/org/freedesktop/systemd1")) (type dbus.BusObject) as type *dbus.Object in return argument: need type assertion

Since #110 go-systemd now uses Cgo, and depends on both libsystemd-journal (at build and runtime) and pkg-config (at build). These new dependencies are only used by the code for reading from the systemd journal.

This is moderately inconvenient, as previously go-systemd only used native Go packages to communicate with the journal or systemd.

Ideally native Go could be used, but maybe hiding the journal-reading feature behind a build flag hides the pain.

CC @philips @jonboulle

Hi there,

I was taking a looking at go-systemd's machined functionality and I was wondering how come there's only RegisterMachine()?

If adding additional functionality in there is something desired, I'd be happy to submit a pull request adding features such as GetMachine, GetImage, ListMachines, TerminateMachine, etc. And if it'd be preferred that I add each one in a separate Pull Request, I can do that as well.

Thanks!

I'm filing this because I want SkyDNS on windows, but I won't resent a "wont fix" close. Obviously there is no hope of systemd on windows....

C:\Users\rober_000\Documents>go build github.com/skynetservices/skydns
github.com/coreos/go-systemd/activation
C:\Projects\Go\src\github.com\coreos\go-systemd\activation\files.go:47: cannot use fd (type int) as type syscall.Handle in argument to syscall.CloseOnExec

Sometimes it is helpful to look at an enabled unit file. I'd rather not go look up the file in /etc/systemd, etc., which is kind of the point of using the Go bindings in the first place. :) At the commandline I can just run "systemctl cat ...". Having an equivalent method on dbus.Conn would really helpful. I expect it would dovetail onto work for issue #77.

As a temporary workaround I am reading the dbus properties I need (after hunting down the mapping from the conf directives) and manually converting them into unit.UnitOption values. From there I can use them or call unit.Serialize to get the unit file. However that's a lot of work, full of risk of getting something wrong (or out of date), for something that I would expect is not an out-of-the-ordinary need.

Hi!

I have a problem with

I'm using a slightly modified version of (r *JournalReader) Follow() returning a channel of *JournalEntrys but the waiting logic is preserved so I'm writing here.

I noticed that after some time the overall number of active goroutines grows slowly and differs from one host to another:

goroutine profile: total 29
17 @ 0x42ef3a 0x42f02e 0x406038 0x405dfd 0x4fa712 0x45fc51
#	0x4fa711	github.com/mheese/journalbeat/journal.Follow.func2.1+0x91	/github.com/mheese/journalbeat/journal/follow.go:86

Here is an example of a single goroutine dump:

goroutine 180549 [chan send, 16014 minutes]:
github.com/mheese/journalbeat/journal.Follow.func2.1(0xc4215d8230, 0xc4215d81c0, 0xc420177920)
	/github.com/mheese/journalbeat/journal/follow.go:86 +0x92
created by github.com/mheese/journalbeat/journal.Follow.func2
	/github.com/mheese/journalbeat/journal/follow.go:89 +0x29e

Here is a rough repro: https://play.golang.org/p/PNGLjD_S5m
When using a buffered channel of size 1 you are likely to get one event already sent to channel buffer, and one event pending to be sent to the channel before you get a pollDone signal. This causes the aforementioned issue with orphaned goroutines being stuck.

A couple of questions in this regard:

1. Is there any rationale behind handling a blocking "wait for new events to arrive" operation through spawning a goroutine with consequent select?
2. If there is any then why use buffered channels? And why use default: events <- ...? Both of them being reasons for a leak.

Hello there,

I am trying to improve the performance of dbus methods and I realized that the linking operation is quite expensive. This operation becomes a bottleneck whenever you need to link-start many systemd units. In fact, the response time increases over the time whenever you want to link multiple units.
As a result of my investigation, I found that the force (force=true) property of the dbus function doesn't bring any benefit (when using the latest version of dbus). It is true that the force property set to true could bring some consistency in case of race condition. However, I think it should be configured to be false to offer a better performance.

func (c *Conn) LinkUnitFiles(files []string, runtime bool, force bool) ([]LinkUnitFileChange, error) {
    result := make([][]interface{}, 0)
    err := c.sysobj.Call("org.freedesktop.systemd1.Manager.LinkUnitFiles", 0, files, runtime, force).Store(&result)
    if err != nil {
        return nil, err
    }

What is it your impression ? The improvements in terms of performance are in the order of seconds and could change the complexity of this operation.

Why did you decide to define force=true as default ?

Thanks.

Right now, sdjournal only lets you access fields you already know to expect. The C library has sd_journal_enumerate_data for accessing all fields of a single message, and sd_journal_enumerate_fields for listing all fields that all messages in the journal contain.

https://www.freedesktop.org/software/systemd/man/sd_journal_get_data.html
https://www.freedesktop.org/software/systemd/man/sd_journal_enumerate_fields.html

I think it will be better if this function returns 0 in the case that $NOTIFY_SOCKET is not set as stated in sd_notify() documentation.

I am working on enabling systemd-cgroup support in runc by adding a flag and seeing this error.
Here is the branch
https://github.com/opencontainers/runc/compare/master...mrunalp:systemd_cgroups?expand=1

[root@dhcp-16-129 testroot]# runc --systemd-cgroup start 1234
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x68839f]

goroutine 1 [running]:
panic(0x7a5460, 0xc820012130)
        /usr/lib/golang/src/runtime/panic.go:464 +0x3e6
github.com/coreos/go-systemd/dbus.(*Conn).startJob(0x0, 0x0, 0x8a26e0, 0x33, 0xc820136100, 0x4, 0x4, 0x0, 0x0, 0x0)
        /root/gosrc/src/github.com/opencontainers/runc/Godeps/_workspace/src/github.com/coreos/go-systemd/dbus/methods.go:47 +0x32f
github.com/coreos/go-systemd/dbus.(*Conn).StartTransientUnit(0x0, 0xc820122210, 0xf, 0x81f300, 0x7, 0xc820148000, 0x7, 0x8, 0x0, 0xc820122245, ...)
        /root/gosrc/src/github.com/opencontainers/runc/Godeps/_workspace/src/github.com/coreos/go-systemd/dbus/methods.go:137 +0x2d5
github.com/opencontainers/runc/libcontainer/cgroups/systemd.(*Manager).Apply(0xc82000fac0, 0x27a3, 0x0, 0x0)
        /root/gosrc/src/github.com/opencontainers/runc/Godeps/_workspace/src/github.com/opencontainers/runc/libcontainer/cgroups/systemd/apply_systemd.go:229 +0x1086
github.com/opencontainers/runc/libcontainer.(*initProcess).start(0xc820132000, 0x0, 0x0)
        /root/gosrc/src/github.com/opencontainers/runc/Godeps/_workspace/src/github.com/opencontainers/runc/libcontainer/process_linux.go:247 +0x41e
github.com/opencontainers/runc/libcontainer.(*linuxContainer).Start(0xc8200ce180, 0xc8200e6d80, 0x0, 0x0)
        /root/gosrc/src/github.com/opencontainers/runc/Godeps/_workspace/src/github.com/opencontainers/runc/libcontainer/container_linux.go:189 +0x1d2
main.(*runner).run(0xc8200d6f90, 0xc820090930, 0x8, 0x0, 0x0)
        /root/gosrc/src/github.com/opencontainers/runc/utils.go:342 +0x630
main.startContainer(0xc82008ec60, 0xc820090900, 0x0, 0x0, 0x0)
        /root/gosrc/src/github.com/opencontainers/runc/start.go:122 +0x406
main.glob.func11(0xc82008ec60)
        /root/gosrc/src/github.com/opencontainers/runc/start.go:72 +0x17b
github.com/codegangsta/cli.Command.Run(0x81f838, 0x5, 0x0, 0x0, 0x0, 0x0, 0x0, 0x8703a0, 0x1a, 0x8d16a0, ...)
        /root/gosrc/src/github.com/opencontainers/runc/Godeps/_workspace/src/github.com/codegangsta/cli/command.go:137 +0x1081
github.com/codegangsta/cli.(*App).Run(0xc82008ea20, 0xc82000a200, 0x4, 0x4, 0x0, 0x0)
        /root/gosrc/src/github.com/opencontainers/runc/Godeps/_workspace/src/github.com/codegangsta/cli/app.go:176 +0xffa
main.main()
        /root/gosrc/src/github.com/opencontainers/runc/main.go:123 +0xc8e

Basically this TODO needs to be implemented:

func (s *SubscriptionSet) Subscribe() (<-chan map[string]*UnitStatus, <-chan error) {
	// TODO: Make fully evented by using systemd 209 with properties changed values
	return s.conn.SubscribeUnitsCustom(time.Second, 0,
		mismatchUnitStatus,
		func(unit string) bool { return s.filter(unit) },
	)
}

can't accurately monitor units being restarted without calling SubscribeUnitsCustom with a very short delay. which wastes a bunch of cpu.

We encountered this issue today twice on a production system. We haven't yet identified the root cause, however this appeared to be an ancillary issue.

The host was running out of memory, likely from journald. It attempted to compress and rotate logs, but was unable to allocate the memory. Journald died but was restarted. However' we're also using the docker journald log driver along with journald's syslog transport. After the restart, all docker containers failed to continue logging with EPIPE errors to stdout/stderr.

In looking at the journal code, it connects to journald on init, however doesn't have any error handling where it might reconnect if journald was restarted.

The log output we had was:

May 26th 2017, 05:34:03.204    System journal (/var/log/journal/) is 1.7G, max 2.6G, 931.8M free.
May 26th 2017, 05:34:04.220    Failed to initialize XZ encoder: code 5
May 26th 2017, 05:34:04.231    systemd-journald.service: Main process exited, code=dumped, status=6/ABRT
May 26th 2017, 05:34:04.232    Failed to compress (unnamed temporary file): Invalid argument
May 26th 2017, 05:34:04.242    Detected coredump of the journal daemon or PID 1, diverted to /var/lib/systemd/coredump/core.systemd-journal.0.9b634f7d87464833a67ca9124f25ab86.14979.1495802022000000.
May 26th 2017, 05:34:04.242    systemd-journald.service: Unit entered failed state.
May 26th 2017, 05:34:04.245    systemd-journald.service: Service has no hold-off time, scheduling restart.
May 26th 2017, 05:34:04.245    systemd-journald.service: Failed with result 'core-dump'.
May 26th 2017, 05:34:04.259    Stopped Flush Journal to Persistent Storage.
May 26th 2017, 05:34:04.259    Stopped Journal Service.
May 26th 2017, 05:34:04.259    Stopping Flush Journal to Persistent Storage...
May 26th 2017, 05:34:04.268    Starting Journal Service...

After that, we only get huge error spikes to New Relic with EPIPE.

Hi there,

Would there be any opposition to implementing support for importd? importd docs found here.

I'd be happy to implement this and submit a pull request is this is something desired.

Thanks

Instead of using a timer in SubscribeUnits react to PropertiesChanged event:

http://lists.freedesktop.org/archives/systemd-devel/2013-October/013546.html

This is a tracking issue for the upstream Golang bug that keeps the large journal entry stuff from working right now. Large journal entries won't work until this is fixed upstream in Golang:

https://code.google.com/p/go/issues/detail?id=6476

In coreos/go-systemd/activation/listeners.go, line 34 method net.FileListener() is used to create net.Listener from file. For unix domain sockets it returns UnixListener, whose method Close() unlinks it, which leads to deletion of the file in the file system. Here is method Close() from unixsock_posix.go:

// Close stops listening on the Unix address.  Already accepted
// connections are not closed.
func (l *UnixListener) Close() error {
    if l == nil || l.fd == nil {
        return syscall.EINVAL
    }

    // The operating system doesn't clean up
    // the file that announcing created, so
    // we have to clean it up ourselves.
    // There's a race here--we can't know for
    // sure whether someone else has come along
    // and replaced our socket name already--
    // but this sequence (remove then close)
    // is at least compatible with the auto-remove
    // sequence in ListenUnix.  It's only non-Go
    // programs that can mess us up.
    if l.path[0] != '@' {
            syscall.Unlink(l.path)
    }
    return l.fd.Close()
}

I think this behavior conflicts with socket activation, because systemd doesn't anticipate this and doesn't recreate file or replace socket. So, when I kill/reload a socket activated service (for test), systemd restarts it with the same socket. but without correspondent file, which leads to immediate crash of service with "file not found" error. In my opinion, this disable important use case of socket activation.

I tried to use abstract names (starting with '@') and they work perfectly (see 'if' block in the code above). Unfortunately, server, which I use as a proxy, doesn't support them.

I am about to implement somewhat cumbersome solution in my own code, but may be you have a better idea how to deal with this issue (may be in your lib, or in it's client code). My idea is to wrap net.Listener, returned from activation.Listeners() with my implementation of net.Listener interface and replace method Close() so, that for UnixListener it do nothing, but with preceding invocation of activation.Files() I am going to obtain and remember file descriptors to be able to close them before exit. A bit better, I can store file descriptor in the wrapper as well and close it in my Close() method.