ctdk / goiardi Goto Github PK

I've a test setup here running 6 chef-clients against goiardi running on a VPS and goiardi is taking almost the whole memory:

Number of nodes: 6

Options:
'GOIARDI_LOG_EVENTS=1',
'GOIARDI_LOG_EVENT_KEEP=100'
'GOIARDI_LOG_LEVEL=debug'

(stdout shows event purge works)

config file:

index-file = "/var/lib/goiardi/goiardi-index.bin"
data-file = "/var/lib/goiardi/goiardi-data.bin"
freeze-interval = 10
time-slew = "15m"
conf-root = "/etc/goiardi"
use-auth = true
use-ssl = false
https-urls = true
disable-webui = false
local-filestore-dir = "/var/lib/goiardi/lfs

Size of /var/lib/goiardi is 101MB

docker ps:

CONTAINER           CPU %               MEM USAGE / LIMIT    MEM %               NET I/O             BLOCK I/O           PIDS
72b6855d467d        0.00%               999.8MiB / 1000MiB   99.98%              1.34MB / 1.55MB     575MB / 696MB       7

Is this expected behaviour? Do you have numbers/experiences on how much memory goiardi needs?
Thanks!

I'm able to crash the goardi server when delivering data bags to it using :
https://github.com/one-com/running_sushi

I've not been able to find the exact call that crashes it and will continue to look for it.
I've attached the stack trace of the crash.

goardi_stack_overflow.txt

Sorry for the terrible bug-report, I cant share the data I'm using but will try to create a smaller test-case.

Hi,

We have been using goiardi (0.11.9) for a few month now and everything runs smoothly so far. There is just the Postgres Database storage consumption that we are quite worried about (see below). Our Chef server is really small (lfs dir has only 6 MB, cookbooks cleaned up and about 120 nodes), but the search_items tables keeps growing and is currently at 60 GB. Is this a normal behavior or should we be worried about this? If this is normal then there probably should be some documentation about it, because larger instances will then have huge databases.

regards,
Jörg

goiardi=# SELECT
goiardi-#    relname as "Table",
goiardi-#    pg_size_pretty(pg_total_relation_size(relid)) As "Size",
goiardi-#    pg_size_pretty(pg_total_relation_size(relid) - pg_relation_size(relid)) as "External Size"
goiardi-#    FROM pg_catalog.pg_statio_user_tables ORDER BY pg_total_relation_size(relid) DESC;
       Table        |  Size   | External Size
--------------------+---------+---------------
 search_items       | 60 GB   | 59 GB
 log_infos          | 16 GB   | 16 GB
 nodes              | 8512 kB | 8296 kB
 data_bag_items     | 3464 kB | 2736 kB
 cookbook_versions  | 560 kB  | 384 kB
 file_checksums     | 352 kB  | 168 kB
 sandboxes          | 232 kB  | 120 kB
 clients            | 224 kB  | 72 kB
 events             | 128 kB  | 64 kB
 data_bags          | 104 kB  | 72 kB
 node_statuses      | 96 kB   | 80 kB
 changes            | 80 kB   | 64 kB
 environments       | 80 kB   | 64 kB
 search_collections | 64 kB   | 56 kB
 users              | 64 kB   | 56 kB
 shovey_runs        | 56 kB   | 56 kB
 projects           | 48 kB   | 40 kB
 tags               | 48 kB   | 40 kB
 shoveys            | 48 kB   | 48 kB
 cookbooks          | 48 kB   | 40 kB
 organizations      | 48 kB   | 40 kB
 reports            | 40 kB   | 40 kB
 dependencies       | 32 kB   | 24 kB
 releases           | 32 kB   | 24 kB
 shovey_run_streams | 32 kB   | 32 kB
 roles              | 24 kB   | 24 kB
(26 rows)

Specifying a sort order should actually do something.

The README is getting too big and unwieldy.

We are running goiardi in the containers in AWS ECS.
Server works well and no failures during the chef runs. But I am seeing this message in the server logs:

2020/06/03 12:27:50 [INFO] [main] parsing time "2020-06-03 12:27:50 UTC" as "2006-01-02 15:04:05 -0700": cannot parse "UTC" as "-0700"

Should I worry or do anything because of this message?

Pretty low hanging fruit - chef-client makes a post to /reporting during a run, it should be easy to add that in.

People have been asking for this already.

If the client and server agree, it'd be nice to allow them to communicate with gob encoded streams rather than JSON. Obviously this is limited to a golang based client.

Right now, you get all the logging statements. The verbose flag should actually do something and let you adjust the amount of logging dumped out.

Relevant log messages:

2015/01/16 14:23:41 http: panic serving 192.168.33.10:46449: runtime error: invalid memory address or nil pointer dereference
goroutine 326 [running]:
net/http.func·011()
    /usr/local/go/src/net/http/server.go:1130 +0xbb
main.fileStoreHandler(0xa374b8, 0xc20814a820, 0xc2083ea750)
    /Users/jeremy/go/src/github.com/ctdk/goiardi/file_store.go:45 +0x20d
net/http.HandlerFunc.ServeHTTP(0x7777f8, 0xa374b8, 0xc20814a820, 0xc2083ea750)
    /usr/local/go/src/net/http/server.go:1265 +0x41
net/http.(*ServeMux).ServeHTTP(0xc20800a720, 0xa374b8, 0xc20814a820, 0xc2083ea750)
    /usr/local/go/src/net/http/server.go:1541 +0x17d
main.(*interceptHandler).ServeHTTP(0x93b5f0, 0xa374b8, 0xc20814a820, 0xc2083ea750)
    /Users/jeremy/go/src/github.com/ctdk/goiardi/goiardi.go:283 +0xed6
net/http.serverHandler.ServeHTTP(0xc2080545a0, 0xa374b8, 0xc20814a820, 0xc2083ea750)
    /usr/local/go/src/net/http/server.go:1703 +0x19a
net/http.(*conn).serve(0xc20814a780)
    /usr/local/go/src/net/http/server.go:1204 +0xb57
created by net/http.(*Server).Serve
    /usr/local/go/src/net/http/server.go:1751 +0x35e

Hi, when I run knife status --hide-by-mins 3600

The server crashes with:

goiardi[6526]: panic: runtime error: index out of range
goiardi[6526]: goroutine 5487 [running]:
goiardi[6526]: github.com/ctdk/goiardi/indexer.(*IdxDoc).TextSearch(0xc42000fb80, 0x0, 0x0, 0xc422078700, 0x0, 0x0)
goiardi[6526]:         /Users/jeremy.bingham/go/src/github.com/ctdk/goiardi/indexer/file_index.go:474 +0x34c
goiardi[6526]: github.com/ctdk/goiardi/indexer.(*IdxCollection).searchTextCollection.func1(0x0, 0x0, 0xc426dabe60, 0xc426dabec0
goiardi[6526]:         /Users/jeremy.bingham/go/src/github.com/ctdk/goiardi/indexer/file_index.go:330 +0x5a
goiardi[6526]: created by github.com/ctdk/goiardi/indexer.(*IdxCollection).searchTextCollection
goiardi[6526]:         /Users/jeremy.bingham/go/src/github.com/ctdk/goiardi/indexer/file_index.go:344 +0x252
systemd[1]: goiardi.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
systemd[1]: goiardi.service: Unit entered failed state.
systemd[1]: goiardi.service: Failed with result 'exit-code'.

This is happening for quite some time, I did an upgrade today to the latest release hoping this would be resolved. Guess I should have reported it sooner.

I am happy to provide any info required and debug this further.

Describe the bug

when I run chef-client against goiardi it can't report node state back
to server:

INFO: HTTP Request Returned 500 Internal Server Error: Error 1406: Data too long for column 'extended_info' at row 1

It turned out the report is >100k which is too large for MariaDB's 'text' type of 64k

You should modify the schema:

--- /usr/share/goiardi/sql-files/sql-files/goiardi-schema-mysql.sql_orig	2019-02-14 18:44:26.000000000 +0100
+++ /usr/share/goiardi/sql-files/sql-files/goiardi-schema-mysql.sql	2019-12-07 21:52:23.159692624 +0100
@@ -269,7 +269,7 @@
   `action` enum('create','delete','modify') NOT NULL,
   `object_type` varchar(100) NOT NULL,
   `object_name` varchar(255) NOT NULL,
-  `extended_info` text,
+  `extended_info` mediumtext,
   PRIMARY KEY (`id`),
   KEY `actor_id` (`actor_id`),
   KEY `action` (`action`),

We have two cookbooks depending on each other (I know it's pretty bad), and it makes the memory usage explode as soon as a client tries to use these cookbooks. It starts to allocate memory without end until the process is killed. If I break the dependency cycle it works normally.

We are using these same cookbooks with Opscode Chef Server 11, without problems.

Hi,

I'm using goiardi 0.11.5 and knife backup to backup/restore all data from/to git into/out of goiardi.

However when using the latest chefdk (2.0.26) that comes with chef 13 (13.2.20), I cannot download cookbooks anymore:

➜   knife backup export -V        
…
Backing up environments
Backing up cookbooks
Backing up cookbook acme
Downloading acme cookbook version 3.1.0
Failed to download cookbook acme version 3.1.0... Skipping
ERROR: Errno::ENOENT: No such file or directory @ unlink_internal - .chef/chef_server_backup/cookbooks/acme-3.1.0

Core knife functionality is broken, too:

➜  knife cookbook download acme 3.1.0
Downloading acme cookbook version 3.1.0
ERROR: TypeError: no implicit conversion of String into Integer

(acme is just the first cookbook in alphabetic order on my goiardi instance)

UPDATE

Okay, started to debugging with pry:

=> 3.1.0
[5] pry(#<Chef::Knife::CookbookDownload>)>     cookbook = Chef::CookbookVersion.load(@cookbook_name, @version)
TypeError: no implicit conversion of String into Integer
from /tmp/chef/lib/chef/server_api_versions.rb:33:in `[]'

UPDATE 2

Downloading acme cookbook version 3.1.0

From: /tmp/chef/lib/chef/server_api_versions.rb @ line 30 Chef::ServerAPIVersions#min_server_version:

    29: def min_server_version
 => 30:   binding.pry
    31: 
    32:   # If we're working with a pre-api-versioning server, always claim to be zero
    33:   if @versions.nil?
    34:     unversioned? ? 0 : nil
    35:   else
    36:     Integer(@versions["min_version"])
    37:   end
    38: end

[1] pry(#<Chef::ServerAPIVersions>)> @versions
=> 0
[2] pry(#<Chef::ServerAPIVersions>)> Integer(@versions["min_version"])
TypeError: no implicit conversion of String into Integer
from (pry):4:in `[]'

Bingo…

Today while working a change, I noticed that I couldn't get my tests to pass. That included a fresh checkout from the master branch. I reached out on Twitter, and managed to dig in a little bit. I figured I'd just open this so you won't need to look on Twitter to find the details.

I'm running Mac OS X 10.10.5 with the following Go runtime: go version go1.5.1 darwin/amd64. My testing of these issues did include recompiling Go 1.5.1, with no change in behavior.

Here is the Twitter thread for reference:

• https://twitter.com/theckman/status/648969453762252801

The first issue I hit was this test failure here:

• https://gist.github.com/theckman/3444005b43296394508d

Based on the stacktrace and the code it points to, it seems like there is a sync.RWMutex.RLock() function call that doesn't get the lock within 10 minutes:

• goiardi/indexer/file_index.go

  Line 599 in 0eb7984

  i.m.RLock()

I believe that code gets transitively called from here:

• goiardi/indexer/file_index.go

  Line 678 in 0eb7984

  err = enc.Encode(i)

After I hit that issue, I then ran go get -u github.com/ctdk/goiardi and the next test failed like this:

• https://gist.github.com/theckman/14a2f90fb7dbe03dac06#file-test1-out-L104-L105

It looks like none of the mocked information made it to the search index. I'm only opening one issue because these two things seem like they may be related (to the untrained eye).

I've been holding off on this one, but people have asked about it.

Right now when cookbook files are uploaded they're stored locally. This is great and all, but it'd be nice to also allow uploading to a 3rd party storage service like s3.

I am using goiardi 0.8.1 with postgresql. I am able to use goiardi (i.e. cookbook upload and new node bootstrap against it) soon after the installation, but after a restart of the service, the admin as well as node client is failing to authorize. My postgres and goiardi configuration is available here and here
I tried raising the log level to debug, but goiardi is not showing any error. any clue?

The unit tests for https://github.com/marpaia/chef-golang are failing because of a goiardi error:

[marpaia-mbp1] goiardi (master) goiardi -A -V -H localhost -P 8443 -D /tmp/goiardi/data -i /tmp/goiardi/index -F 30 --conf-root /tmp/goiardi
2014/08/09 17:54:40 data_store.go:224: error at dscache
2014/08/09 17:54:40 goiardi.go:58: gob: name not registered for interface: "*client.Client"
[marpaia-mbp1] goiardi (master) git log -1                                                                             5:54PM 08/09/14
commit 1d55046daaa099faa2bc4e007996d058806184a3
Author: ctdk <[email protected]>
Date:   Sat Aug 9 12:47:56 2014 -0700

    Yak shaving release
[marpaia-mbp1] goiardi (master)

Does this look familiar?

Domain Name: goiardi.gl
Registry Domain ID: 30141-GL
Registry WHOIS Server:: whois.nic.gl
Updated Date: 2019-12-04T18:26:25.631Z
Creation Date: 2014-12-04T18:05:04.980Z
Registry Expiry Date: 2019-12-04T18:05:05.24Z
Registrar Registration Expiration Date: 2019-12-04T18:05:05.24Z

looks like the domain expired a couple of days ago but can be recovered.

Had a test run where I had a ~> 1.4.1 dep in a cook, and goiardi said it coudln't find the cook that satisfied that, even tho there was an exact 1.4.1. Chef-zero and chef-server had no probs resolving this dep.

The installation process is relatively straightforward, but an even smoother one might be nice, along with init scripts.

It's meant to be a convenient way to track changes to nodes, clients, users, etc.

Hi,

I found some images on the Docker hub (https://hub.docker.com/r/ctdk/goiardi/tags) but they are quite outdated. Please provide (automated) up-to-date builds of goiardi.

regards,
Jörg

Not sure what changed, I think that knife is expecting some kind of header from the server

$ knife cookbook -V upload commonlib
INFO: Using configuration from /home/mistastn/RubymineProjects/chef/.chef/knife.rb
Uploading commonlib      [0.0.0]
INFO: Validating ruby files
INFO: Validating templates
INFO: Syntax OK
INFO: Saving commonlib
ERROR: TypeError: no implicit conversion of String into Integer

The only line in the server log is this:
2017/07/20 12:55:55 [DEBUG] [main] (goiardi.go:(*interceptHandler).ServeHTTP:299) Serving /cookbooks -- GET

$ knife -v
Chef: 13.2.20
upgraded chef-dk (1.5.0-1 -> 2.0.28-1)

The cookbooks I tested are tested on the previous version of knife, and also on the chef-server where it still works. Not sure if there is further debug possible with the knife client.

This update happened only yesterday and I guess I can roll back to the older version, so this is just heads up that something is happening.

We're testing goiardi with ~1000 nodes and searches can take 2minutes to complete. I've started with a memory profile build, ran a search, interrupted the process and generated a memory profile of all allocations: https://pdf.yt/d/6HglUhBL9PGPKaOr

It seems like 6G of memory is allocated for a single search with this many nodes.

This would go hand in hand with the etcd/serf/consul issue, I think. The biggest problem with clustering goiardi now is with search, since search is currently inside goiardi.

Possibilities include supporting using solr, splitting the goiardi search into its own separate program, or having the goiardi processes communicate amongst each other about updates that need indexing.

I ran into an issue which I don't fully understand:

setup:

• 2+ environments
• hard version constraint of a dependency cookbook

$ knife environment compare                   
                     production  rollout  staging
windows              latest      = 3.1.1  latest 

However the node resolves to the latest version (3.1.2 in this case):

$ chef-client -E rollout
...
  - windows (3.1.2)
  - yum-epel (2.1.2)
  - zap (0.15.1)
  - zypper (0.4.0)
…

Using knife download cookbook windows -E rollout i'm also getting the 3.1.2 release instead of the expected 3.1.1 release. That said, knife download cookbook windows --cookbook-version 3.1.1 works as expected. 😕

Hello.
I'm currently running goiardi as docker containers in AWS ECS. Btw, it working well as multiple instances behind the balancer (postgres and s3 storage enabled), so it works as a cluster now.
When I'm setting use_auth=true every anonymous http request returns HTTP 401. Which is not good because of AWS load balancer using http requests to evaluate container health.
Do goiardi supports any endpoint which returns HTTP 200 for anonymous requests when authentication is enabled.
Thank you.

I've started goiardi without a hostname parameter, uploaded a cookbook. The client could download it. I've restarted the server with a hostname parameter, and the client tried the filestore urls with the old (default) hostname. After I've uploaded the cookbook again it worked.

Describe the bug
When doing knife upload user, goiardi wont permit the operation because knife adds several fields which are not recognised by goiardi.

I.e. given this user.json file

{
"name": "user",
"email": "[email protected]",
"public_key": "-----BEGIN PUBLIC KEY-----xxx-----END PUBLIC KEY-----\n\n"
}

knife upload user will attempt following upload

{
   "name":"user",
   "username":"user",
   "display_name":"user",
   "admin":false,
   "json_class":"Chef::WebUIUser",
   "chef_type":"webui_user",
   "salt":null,
   "password":null,
   "openid":null,
   "email":"[email protected]",
   "public_key":"-----BEGIN PUBLIC KEY-----xxxxxx-----END PUBLIC KEY-----\n\n"
}

which will trigger at least 2 different failures:

1. password cannot be null (validation fails, it would not fail with "")
2. following fields are not recognised: "chef_type", "display_name", "json_class", "openid" (goiardi has a very strict validation rules, if there are any extra fields in request, it would be refused.

Expected behavior
standard operation such as knife upload working

Additional context
This was observed on both cinc (15) and chefdk (3.9) knife versions

I had to remove the following two lines in order to get the schema to run in Amazon RDS MySQL (no SUPER):
https://github.com/ctdk/goiardi/blob/master/sql-files/goiardi-schema-mysql.sql#L649
https://github.com/ctdk/goiardi/blob/master/sql-files/goiardi-schema-mysql.sql#L668

Removing them allows an error free loading of the schema.

When using goiardi with serf for shovey/shob, goiardi does not reconnect to serf when the connection gets dropped (e.g. serf instance restarts).
Log:

2017/07/28 16:38:57 [ERR] agent.client: Failed to decode response header: EOF

I reached out to @ctdk on Twitter, but I figured I'd open an issue for posterity.

I'm fairly certain Goiardi will no longer work if built with Go 1.6.0+ (at the time of writing). I don't believe there is anything the Goiardi project can do to alleviate this issue as it is a problem in net/http. The problem is due to this commit:

• golang/go@c052222

One of the HTTP headers used by the Chef authentication mechanism has a space in it (Hashed Path). Unfortunately, net/http appears to reject the request before your handler has the chance to see it due to the behavior introduced by the commit above. This was tested using an *http.Server with a hand-built ` I've opened an issue on the Go project regarding this being a problem:

• golang/go#14392

Hi, we'd like to contribute to the project but atm it's not clear what branch should be used. Master branch and latest stable release are pretty old atm, should we start on 1.0.0-dev?

Thanks @ctdk

When specifying an IP address as an option, goiardi should check the input and make sure it's a reasonable value. It ought to die eventually in there somewhere, but it should be more explicit with checking rather than hoping for the right behavior.

When using postgresql those records won't get cleaned up/purged. For node_statuses and reports goiardi users may have custom requirements to keep old records forever, so this is not an issue - I can add a cron job to delete very old entries.

However I'm not sure about the rows in sandboxes. Can I delete them after a while, too?

Thanks :)

I don't know if you are aware of the Chef-Browser Project [1] but this is a very nice and simple web frontend for the official chef-server and thus should also work with goiardi. Maybe someone could do a quick test of this with goiardi and if it works some small documentation (mainly that it exists and that it is an alternative to the old official frontend) would be nice.
They also provide a docker container [2] which for sure would be easiest way to get everything running.

[1] https://github.com/3ofcoins/chef-browser
[2] https://quay.io/repository/3ofcoins/chef-browser

But it should.

Is your feature request related to a problem? Please describe.
I'm writing a test of my project using goiardi, where a client (c1) fetches the public key of another client (c2).
Endpoint: clients/c2/keys/default
In real chef-server, the test passes when c1 is non-admin, whereas in goiardi the test cannot pass, because one (non-admin) client cannot fetch another client's public key (returned 403) by this code.
I have not yet understand chef-server's code, but I guess there might be an implementation difference between chef-server and goiardi.

Describe the solution you'd like
It should be correct behaviour that non-admin client c1 can fetch c2's public key.

Describe alternatives you've considered
For now, I'm testing with c1 admin in the test.

Additional context
With chef-server, c1 can fetch c2's public key like following:

$ knife client show c1
admin:     false
chef_type: client
name:      c1
validator: false

$ knife client key show c2 default
client:          c2
expiration_date: infinity
name:            default
public_key:      -----BEGIN PUBLIC KEY-----
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAyY7Acmtj1Y68QBPz5RoV
(snip)
-----END PUBLIC KEY-----

Hi,

We recently migrated from the official chef-server to goiardi and just stumbled upon an error when manually running "knife client create".

% knife client create delete_me -d -f delete_me.pem
ERROR: The data in your request was invalid
Response: Invalid key create_key in request body
% knife --version
Chef: 14.1.12

According to the chef-server API docs this create_key is valid [1], but goiardi does not know about it [2]. Currently this means for us that Terrform can not use the chef provisioner which is kind of a blocker for us. Any suggestion on how to quickly fix this? Or do you think this option could simply be added/allowed?

[1] https://docs.chef.io/api_chef_server.html#id24
[2]

Sometimes it's nice to do integration tests when for talking to chef. The net/http package does these end-to-end tests in places like src/pkg/net/http/client_test.go:70.

It would be lovely to have a one-liner to spin up goiardi from code with something less dirty than exec.Command(). Obviously it wouldn't need real persistence or TLS or perhaps even signal handling.

(This is all part of my grand scheme to get discoteq, chef-golang and goiardi into debian unstable.)