Events that are from an app should have a real app name, space and org information. about firehose-to-syslog HOT 7 CLOSED

https://github.com/cloudfoundry-community/go-cfclient could be used for this I think.

@rkoster mentioned this https://github.com/sourcegraph/apiproxy that could be used to provide caching.

from firehose-to-syslog.

Yes caching is needed to avoid asking to CC every request.

from firehose-to-syslog.

@simonjohansson Given that the loggregator team seem pretty adamant about no adding this information in loggregator - cloudfoundry/loggregator#48 - and we would like this "firehose-to-syslog" nozzle to eventually be adopted by them; I propose that we don't add this functionality here.

I also think we can sidestep the whole caching complexity by leveraging the translate filter's ability to lookup values from an external text file.

I've put together a simple prototype of how this would work. Essentially,

• The cf_app_details_cache job creates a yml file containing the lookup information at /var/vcap/store/cf_app_details_cache/cf_app_space_org_dictionary.yml

  '9af4f832-32cf-47c1-bce8-f2e852ea0730': '{"cf_app_id":"9af4f832-32cf-47c1-bce8-f2e852ea0730","cf_app_name":"app-logs","cf_space_id":"5c98b860-f5d2-444c-b923-d91b277b5269","cf_space_name":"production","cf_org_id":"8005f45b-76d9-4038-8ca1-9e0a85ed5be0","cf_org_name":"system"}'
  '54967f0c-5069-4428-83de-84f86c1286e2': '{"cf_app_id":"54967f0c-5069-4428-83de-84f86c1286e2","cf_app_name":"devApp1","cf_space_id":"a871a722-cad2-4d46-8061-2c9b728b7d8f","cf_space_name":"development","cf_org_id":"8005f45b-76d9-4038-8ca1-9e0a85ed5be0","cf_org_name":"system"}'
  

• The log_filter rules use this lookup file as their dictionary_path to decorate each log message (reloading the file every refresh_interval seconds)

I'm hoping you can help me write a simple CLI app that can be triggered to query the CC for space data and write said yml file; which, since it won't need do to any caching etc should hopefully be really simple.

Thoughts?

from firehose-to-syslog.

@mrdavidlaing The name lookup and space(including caching) is already on going using boltdb you can check the PR there #16 is not finished yet, but it s pretty close.

Using boltd is much more flexible than using a plan yam file.

#16

from firehose-to-syslog.

@mrdavidlaing I finish to integrate org, space and apps
So please try out my branch https://github.com/shinji62/firehose-to-syslog/tree/feature/Application_name_lookup

from firehose-to-syslog.

Should be done.

from firehose-to-syslog.

👍

from firehose-to-syslog.