Code Monkey home page Code Monkey logo

go-mod-configuration's Introduction

go-mod-configuration

Build Status Code Coverage Go Report Card GitHub Latest Dev Tag) GitHub Latest Stable Tag) GitHub License GitHub go.mod Go version GitHub Pull Requests GitHub Contributors GitHub Committers GitHub Commit Activity

Configuration client library for use by Go implementation of EdgeX micro services. This project contains the abstract Configuration API and an implementation for Consul. The API initializes a connection to the Configuration service and push/pull configuration values to/from the Configuration service.

What is this repository for?

  • Initialize connection to a Configuration service
  • Push a service's configuration in to the Configuration
  • Pull service's configuration from the Configuration service into its configuration struct
  • Listen for configuration updates

How to Use

This library is used by Go programs for interacting with the Configuration service (i.e. Consul) and requires that a Configuration service be running somewhere that the Configuration Client can connect. The types.ServiceConfig struct is used to specify the service implementation details :

type ServiceConfig struct {
	Protocol string
	Host string
	Port int
	Type string
	BasePath string
	AccessToken string
}

The following code snippets demonstrate how a service uses this Configuration module to store and load configuration, listen to for configuration updates.

This code snippet shows how to connect to the Configuration service, store and load the service's configuration from the Configuration service.

func initializeConfiguration(useConfigService bool, useProfile string) (*ConfigurationStruct, error) {
	configuration := &ConfigurationStruct{}
	err := config.LoadFromFile(useProfile, configuration)
	if err != nil {
		return nil, err
	}

    if useConfigService {
        serviceConfig := types.Config{
            Host:            conf.Configuration.Host,
            Port:            conf.Configuration.Port,
            Type:            conf.Configuration.Type,
            BasePath:        internal.ConfigStem + internal.MyServiceKey,
            AccessToken:     <AccessTokenLoadedFromSecretFile>,
        }

        ConfigClient, err = configuration.NewConfigurationClient(serviceConfig)
    	if err != nil {
    		return fmt.Errorf("connection to Configuration service could not be made: %v", err.Error())
    	}


		hasConfig, err := ConfigClient.HasConfiguration()
		if hasConfig {
            // Get the service's configuration from the Configuration service
            rawConfig, err := ConfigClient.GetConfiguration(configuration)
            if err != nil {
                return fmt.Errorf("could not get configuration from Configuration: %v", err.Error())
            }

            actual, ok := rawConfig.(*ConfigurationStruct)
            if !ok {
                return fmt.Errorf("configuration from Configuration failed type check")
            }

            *configuration = actual
        } else {
            err = ConfigClient.PutConfiguration(configuration, true)
			if err != nil {
				return fmt.Errorf("could not push configuration into Configuration service: %v", err)
			}
        }
        
        // Run as go func so doesn't block
        go listenForConfigChanges()
    }

This code snippet shows how to listen for configuration changes from the Configuration service after connecting above.

func listenForConfigChanges() {
	if ConfigClient == nil {
		LoggingClient.Error("listenForConfigChanges() configuration client not set")
		return
	}

	ConfigClient.WatchForChanges(updateChannel, errChannel, &WritableInfo{}, internal.WritableKey)

	signalChan := make(chan os.Signal)
	signal.Notify(signalChan, syscall.SIGINT, syscall.SIGTERM)

	for {
		select {
		case <- signalChan:
			// Quietly and gracefully stop when SIGINT/SIGTERM received
			return

		case ex := <-errChannel:
			LoggingClient.Error(ex.Error())

		case raw, ok := <-updateChannel:
			if !ok {
				return
			}

			actual, ok := raw.(*WritableInfo)
			if !ok {
				LoggingClient.Error("listenForConfigChanges() type check failed")
				return
			}

			Configuration.Writable = *actual

			LoggingClient.Info("Writeable configuration has been updated")
			LoggingClient.SetLogLevel(Configuration.Writable.LogLevel)
		}
	}
}

go-mod-configuration's People

Contributors

akramtexas avatar bill-mahoney avatar bnevis-i avatar cloudxxx8 avatar dependabot[bot] avatar ejlee3 avatar ernestojeda avatar jackchenjc avatar jameskbutcher avatar jamesrgregg avatar jim-wang-intel avatar jpwhitemn avatar jumpingliu avatar jwagantall avatar lenny-goodell avatar rsdmike avatar soda480 avatar tsconn23 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

go-mod-configuration's Issues

[Configuration] New APIs to simplify pull and updates

๐Ÿš€ Tech Debt

Relevant Package [REQUIRED]

This feature request is for APIs to pull configuration and receive updated configuration (i.e. `Writable`) and the Consul implementation.

Per Napa planning this will not be included due to risk of destabilizing the release. Task for this release is to agree on the approach and possibly create a POC.

Description [REQUIRED]

The current APIs expect the service's complete config struct to be passed in when pulling config and waiting for updates. The Consul implementation uses `github.com/mitchellh/consulstructure` in both cases.

There are two issues with this approach:

  1. github.com/mitchellh/consulstructure doesn't support pull/marshaling slices.
  2. github.com/mitchellh/consulstructure updates the passed in service's config struct with the complete Writable section when a setting changes and doesn't provide indication of which settings have changed. With the need to merge configurations for the recent Common Config implementation, this has lead to this issue when map items are removed.

Describe the solution you'd like

  • Add new APIs (deprecate old versions) for pulling complete configuration and waiting for updates.
    • GetConfigurationMap() {map[string]any, error)
      This new API returns the service's complete configuration as a map which the caller can then use to merge/marshal into the service's config struct.
    • WatchForUpdates(updateChannel chan<- map[string]Update, errorChannel chan<- error, watchKey string)
      This new API will watch for updates, post a map containing only the fields that have changed. The setting paths are the map keys and the map values contain the new setting value and if the setting is existing, new or deleted (for handling of maps/slices).
  • Consul implementation would no longer use github.com/mitchellh/consulstructure. Instead would implement pulling/waiting internally. Long request on watchKey url for Updates and simple pull on root key for Get.

go-mod-bootstrap can then be refactored to use these new APIs to resolve edgexfoundry/go-mod-bootstrap#534 and enable slice as a type for configuration items.

Describe alternatives you've considered

Have you considered any alternative solutions or workarounds?

unable to decode "[]" into an empty slice

๐Ÿž Bug Report

Affected Services [REQUIRED]

The issue is located in: consul/client.go and keeper/client.go

Is this a regression?

Yes, the previous version in which this bug was not present was: Levski

Description and Minimal Reproduction [REQUIRED]

Run Napa non-secure stack compose file
Query all device services: http://localhost:59881/api/v3/deviceservice/all

๐Ÿ”ฅ Exception or Error



Actual Result:
{
    "apiVersion": "v3",
    "statusCode": 200,
    "totalCount": 2,
    "services": [
        {
            "created": 1714028539826,
            "modified": 1714028539826,
            "id": "f4723958-3c7a-4eea-a384-c1ac77490b80",
            "name": "device-virtual",
            "labels": [
                "[]"
            ],
            "baseAddress": "http://edgex-device-virtual:59900",
            "adminState": "UNLOCKED"
        },
        {
            "created": 1714028539824,
            "modified": 1714028539824,
            "id": "45e07fe7-7fe8-4116-b32b-84f788cfd361",
            "name": "device-rest",
            "labels": [
                "[]"
            ],
            "baseAddress": "http://edgex-device-rest:59986",
            "adminState": "UNLOCKED"
        }
    ]
}

Expected Result:
{
    "apiVersion": "v3",
    "statusCode": 200,
    "totalCount": 2,
    "services": [
        {
            "created": 1714028539826,
            "modified": 1714028539826,
            "id": "f4723958-3c7a-4eea-a384-c1ac77490b80",
            "name": "device-virtual",
            "baseAddress": "http://edgex-device-virtual:59900",
            "adminState": "UNLOCKED"
        },
        {
            "created": 1714028539824,
            "modified": 1714028539824,
            "id": "45e07fe7-7fe8-4116-b32b-84f788cfd361",
            "name": "device-rest",
            "baseAddress": "http://edgex-device-rest:59986",
            "adminState": "UNLOCKED"
        }
    ]
}

๐ŸŒ Your Environment

Deployment Environment:

EdgeX Version [REQUIRED]: Napa

Anything else relevant?

Add API to Config Client to All absolute path for key

Add new API that allows bypassing the clients base stem. This allows the additional client instance created just for the Common Config ready check in go--mod-bootstrap

GetConfigurationValueByFullPath(path string) ([]byte, error)

Upgrade to use Consul API 1.12.0

๐Ÿš€ Feature Request

Relevant Package [REQUIRED]

Consul API module

Description [REQUIRED]

Version 1.12 causes breaks our code
See #56

Describe the solution you'd like

Adjust code to work with 1.12

Describe alternatives you've considered

Stay on 1.9.x

ServiceConfig.PopulateFromUrl default to HTTP if protocol not specified

For backwards compatibility, the helper function ServiceConfig.PopulateFromUrl need to default to HTTP rather then error out when protocol is not specified.

Current URL spec is <provider type>.<protocol>://<host name>:<port>
but need s to also support <provider type>://<host name>:<port> for backwards compatibility.

This is in support of Device SDK using go-mod-bootstrap and backwards support for --registry=<url>.

Add Optional field for configuration provider ServiceConfig struct

๐Ÿš€ Feature Request

Relevant Package [REQUIRED]

This feature request is for all services using configuration provider

Description [REQUIRED]

Currently, we have the implementation to support Consul as the configuration provider.
For more configuration providers implementation in the future, we should add an extendable field in the ServiceConfig struct to make the extension easier.

Describe the solution you'd like

Add an Optional field with the map[string]interface type in the ServiceConfig struct to handle any future extension.

Add API to check if sub-configuration exists

Currently the API HasConfiguration() simply checks if the top level key exists for the service.
For the App Service new capability of a custom configuration section we need the ability to check for the existence of this custom sub configuration.

Upgrade to Go 1.15.2

The following files also need to be updated:

Dockerfile.build
go.mod
Jenkinsfile (pull go reference)
README (update min go??)

Add Access Token renew callback function capability

The Access Token (when in secure mode) gets revoked when the Vault token is revoked. This will occur if Secret Store Setup is restarted when adding add-on services.

The configuration needs to have a RenewAccessToken callback function that the client will call any time it receives a access denied error. It will then retry the failed request with the new access token

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.