Property reference resolution implemented? about serverless-plugin-modularize HOT 14 CLOSED

@tomhalley Hey, thanks for using the plugin. Can you provide a sample (subset) of the provider defined in one of your services located with the glob pattern?

Also, last I was aware, variables for the Serverless framework are defined within the custom block of yaml. If I'm misunderstanding the issue please let me know.

from serverless-plugin-modularize.

In example:

service/a/serverless.yaml

custom:
  some_variable: true
  
provider:
  environment:
    VARIABLE_FOR_A: 'http://localhost:3000'

service/b/serverless.yaml

custom:
  another_variable: 'a variable defined in b'

When merged at the root level, these values will be:

custom:
  some_variable: true
  another_variable: 'a variable defined in b'

provider:
  environment:
    VARIABLE_FOR_A: 'http://localhost:3000'

from serverless-plugin-modularize.

Ah sorry maybe the wrong terminology, being a bit simple today. I mean when you're referencing properties within the same serverless.yml file, like ${self:provider.environment.DYNAMODB_TABLE}

Here is a subset of one of the sub-services serverless.yml configs with just the DYNAMODB_TABLE property that isn't resolving properly.

service: my-funky-service-api

unresolvedVariablesNotificationMode: error
configValidationMode: error

provider:
  name: aws
  runtime: nodejs14.x
  region: XXXXX
  environment:
    DYNAMODB_TABLE: ${self:custom.dynamoDbTableName}

custom:
  dynamoDbTableName: ${self:service}-${env:stage, self:provider.stage}-Table

resources:
  Resources:
    FunkyDynamoDBTable:
      Type: 'AWS::DynamoDB::Table'
      DeletionPolicy: Retain
      Properties:
        AttributeDefinitions:
          - AttributeName: id
            AttributeType: S
        KeySchema:
          - AttributeName: id
            KeyType: HASH
        BillingMode: PAY_PER_REQUEST
        TableName: ${self:custom.dynamoDbTableName}

And judging by the error I'm seeing in AWS, it looks like these variables are going straight through to AWS without being resolved by serverless first. Again, tested these independently and all fine, but have issues when running through the root serverless.yml.

from serverless-plugin-modularize.

This is the serverless documentation I'm working from: https://www.serverless.com/framework/docs/providers/aws/guide/variables#reference-properties-in-serverlessyml

from serverless-plugin-modularize.

How are you deriving env:stage here?

custom:
  dynamoDbTableName: ${self:service}-${env:stage, self:provider.stage}-Table

If this env:stage isn't provided, it should default to self:provider.stage, but from the example's provided I don't see a stage value present in either provider.

That could be why you aren't able to resolve the values 🤷🏻 . Unless they changed something shouldn't stage be resolved via options like: ${opts:stage, '<default_value'>}

from serverless-plugin-modularize.

As I said before, the independent services deployed absolutely fine by themselves with sls deploy. I've gone back and changed it to try your suggestion and then running sls deploy --stage=dev and exactly the same result; both services deploying independently, but same error as before hen deploying through serverless-plugin-modularize:

1 validation error detected: Value '${self:custom.dynamoDbTableName}' at 'tableName' failed to satisfy constraint: Member must satisfy regular expression pattern: [a-zA-Z0-9_.-]+ (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ValidationException; Request ID: BDO9O35Q8EIOOFU5JE8QD9DURNVV4KQNSO5AEMVJF66Q9ASUAAJG; Proxy: null)

from serverless-plugin-modularize.

Okay, I think there may have been some misunderstanding about the way this plugin operates. It doesn't merge multiple services together, it allows you to modularize sections of your project/service. For example, lets say apis are separated from some non user-facing lambdas. You would/could define them in an api/ folder with a potential file called apis.yaml which could be merged into the main root service.

from serverless-plugin-modularize.

Assuming I'm wrong and your aren't trying to merge services. I have seen a lot of projects using the file suffix .yaml. I'm wondering if it could be something as simple as that?

Also, I generated a small project using your examples and modified the glob pattern to use:

custom:
  modularize:
    glob: 'services/*/serverless.yaml'

... and it worked.

You can verify if things are merging correctly by running this command in the root of your project.

npx serverless modularize merged

This should spit out the result of the merged yaml/yml files.

Example project tree:

├── node_modules
├── package.json
├── serverless.yaml
├── services
│   └── a
│       └── serverless.yaml
└── yarn.lock

from serverless-plugin-modularize.

from serverless-plugin-modularize.

I run into the same issue when upgrading from Serverless 2.11.1 to 2.45.1

Apparently referenced variables that are in the same file will render, but those references that are in other files do not.

Example, ${self:provider.stage}, defined in the root serverless.yml file, will not render in any "child" file, unless it's redefined in it.
${self:custom.x} will not render if x is defined under the root file's custom variable, but it will if it's defined under the same file's custom.

When the variable is not rendered, the expression will be kept in the string (instead of rendering empty), hence the validation error, as table names cannot contain $ or {.

Thanks :)

from serverless-plugin-modularize.

I assumed what your plugin would be doing would be to ignore any service
names pulled in from child files, merge all of the config sections you have
listed as supported in the readme, and ignore everything else.

Now that's a thought, I just released a 2.0.0 version. I can probably add a delete for any service value for incoming configs and see if it does what you want.

from serverless-plugin-modularize.

@icarus-sullivan Thanks for developing this plugin.
This is a very nice program but while it's aim is to increase flexiibility and modularity it gives extra restrictions.

I run into the same problem as @bros4president .

I have 1 root serverless.yml
in a child example.module.yml i want to use the serverless.com variable resolution.
See example

functions:
  some-function:
    handler: index.doSomething
    environment:
      name: ${self:custom.test} #this does not get resolved
    events:
      - schedule: rate(1 hour)

But this doesnt work

As you can see in AWS lambda env:

Any solution on how to fix this ?

from serverless-plugin-modularize.

I created a work-around function.
It's a bit hacky but it works.
It uses the "serverless modularize merged" command to write to a json file.
Then i removed the plugin from the generated serverless json file.
Then i use this file as the "serverless.yml". When i deploy this file variable resolution will work and i still have the benefits of module merging.

async function prepareDeploy() {
    console.log('Preparing deployment serverless.yaml')

    const generatedServerlessJsonName = '__generated_serverless.json'
    const outputServerlessConfigName = 'deployable_serverless_config.yml'

    //generate serverless yaml for deployment based on serverless template using the modularize plugin.
    await executeCommand(`cd serverless && rm -rf ${generatedServerlessJsonName}`)
    //command "serverless modularize merged" needs existing "serverless.yml" file, so create it temporary
    await executeCommand(`cd serverless && cp serverless_template.yml serverless.yml`)
    await executeCommand(`cd serverless && serverless modularize merged >> ${generatedServerlessJsonName}`)
    await executeCommand(`cd serverless && rm -rf serverless.yml`)

    //fix generated file
    let fileContents = fs.readFileSync(`serverless/${generatedServerlessJsonName}`, 'utf8')

    //remove output generated by "serverless modularize merged command"
    fileContents = fileContents.replace('�[32mmodularize:�[0m ', '')

    //remove the modularize plugin, or it will override on deploy and variable resolving will be ignored like before
    const fileJson = JSON.parse(fileContents)
    fileJson.plugins = fileJson.plugins.filter((x) => x !== 'serverless-plugin-modularize')
    fileJson.service = getServiceNameFromServerlessConfig()

    fs.writeFileSync(`serverless/${outputServerlessConfigName}`, `#GENERATED FILE. DO NOT EDIT DIRECTLY. USE serverless_template.yaml\n${yaml.stringify(fileJson)}`)

    //cleanup generated files
    await executeCommand(`cd serverless && rm -rf ${generatedServerlessJsonName}`)
    console.log(`Done generating ${outputServerlessConfigName}`)
}

from serverless-plugin-modularize.

I got this problem too but I was able to hack it to parse variables, ssm, or even json config (like ${file(../../env.${opt:stage, 'local'}.json):HASURA_ENDPOINT}) by triggering this error

.bin/serverless modularize merged --stage dev
Running "serverless" from node_modules
Serverless: Deprecation warning: Variables resolver reports following resolution errors:
              - Cannot resolve variable at "provider.environment.DON_NOT_REMOVE_ME": Value not found at "ssm" source
            From a next major this will be communicated with a thrown error.
            Set "variablesResolutionMode: 20210326" in your service config, to adapt to new behavior now
            More Info: https://www.serverless.com/framework/docs/deprecations/#NEW_VARIABLES_RESOLVER

 Serverless Warning --------------------------------------

  A valid SSM parameter to satisfy the declaration 'ssm:/DON_NOT_REMOVE_ME' could not be found.


 Serverless Warning --------------------------------------

  A valid SSM parameter to satisfy the declaration 'ssm:/DON_NOT_REMOVE_ME' could not be found.

DON_NOT_REMOVE_ME must not exist in SSM Parameter Store, otherwise no warnings => nothing is parsed.

I have no clue why. This only works for serverless 2.45.1. I've tried serverless 3.5.1, this warning is now an error so it won't work.

from serverless-plugin-modularize.