goldfiglabs / introspector Goto Github PK

A schema and set of tools for using SQL to query cloud infrastructure.

License: Mozilla Public License 2.0

Python 53.03% PLpgSQL 45.80% Shell 0.31% Go 0.85%

aws aws-security cloud-infrastructure cloud-security cloudsecurity cmdb compliance-as-code compliance-automation cspm database-schema devsecops iam infosec infrastructure-as-deployed postgres secops security security-groups security-tools sql

introspector's People

Contributors

Stargazers

Watchers

Forkers

atdavidpark syllogy devesh-bedrock devdoshi acumenix

introspector's Issues

Skip AWS Organizations

The tool looks interesting, but assumes that there's an AWS organization in place (and that it's accessible to the user). Is there a plan to remove that requirement? https://github.com/goldfiglabs/goldfig/blob/main/goldfig/aws/__init__.py#L92

Thanks for publishing your work.

Move tool-specific migrations into their own migrations folder

Currently, some migrations and db requirements (like plv8) are only used for some of the tools, like cis foundation. If these migrations move into their own migration directory and only run when using those tools, other users who don't need plv8 can depend on a much smaller postgres image.

How to find long policy documents?

IAM limits the length of a managed policy document after removing whitespace to 6144 characters.

Uncontrolled growth in of a policy document will eventually lead to an error when the length exceeds the limit. This will impair or block anything depending on the policy to provide it permissions.

I've tried querying the instrospector database like this to find such long documents.

SELECT document::text FROM aws_iam_policyversion;

Unfortunately the jsonb type when cast to text puts spaces after the commas and the colons.

I don't see any other way in SQL to get the compact string representation (like jq -c).

I have a custom Python script for doing it which uses a function like this:

def get_default_document_for_policy(policy):

    resp = boto3.client("iam").get_policy_version(
        PolicyArn=policy["Arn"], VersionId=policy["DefaultVersionId"]
    )

    return json.dumps(resp["PolicyVersion"]["Document"], separators=(",", ":"))

How would I adapt this to Introspector?

Help message shouldn't depend on valid AWS profile

Just a small usability feature to make introspector easier to learn.

I don't have a default AWS profile in my environment.

When I set an AWS profile for the command the help message does appear.

$ AWS_PROFILE=introspector ./introspector account aws import --help
Usage: introspector.py account aws import [OPTIONS]

  Imports the assets from an AWS account into introspector

Options:
  -f, --force         Skip prompts for adding accounts
  -s, --service TEXT  Only import the specified service(s)
  -g, --gov-cloud     Set this flag to import a govcloud account
  --help              Show this message and exit.

I would expect the help message to appear without an AWS profile set, but instead it prints an error.

$ ./introspector account aws import --help
panic: no EC2 IMDS role found, operation error ec2imds: GetMetadata, request canceled, context deadline exceeded

goroutine 1 [running]:
main.main()
	/Users/greg/workspace/introspector/launcher/main.go:208 +0x3d9

Tabulate version issue is causing results to not render via CLI

Wrapped results via the Introspector CLI aren't rendering properly:

Traceback (most recent call last):
  File "/app/introspector.py", line 5, in <module>
    run_cli()
  File "/app/introspector/cli/__init__.py", line 60, in run_cli
    cli()
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/app/introspector/cli/query.py", line 34, in cmd
    print(tabulate(results.rows, headers=results.columns))
  File "/usr/local/lib/python3.9/site-packages/tabulate.py", line 1528, in tabulate
    list_of_lists, headers = _normalize_tabular_data(
  File "/usr/local/lib/python3.9/site-packages/tabulate.py", line 1161, in _normalize_tabular_data
    raise ValueError(
ValueError: headers for a list of dicts is not a dict or a keyword

🤖 15:42:34 anglegrinder:tmp$

Support AWS sub-accounts

Introspector only supports master accounts. Add support for sub-accounts.

introspector is not authorized to perform: acm-pca:ListCertificateAuthorities

It looks like the recommended permissions are lacking acm-pca:ListCertificateAuthorities.

I don't see it listed in SecurityAudit, ViewOnlyAccess or Introspector-Ro-Additions.

Introspector is failing for me because of that.

$ AWS_PROFILE=introspector ./introspector account aws import
Add AWS account 01234567890 using identity arn:aws:iam::01234567890:user/introspector? [Y/n] Y
[2021-03-19 19:09:19,636] {__init__.py:88} ERROR - exception caught in import
concurrent.futures.process._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/concurrent/futures/process.py", line 243, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "/app/introspector/aws/svc.py", line 215, in _async_proxy
    for resource_name, raw_resources in import_fn(service_proxy, region,
  File "/app/introspector/aws/acm_pca.py", line 34, in _import_acm_pca_region
    yield from _import_certificate_authorities(proxy)
  File "/app/introspector/aws/acm_pca.py", line 23, in _import_certificate_authorities
    certificate_authorities_resp = proxy.list('list_certificate_authorities')
  File "/app/introspector/aws/fetch.py", line 396, in list
    return self._impl.list(resource, kwargs)
  File "/app/introspector/aws/fetch.py", line 100, in list
    raise e
  File "/app/introspector/aws/fetch.py", line 58, in list
    result = iterator.build_full_result()
  File "/usr/local/lib/python3.9/site-packages/botocore/paginate.py", line 449, in build_full_result
    for response in self:
  File "/usr/local/lib/python3.9/site-packages/botocore/paginate.py", line 255, in __iter__
    response = self._make_request(current_kwargs)
  File "/usr/local/lib/python3.9/site-packages/botocore/paginate.py", line 332, in _make_request
    return self._method(**current_kwargs)
  File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 676, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (AccessDeniedException) when calling the ListCertificateAuthorities operation: User: arn:aws:iam::01234567890:user/introspector is not authorized to perform: acm-pca:ListCertificateAuthorities
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/app/introspector/__init__.py", line 85, in collect_exceptions
    _ = result.result()
  File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
    return self.__get_result()
  File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
botocore.exceptions.ClientError: An error occurred (AccessDeniedException) when calling the ListCertificateAuthorities operation: User: arn:aws:iam::01234567890:user/introspector is not authorized to perform: acm-pca:ListCertificateAuthorities

The above errors repeat several times and finally it prints results.

Results - Import #1
  Start:  2021-03-19 19:09:07
  End:  2021-03-19 19:09:51
  Errors:  16
  Resources added:  0
  Resources removed:  0
  Resources updated:  0
    Attributes added:  0
    Attributes removed:  0
    Attributes updated:  0

Suppress logging when URI for a relation cannot be constructed

In my first run of Inspector (#11) I saw this warning about billing alarm.

[2021-03-20 14:23:25,481] {resource.py:65} WARNING - Missing target for relation arn:aws:cloudwatch:us-east-1:01234567890:alarm:BillingAlert fires-on [metrics/us-east-1/None/None/] (1)

It turns out that it's not actually a problem with the alarm configuration but a limitation of introspector when deriving relations from the resources.

@gsoltis commented:

For some of the calculated relations, as you can see from the None token, we don't know the correct thing to relate them to (or it might not exist). These shouldn't actually be warnings at this point. If we can't construct the URI for a relation, we should not then be logging that we can't find the target...

(Please correct me if I have mixed this one up. It seems to be about an implementation detail that I don't understand yet.)

Warn when permissions are insufficient to complete the inventory

Version 4ec6c21 (commit ID) of the Introspector-Ro-Additions policy lacks the acm-pca:ListCertificateAuthorities permission.

Version e934e4551c83 (image ID) of the Introspector Docker image silently ignores permission errors.

So when the Introspector application attempts to list the certificate authorities using the introspector IAM user, it fails to list the private certificate authorities, but does not indicate that anything is missing from the inventory.

It would be nice if this would generate at least a warning so that I know that something might be missing from the database.

@gsoltis commented in #11:

The code does currently just swallow the error in the case of missing permissions, I agree that a warning would be a good idea. I think eventually we will want to record what was skipped as part of the import.

Switch released docker-compose file to use a version of the docker image

Currently the docker-compose.yml that gets packaged in a release points to goldfig/introspector:latest. This is brittle because we will continue to update the image.

We should instead pin github releases to a particular version of the docker image.

An error occurred (ReportInProgress) when calling the GetCredentialReport operation

I got an error about the IAM credential report when running an import today.

botocore.errorfactory.CredentialReportNotReadyException: An error occurred (ReportInProgress) when calling the GetCredentialReport operation: Unknown

I understand that GenerateCredentialReport and GetCredentialReport are distinct APIs and that Introspector will have to figure out somehow when the credential report is ready.

If it fails like this, does it mean that Introspector didn't get a credential report in the end?

Is my import incomplete?

$ ./introspector account aws import
[2021-03-24 19:53:38,385] {iam.py:93} ERROR - credenetial report fetch error
Traceback (most recent call last):
  File "/app/introspector/aws/iam.py", line 91, in _import_credential_report
    report = proxy.get('get_credential_report')
  File "/app/introspector/aws/fetch.py", line 404, in get
    return self._impl.get(resource, kwargs)
  File "/app/introspector/aws/fetch.py", line 131, in get
    raise e
  File "/app/introspector/aws/fetch.py", line 106, in get
    full_result = api_call(**kwargs)
  File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 676, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.CredentialReportNotReadyException: An error occurred (ReportInProgress) when calling the GetCredentialReport operation: Unknown
[2021-03-24 19:55:34,548] {resource.py:65} WARNING - Missing target for relation arn:aws:cloudwatch:us-east-1:480783779961:alarm:BillingAlert fires-on [metrics/us-east-1/None/None/] (1)
Results - Import #3
  Start:  2021-03-24 19:52:41
  End:  2021-03-24 19:55:37
  Errors:  0
  Resources added:  1
  Resources removed:  0
  Resources updated:  10
    Attributes added:  0
    Attributes removed:  0
    Attributes updated:  14

I'm using Docker image e934e4551c83 and release v0.1.2.

psycopg2.errors.CardinalityViolation: ON CONFLICT DO UPDATE command cannot affect row a second time

I'm trying to import a second AWS account to my introspector database.

The first account is the organization management account. The second acount is a member of that organization.

I created the introspector IAM user in the member acount and set its credentials via the AWS_PROFILE environment variable.

I got the following error:

[2021-03-24 21:11:47,038] {aws.py:106} ERROR - exception caught in map
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1245, in _execute_context
self.dialect.do_execute(
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 588, in do_execute
cursor.execute(statement, parameters)
psycopg2.errors.CardinalityViolation: ON CONFLICT DO UPDATE command cannot affect row a second time
HINT: Ensure that no rows proposed for insertion within the same command have duplicate constrained values.

Error in context:

$ ./introspector account aws import
[...truncate IAM credential report error...]
[2021-03-24 21:11:40,363] {resource.py:57} WARNING - Missing parent for relation arn:aws:ec2:us-west-2:012345678901:security-group/sg-00000000 in arn:aws:organizations::012345678901:account/o-abcdefghij/098765432109 (1)
[...truncate 100+ warnings about "Missing parent for relation"...]
[2021-03-24 21:11:44,653] {resource.py:57} WARNING - Missing parent for relation arn:aws:ec2:us-west-2:012345678901:vpc/vpc-00000000 in arn:aws:organizations::012345678901:account/o-abcdefghij/098765432109 (1)
[2021-03-24 21:11:47,038] {aws.py:106} ERROR - exception caught in map
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1245, in _execute_context
    self.dialect.do_execute(
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 588, in do_execute
    cursor.execute(statement, parameters)
psycopg2.errors.CardinalityViolation: ON CONFLICT DO UPDATE command cannot affect row a second time
HINT:  Ensure that no rows proposed for insertion within the same command have duplicate constrained values.


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/app/introspector/cli/account/aws.py", line 103, in import_aws_cmd
    refresh_views(db, reloaded_import_job.provider_account_id)
  File "/app/introspector/bootstrap_db.py", line 157, in refresh_views
    result = db.execute(query)
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 1277, in execute
    return self._connection_for_bind(bind, close_with_result=True).execute(
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 982, in execute
    return meth(self, multiparams, params)
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/sql/elements.py", line 293, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1095, in _execute_clauseelement
    ret = self._execute_context(
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1249, in _execute_context
    self._handle_dbapi_exception(
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1476, in _handle_dbapi_exception
    util.raise_from_cause(sqlalchemy_exception, exc_info)
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 398, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 152, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1245, in _execute_context
    self.dialect.do_execute(
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 588, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (psycopg2.errors.CardinalityViolation) ON CONFLICT DO UPDATE command cannot affect row a second time
HINT:  Ensure that no rows proposed for insertion within the same command have duplicate constrained values.

[SQL: INSERT INTO aws_config_configurationrecorder (
  _id,
  uri,
  provider_account_id,
  rolearn,
  allsupported,
  includeglobalresourcetypes,
  resourcetypes,
  name,
  laststarttime,
  laststoptime,
  recording,
  laststatus,
  lasterrorcode,
  lasterrormessage,
  laststatuschangetime,
  _iam_role_id,_account_id
)
SELECT
  R.id AS _id,
  R.uri,
  R.provider_account_id,
  rolearn.attr_value #>> '{}' AS rolearn,
  (allsupported.attr_value #>> '{}')::boolean AS allsupported,
  (includeglobalresourcetypes.attr_value #>> '{}')::boolean AS includeglobalresourcetypes,
  resourcetypes.attr_value::jsonb AS resourcetypes,
  name.attr_value #>> '{}' AS name,
  (TO_TIMESTAMP(laststarttime.attr_value #>> '{}', 'YYYY-MM-DD"T"HH24:MI:SS')::timestamp at time zone '00:00') AS laststarttime,
  (TO_TIMESTAMP(laststoptime.attr_value #>> '{}', 'YYYY-MM-DD"T"HH24:MI:SS')::timestamp at time zone '00:00') AS laststoptime,
  (recording.attr_value #>> '{}')::boolean AS recording,
  laststatus.attr_value #>> '{}' AS laststatus,
  lasterrorcode.attr_value #>> '{}' AS lasterrorcode,
  lasterrormessage.attr_value #>> '{}' AS lasterrormessage,
  (TO_TIMESTAMP(laststatuschangetime.attr_value #>> '{}', 'YYYY-MM-DD"T"HH24:MI:SS')::timestamp at time zone '00:00') AS laststatuschangetime,
  
    _iam_role_id.target_id AS _iam_role_id,
    _account_id.target_id AS _account_id
FROM
  resource AS R
  INNER JOIN provider_account AS PA
    ON PA.id = R.provider_account_id
  LEFT JOIN resource_attribute AS rolearn
    ON rolearn.resource_id = R.id
    AND rolearn.type = 'provider'
    AND lower(rolearn.attr_name) = 'rolearn'
  LEFT JOIN resource_attribute AS allsupported
    ON allsupported.resource_id = R.id
    AND allsupported.type = 'provider'
    AND lower(allsupported.attr_name) = 'allsupported'
  LEFT JOIN resource_attribute AS includeglobalresourcetypes
    ON includeglobalresourcetypes.resource_id = R.id
    AND includeglobalresourcetypes.type = 'provider'
    AND lower(includeglobalresourcetypes.attr_name) = 'includeglobalresourcetypes'
  LEFT JOIN resource_attribute AS resourcetypes
    ON resourcetypes.resource_id = R.id
    AND resourcetypes.type = 'provider'
    AND lower(resourcetypes.attr_name) = 'resourcetypes'
  LEFT JOIN resource_attribute AS name
    ON name.resource_id = R.id
    AND name.type = 'provider'
    AND lower(name.attr_name) = 'name'
  LEFT JOIN resource_attribute AS laststarttime
    ON laststarttime.resource_id = R.id
    AND laststarttime.type = 'provider'
    AND lower(laststarttime.attr_name) = 'laststarttime'
  LEFT JOIN resource_attribute AS laststoptime
    ON laststoptime.resource_id = R.id
    AND laststoptime.type = 'provider'
    AND lower(laststoptime.attr_name) = 'laststoptime'
  LEFT JOIN resource_attribute AS recording
    ON recording.resource_id = R.id
    AND recording.type = 'provider'
    AND lower(recording.attr_name) = 'recording'
  LEFT JOIN resource_attribute AS laststatus
    ON laststatus.resource_id = R.id
    AND laststatus.type = 'provider'
    AND lower(laststatus.attr_name) = 'laststatus'
  LEFT JOIN resource_attribute AS lasterrorcode
    ON lasterrorcode.resource_id = R.id
    AND lasterrorcode.type = 'provider'
    AND lower(lasterrorcode.attr_name) = 'lasterrorcode'
  LEFT JOIN resource_attribute AS lasterrormessage
    ON lasterrormessage.resource_id = R.id
    AND lasterrormessage.type = 'provider'
    AND lower(lasterrormessage.attr_name) = 'lasterrormessage'
  LEFT JOIN resource_attribute AS laststatuschangetime
    ON laststatuschangetime.resource_id = R.id
    AND laststatuschangetime.type = 'provider'
    AND lower(laststatuschangetime.attr_name) = 'laststatuschangetime'
  LEFT JOIN (
    SELECT
      _aws_iam_role_relation.resource_id AS resource_id,
      _aws_iam_role.id AS target_id
    FROM
      resource_relation AS _aws_iam_role_relation
      INNER JOIN resource AS _aws_iam_role
        ON _aws_iam_role_relation.target_id = _aws_iam_role.id
        AND _aws_iam_role.provider_type = 'Role'
        AND _aws_iam_role.service = 'iam'
    WHERE
      _aws_iam_role_relation.relation = 'acts-as'
  ) AS _iam_role_id ON _iam_role_id.resource_id = R.id
  LEFT JOIN (
    SELECT
      _aws_organizations_account_relation.resource_id AS resource_id,
      _aws_organizations_account.id AS target_id
    FROM
    (
      SELECT
        _aws_organizations_account_relation.resource_id AS resource_id
      FROM
        resource_relation AS _aws_organizations_account_relation
        INNER JOIN resource AS _aws_organizations_account
          ON _aws_organizations_account_relation.target_id = _aws_organizations_account.id
          AND _aws_organizations_account.provider_type = 'Account'
          AND _aws_organizations_account.service = 'organizations'
      WHERE
        _aws_organizations_account_relation.relation = 'in'
      GROUP BY _aws_organizations_account_relation.resource_id
      HAVING COUNT(*) = 1
    ) AS unique_account_mapping
    INNER JOIN resource_relation AS _aws_organizations_account_relation
      ON _aws_organizations_account_relation.resource_id = unique_account_mapping.resource_id
    INNER JOIN resource AS _aws_organizations_account
      ON _aws_organizations_account_relation.target_id = _aws_organizations_account.id
      AND _aws_organizations_account.provider_type = 'Account'
      AND _aws_organizations_account.service = 'organizations'
    WHERE
        _aws_organizations_account_relation.relation = 'in'
  ) AS _account_id ON _account_id.resource_id = R.id
  WHERE
  R.provider_account_id = 1 AND 
  PA.provider = 'aws'
  AND R.provider_type = 'ConfigurationRecorder'
  AND R.service = 'config'
ON CONFLICT (_id) DO UPDATE
SET
    rolearn = EXCLUDED.rolearn,
    allsupported = EXCLUDED.allsupported,
    includeglobalresourcetypes = EXCLUDED.includeglobalresourcetypes,
    resourcetypes = EXCLUDED.resourcetypes,
    name = EXCLUDED.name,
    laststarttime = EXCLUDED.laststarttime,
    laststoptime = EXCLUDED.laststoptime,
    recording = EXCLUDED.recording,
    laststatus = EXCLUDED.laststatus,
    lasterrorcode = EXCLUDED.lasterrorcode,
    lasterrormessage = EXCLUDED.lasterrormessage,
    laststatuschangetime = EXCLUDED.laststatuschangetime,
    _iam_role_id = EXCLUDED._iam_role_id,
    _account_id = EXCLUDED._account_id]
(Background on this error at: http://sqlalche.me/e/f405)
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1245, in _execute_context
    self.dialect.do_execute(
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 588, in do_execute
    cursor.execute(statement, parameters)
psycopg2.errors.InFailedSqlTransaction: current transaction is aborted, commands ignored until end of transaction block


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/app/introspector.py", line 5, in <module>
    run_cli()
  File "/app/introspector/cli/__init__.py", line 60, in run_cli
    cli()
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/app/introspector/cli/account/aws.py", line 112, in import_aws_cmd
    db.commit()
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 1036, in commit
    self.transaction.commit()
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 503, in commit
    self._prepare_impl()
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 482, in _prepare_impl
    self.session.flush()
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 2479, in flush
    self._flush(objects)
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 2617, in _flush
    transaction.rollback(_capture_exception=True)
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 153, in reraise
    raise value
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 2577, in _flush
    flush_context.execute()
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 422, in execute
    rec.execute(self)
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 586, in execute
    persistence.save_obj(
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 230, in save_obj
    _emit_update_statements(
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 995, in _emit_update_statements
    c = cached_connections[connection].execute(
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 982, in execute
    return meth(self, multiparams, params)
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/sql/elements.py", line 293, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1095, in _execute_clauseelement
    ret = self._execute_context(
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1249, in _execute_context
    self._handle_dbapi_exception(
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1476, in _handle_dbapi_exception
    util.raise_from_cause(sqlalchemy_exception, exc_info)
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 398, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 152, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1245, in _execute_context
    self.dialect.do_execute(
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 588, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.InternalError: (psycopg2.errors.InFailedSqlTransaction) current transaction is aborted, commands ignored until end of transaction block

[SQL: UPDATE import_job SET end_date=%(end_date)s, error_details=%(error_details)s WHERE import_job.id = %(import_job_id)s]
[parameters: {'end_date': datetime.datetime(2021, 3, 24, 21, 11, 47, 41410, tzinfo=datetime.timezone.utc), 'error_details': '["Traceback (most recent call last):\\n  File \\"/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py\\", line 1245, in _execute_context\ ... (9259 characters truncated) ... \n    _iam_role_id = EXCLUDED._iam_role_id,\\n    _account_id = EXCLUDED._account_id]\\n(Background on this error at: http://sqlalche.me/e/f405)\\n"]', 'import_job_id': 5}]
(Background on this error at: http://sqlalche.me/e/2j85)

I'm using Docker image e934e4551c83 and release v0.1.2.

Improve documentation of import --service flag

See misunderstanding in #18 for context.

I said:

It looks like the problem is in the aws_config_configurationrecorder table.

[SQL: INSERT INTO aws_config_configurationrecorder (

I tried to rule that out by setting the service flag to import just the ec2 service.

./introspector account aws import --service ec2

But I got the same error result.

@gsoltis said:

After running an import, even if it's just a single service (the --service flag definitely needs documentation...), introspector will resync the aws_ tables from the resource tables.

Running into issues at gf init

Hey Goldfig team, I'm trying out this repo for a tutorial I'm writing about maintaining AWS tags.

So far, I've got my containers running, but I run into the following error when I run init:

$ ./gf init
Checking for running goldfig
[2020-08-30 14:48:57,050] {bootstrap_db.py:85} ERROR - failed adding function from aws_logs_metricfilter_match.sql
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1246, in _execute_context
    cursor, statement, parameters, context
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 588, in do_execute
    cursor.execute(statement, parameters)
psycopg2.errors.UndefinedObject: language "plv8" does not exist
...

There's more to the trace here, but it's sounding like plv8 isn't installed.

I believe it's missing from the Postgres docker image, so I tried using this one in the docker-compose.yml file and it worked: https://registry.hub.docker.com/u/clkao/postgres-plv8/

Update: Got it working with the other docker image.

ERROR: Version in "./docker-compose.yml" is unsupported

My Docker and Docker Compose versions are the same ones given in the prerequisites.

(Incidentally, these are the versions installed by apt from the Ubuntu repository.)

$ lsb_release --description --short
Ubuntu 20.04.2 LTS
$ docker --version
Docker version 19.03.8, build afacb8b7f0
$ docker-compose --version
docker-compose version 1.25.0, build unknown

Yet when I follow the "getting started" steps I get an error about an unsupported version.

$ cd ~/tmp
$ curl -sSLO https://github.com/goldfiglabs/introspector/releases/latest/download/introspector_linux.zip
$ unzip -oq introspector_linux.zip 
$ docker-compose up -d
ERROR: Version in "./docker-compose.yml" is unsupported. You might be seeing this error because you're using the wrong Compose file version. Either specify a supported version (e.g "2.2" or "3.3") and place your service definitions under the `services` key, or omit the `version` key and place your service definitions at the root of the file to use version 1.
For more on the Compose file format versions, see https://docs.docker.com/compose/compose-file/

What versions do I really need to use?

Support AWS SSO

Using AWS IAM credentials, all is well:

$ AWS_PROFILE=saa-gen-iam ./introspector account aws import --help
Usage: introspector.py account aws import [OPTIONS]
[...]

Using AWS SSO I expect to get the same result.

Instead Introspector panics with a cryptic error because the AWS Go SDK can't get credentials.

$ AWS_PROFILE=saa-gen ./introspector account aws import --help
panic: EC2RoleRequestError: no EC2 instance role found
caused by: request canceled, context deadline exceeded

goroutine 1 [running]:
main.main()
	/Users/greg/workspace/introspector/launcher/main.go:208 +0x3d9

I believe the minimum version of the Go SDK with good support for AWS SSO is v1.2.0.

https://github.com/aws/aws-sdk-go-v2/releases/tag/v1.2.0

Can you upgrade Inspector to use at least that version?

Pass through return values from inside Docker to the Go wrapper

If a call to Introspector fails or the query returns a failure, the Go wrapper should return a non-zero return value to the calling shell.

🤖 17:08:30 anglegrinder:introspector$ ./introspector asdfasdf
Usage: introspector.py [OPTIONS] COMMAND [ARGS]...
Try 'introspector.py --help' for help.

Error: No such command 'asdfasdf'.

🤖 17:08:40 anglegrinder:introspector$ echo $?
0