goldfiglabs / introspector Goto Github PK
View Code? Open in Web Editor NEWA schema and set of tools for using SQL to query cloud infrastructure.
Home Page: https://www.goldfiglabs.com/
License: Mozilla Public License 2.0
A schema and set of tools for using SQL to query cloud infrastructure.
Home Page: https://www.goldfiglabs.com/
License: Mozilla Public License 2.0
The tool looks interesting, but assumes that there's an AWS organization in place (and that it's accessible to the user). Is there a plan to remove that requirement? https://github.com/goldfiglabs/goldfig/blob/main/goldfig/aws/__init__.py#L92
Thanks for publishing your work.
Currently, some migrations and db requirements (like plv8
) are only used for some of the tools, like cis foundation
. If these migrations move into their own migration directory and only run when using those tools, other users who don't need plv8
can depend on a much smaller postgres image.
IAM limits the length of a managed policy document after removing whitespace to 6144 characters.
Uncontrolled growth in of a policy document will eventually lead to an error when the length exceeds the limit. This will impair or block anything depending on the policy to provide it permissions.
I've tried querying the instrospector database like this to find such long documents.
SELECT document::text FROM aws_iam_policyversion;
Unfortunately the jsonb type when cast to text puts spaces after the commas and the colons.
I don't see any other way in SQL to get the compact string representation (like jq -c
).
I have a custom Python script for doing it which uses a function like this:
def get_default_document_for_policy(policy):
resp = boto3.client("iam").get_policy_version(
PolicyArn=policy["Arn"], VersionId=policy["DefaultVersionId"]
)
return json.dumps(resp["PolicyVersion"]["Document"], separators=(",", ":"))
How would I adapt this to Introspector?
Just a small usability feature to make introspector easier to learn.
I don't have a default AWS profile in my environment.
When I set an AWS profile for the command the help message does appear.
$ AWS_PROFILE=introspector ./introspector account aws import --help
Usage: introspector.py account aws import [OPTIONS]
Imports the assets from an AWS account into introspector
Options:
-f, --force Skip prompts for adding accounts
-s, --service TEXT Only import the specified service(s)
-g, --gov-cloud Set this flag to import a govcloud account
--help Show this message and exit.
I would expect the help message to appear without an AWS profile set, but instead it prints an error.
$ ./introspector account aws import --help
panic: no EC2 IMDS role found, operation error ec2imds: GetMetadata, request canceled, context deadline exceeded
goroutine 1 [running]:
main.main()
/Users/greg/workspace/introspector/launcher/main.go:208 +0x3d9
Wrapped results via the Introspector CLI aren't rendering properly:
Traceback (most recent call last):
File "/app/introspector.py", line 5, in <module>
run_cli()
File "/app/introspector/cli/__init__.py", line 60, in run_cli
cli()
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/app/introspector/cli/query.py", line 34, in cmd
print(tabulate(results.rows, headers=results.columns))
File "/usr/local/lib/python3.9/site-packages/tabulate.py", line 1528, in tabulate
list_of_lists, headers = _normalize_tabular_data(
File "/usr/local/lib/python3.9/site-packages/tabulate.py", line 1161, in _normalize_tabular_data
raise ValueError(
ValueError: headers for a list of dicts is not a dict or a keyword
๐ค 15:42:34 anglegrinder:tmp$
Introspector only supports master accounts. Add support for sub-accounts.
It looks like the recommended permissions are lacking acm-pca:ListCertificateAuthorities.
I don't see it listed in SecurityAudit, ViewOnlyAccess or Introspector-Ro-Additions.
Introspector is failing for me because of that.
$ AWS_PROFILE=introspector ./introspector account aws import
Add AWS account 01234567890 using identity arn:aws:iam::01234567890:user/introspector? [Y/n] Y
[2021-03-19 19:09:19,636] {__init__.py:88} ERROR - exception caught in import
concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
File "/usr/local/lib/python3.9/concurrent/futures/process.py", line 243, in _process_worker
r = call_item.fn(*call_item.args, **call_item.kwargs)
File "/app/introspector/aws/svc.py", line 215, in _async_proxy
for resource_name, raw_resources in import_fn(service_proxy, region,
File "/app/introspector/aws/acm_pca.py", line 34, in _import_acm_pca_region
yield from _import_certificate_authorities(proxy)
File "/app/introspector/aws/acm_pca.py", line 23, in _import_certificate_authorities
certificate_authorities_resp = proxy.list('list_certificate_authorities')
File "/app/introspector/aws/fetch.py", line 396, in list
return self._impl.list(resource, kwargs)
File "/app/introspector/aws/fetch.py", line 100, in list
raise e
File "/app/introspector/aws/fetch.py", line 58, in list
result = iterator.build_full_result()
File "/usr/local/lib/python3.9/site-packages/botocore/paginate.py", line 449, in build_full_result
for response in self:
File "/usr/local/lib/python3.9/site-packages/botocore/paginate.py", line 255, in __iter__
response = self._make_request(current_kwargs)
File "/usr/local/lib/python3.9/site-packages/botocore/paginate.py", line 332, in _make_request
return self._method(**current_kwargs)
File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 357, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 676, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (AccessDeniedException) when calling the ListCertificateAuthorities operation: User: arn:aws:iam::01234567890:user/introspector is not authorized to perform: acm-pca:ListCertificateAuthorities
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/app/introspector/__init__.py", line 85, in collect_exceptions
_ = result.result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
return self.__get_result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
botocore.exceptions.ClientError: An error occurred (AccessDeniedException) when calling the ListCertificateAuthorities operation: User: arn:aws:iam::01234567890:user/introspector is not authorized to perform: acm-pca:ListCertificateAuthorities
The above errors repeat several times and finally it prints results.
Results - Import #1
Start: 2021-03-19 19:09:07
End: 2021-03-19 19:09:51
Errors: 16
Resources added: 0
Resources removed: 0
Resources updated: 0
Attributes added: 0
Attributes removed: 0
Attributes updated: 0
In my first run of Inspector (#11) I saw this warning about billing alarm.
[2021-03-20 14:23:25,481] {resource.py:65} WARNING - Missing target for relation arn:aws:cloudwatch:us-east-1:01234567890:alarm:BillingAlert fires-on [metrics/us-east-1/None/None/] (1)
It turns out that it's not actually a problem with the alarm configuration but a limitation of introspector when deriving relations from the resources.
@gsoltis commented:
For some of the calculated relations, as you can see from the None token, we don't know the correct thing to relate them to (or it might not exist). These shouldn't actually be warnings at this point. If we can't construct the URI for a relation, we should not then be logging that we can't find the target...
(Please correct me if I have mixed this one up. It seems to be about an implementation detail that I don't understand yet.)
Version 4ec6c21 (commit ID) of the Introspector-Ro-Additions policy lacks the acm-pca:ListCertificateAuthorities permission.
Version e934e4551c83 (image ID) of the Introspector Docker image silently ignores permission errors.
So when the Introspector application attempts to list the certificate authorities using the introspector IAM user, it fails to list the private certificate authorities, but does not indicate that anything is missing from the inventory.
It would be nice if this would generate at least a warning so that I know that something might be missing from the database.
The code does currently just swallow the error in the case of missing permissions, I agree that a warning would be a good idea. I think eventually we will want to record what was skipped as part of the import.
Currently the docker-compose.yml
that gets packaged in a release points to goldfig/introspector:latest
. This is brittle because we will continue to update the image.
We should instead pin github releases to a particular version of the docker image.
I got an error about the IAM credential report when running an import today.
botocore.errorfactory.CredentialReportNotReadyException: An error occurred (ReportInProgress) when calling the GetCredentialReport operation: Unknown
I understand that GenerateCredentialReport and GetCredentialReport are distinct APIs and that Introspector will have to figure out somehow when the credential report is ready.
If it fails like this, does it mean that Introspector didn't get a credential report in the end?
Is my import incomplete?
$ ./introspector account aws import
[2021-03-24 19:53:38,385] {iam.py:93} ERROR - credenetial report fetch error
Traceback (most recent call last):
File "/app/introspector/aws/iam.py", line 91, in _import_credential_report
report = proxy.get('get_credential_report')
File "/app/introspector/aws/fetch.py", line 404, in get
return self._impl.get(resource, kwargs)
File "/app/introspector/aws/fetch.py", line 131, in get
raise e
File "/app/introspector/aws/fetch.py", line 106, in get
full_result = api_call(**kwargs)
File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 357, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 676, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.errorfactory.CredentialReportNotReadyException: An error occurred (ReportInProgress) when calling the GetCredentialReport operation: Unknown
[2021-03-24 19:55:34,548] {resource.py:65} WARNING - Missing target for relation arn:aws:cloudwatch:us-east-1:480783779961:alarm:BillingAlert fires-on [metrics/us-east-1/None/None/] (1)
Results - Import #3
Start: 2021-03-24 19:52:41
End: 2021-03-24 19:55:37
Errors: 0
Resources added: 1
Resources removed: 0
Resources updated: 10
Attributes added: 0
Attributes removed: 0
Attributes updated: 14
I'm using Docker image e934e4551c83 and release v0.1.2.
I'm trying to import a second AWS account to my introspector database.
The first account is the organization management account. The second acount is a member of that organization.
I created the introspector IAM user in the member acount and set its credentials via the AWS_PROFILE environment variable.
I got the following error:
[2021-03-24 21:11:47,038] {aws.py:106} ERROR - exception caught in map
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1245, in _execute_context
self.dialect.do_execute(
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 588, in do_execute
cursor.execute(statement, parameters)
psycopg2.errors.CardinalityViolation: ON CONFLICT DO UPDATE command cannot affect row a second time
HINT: Ensure that no rows proposed for insertion within the same command have duplicate constrained values.
Error in context:
$ ./introspector account aws import
[...truncate IAM credential report error...]
[2021-03-24 21:11:40,363] {resource.py:57} WARNING - Missing parent for relation arn:aws:ec2:us-west-2:012345678901:security-group/sg-00000000 in arn:aws:organizations::012345678901:account/o-abcdefghij/098765432109 (1)
[...truncate 100+ warnings about "Missing parent for relation"...]
[2021-03-24 21:11:44,653] {resource.py:57} WARNING - Missing parent for relation arn:aws:ec2:us-west-2:012345678901:vpc/vpc-00000000 in arn:aws:organizations::012345678901:account/o-abcdefghij/098765432109 (1)
[2021-03-24 21:11:47,038] {aws.py:106} ERROR - exception caught in map
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1245, in _execute_context
self.dialect.do_execute(
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 588, in do_execute
cursor.execute(statement, parameters)
psycopg2.errors.CardinalityViolation: ON CONFLICT DO UPDATE command cannot affect row a second time
HINT: Ensure that no rows proposed for insertion within the same command have duplicate constrained values.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/app/introspector/cli/account/aws.py", line 103, in import_aws_cmd
refresh_views(db, reloaded_import_job.provider_account_id)
File "/app/introspector/bootstrap_db.py", line 157, in refresh_views
result = db.execute(query)
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 1277, in execute
return self._connection_for_bind(bind, close_with_result=True).execute(
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 982, in execute
return meth(self, multiparams, params)
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/sql/elements.py", line 293, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1095, in _execute_clauseelement
ret = self._execute_context(
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1249, in _execute_context
self._handle_dbapi_exception(
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1476, in _handle_dbapi_exception
util.raise_from_cause(sqlalchemy_exception, exc_info)
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 398, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 152, in reraise
raise value.with_traceback(tb)
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1245, in _execute_context
self.dialect.do_execute(
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 588, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (psycopg2.errors.CardinalityViolation) ON CONFLICT DO UPDATE command cannot affect row a second time
HINT: Ensure that no rows proposed for insertion within the same command have duplicate constrained values.
[SQL: INSERT INTO aws_config_configurationrecorder (
_id,
uri,
provider_account_id,
rolearn,
allsupported,
includeglobalresourcetypes,
resourcetypes,
name,
laststarttime,
laststoptime,
recording,
laststatus,
lasterrorcode,
lasterrormessage,
laststatuschangetime,
_iam_role_id,_account_id
)
SELECT
R.id AS _id,
R.uri,
R.provider_account_id,
rolearn.attr_value #>> '{}' AS rolearn,
(allsupported.attr_value #>> '{}')::boolean AS allsupported,
(includeglobalresourcetypes.attr_value #>> '{}')::boolean AS includeglobalresourcetypes,
resourcetypes.attr_value::jsonb AS resourcetypes,
name.attr_value #>> '{}' AS name,
(TO_TIMESTAMP(laststarttime.attr_value #>> '{}', 'YYYY-MM-DD"T"HH24:MI:SS')::timestamp at time zone '00:00') AS laststarttime,
(TO_TIMESTAMP(laststoptime.attr_value #>> '{}', 'YYYY-MM-DD"T"HH24:MI:SS')::timestamp at time zone '00:00') AS laststoptime,
(recording.attr_value #>> '{}')::boolean AS recording,
laststatus.attr_value #>> '{}' AS laststatus,
lasterrorcode.attr_value #>> '{}' AS lasterrorcode,
lasterrormessage.attr_value #>> '{}' AS lasterrormessage,
(TO_TIMESTAMP(laststatuschangetime.attr_value #>> '{}', 'YYYY-MM-DD"T"HH24:MI:SS')::timestamp at time zone '00:00') AS laststatuschangetime,
_iam_role_id.target_id AS _iam_role_id,
_account_id.target_id AS _account_id
FROM
resource AS R
INNER JOIN provider_account AS PA
ON PA.id = R.provider_account_id
LEFT JOIN resource_attribute AS rolearn
ON rolearn.resource_id = R.id
AND rolearn.type = 'provider'
AND lower(rolearn.attr_name) = 'rolearn'
LEFT JOIN resource_attribute AS allsupported
ON allsupported.resource_id = R.id
AND allsupported.type = 'provider'
AND lower(allsupported.attr_name) = 'allsupported'
LEFT JOIN resource_attribute AS includeglobalresourcetypes
ON includeglobalresourcetypes.resource_id = R.id
AND includeglobalresourcetypes.type = 'provider'
AND lower(includeglobalresourcetypes.attr_name) = 'includeglobalresourcetypes'
LEFT JOIN resource_attribute AS resourcetypes
ON resourcetypes.resource_id = R.id
AND resourcetypes.type = 'provider'
AND lower(resourcetypes.attr_name) = 'resourcetypes'
LEFT JOIN resource_attribute AS name
ON name.resource_id = R.id
AND name.type = 'provider'
AND lower(name.attr_name) = 'name'
LEFT JOIN resource_attribute AS laststarttime
ON laststarttime.resource_id = R.id
AND laststarttime.type = 'provider'
AND lower(laststarttime.attr_name) = 'laststarttime'
LEFT JOIN resource_attribute AS laststoptime
ON laststoptime.resource_id = R.id
AND laststoptime.type = 'provider'
AND lower(laststoptime.attr_name) = 'laststoptime'
LEFT JOIN resource_attribute AS recording
ON recording.resource_id = R.id
AND recording.type = 'provider'
AND lower(recording.attr_name) = 'recording'
LEFT JOIN resource_attribute AS laststatus
ON laststatus.resource_id = R.id
AND laststatus.type = 'provider'
AND lower(laststatus.attr_name) = 'laststatus'
LEFT JOIN resource_attribute AS lasterrorcode
ON lasterrorcode.resource_id = R.id
AND lasterrorcode.type = 'provider'
AND lower(lasterrorcode.attr_name) = 'lasterrorcode'
LEFT JOIN resource_attribute AS lasterrormessage
ON lasterrormessage.resource_id = R.id
AND lasterrormessage.type = 'provider'
AND lower(lasterrormessage.attr_name) = 'lasterrormessage'
LEFT JOIN resource_attribute AS laststatuschangetime
ON laststatuschangetime.resource_id = R.id
AND laststatuschangetime.type = 'provider'
AND lower(laststatuschangetime.attr_name) = 'laststatuschangetime'
LEFT JOIN (
SELECT
_aws_iam_role_relation.resource_id AS resource_id,
_aws_iam_role.id AS target_id
FROM
resource_relation AS _aws_iam_role_relation
INNER JOIN resource AS _aws_iam_role
ON _aws_iam_role_relation.target_id = _aws_iam_role.id
AND _aws_iam_role.provider_type = 'Role'
AND _aws_iam_role.service = 'iam'
WHERE
_aws_iam_role_relation.relation = 'acts-as'
) AS _iam_role_id ON _iam_role_id.resource_id = R.id
LEFT JOIN (
SELECT
_aws_organizations_account_relation.resource_id AS resource_id,
_aws_organizations_account.id AS target_id
FROM
(
SELECT
_aws_organizations_account_relation.resource_id AS resource_id
FROM
resource_relation AS _aws_organizations_account_relation
INNER JOIN resource AS _aws_organizations_account
ON _aws_organizations_account_relation.target_id = _aws_organizations_account.id
AND _aws_organizations_account.provider_type = 'Account'
AND _aws_organizations_account.service = 'organizations'
WHERE
_aws_organizations_account_relation.relation = 'in'
GROUP BY _aws_organizations_account_relation.resource_id
HAVING COUNT(*) = 1
) AS unique_account_mapping
INNER JOIN resource_relation AS _aws_organizations_account_relation
ON _aws_organizations_account_relation.resource_id = unique_account_mapping.resource_id
INNER JOIN resource AS _aws_organizations_account
ON _aws_organizations_account_relation.target_id = _aws_organizations_account.id
AND _aws_organizations_account.provider_type = 'Account'
AND _aws_organizations_account.service = 'organizations'
WHERE
_aws_organizations_account_relation.relation = 'in'
) AS _account_id ON _account_id.resource_id = R.id
WHERE
R.provider_account_id = 1 AND
PA.provider = 'aws'
AND R.provider_type = 'ConfigurationRecorder'
AND R.service = 'config'
ON CONFLICT (_id) DO UPDATE
SET
rolearn = EXCLUDED.rolearn,
allsupported = EXCLUDED.allsupported,
includeglobalresourcetypes = EXCLUDED.includeglobalresourcetypes,
resourcetypes = EXCLUDED.resourcetypes,
name = EXCLUDED.name,
laststarttime = EXCLUDED.laststarttime,
laststoptime = EXCLUDED.laststoptime,
recording = EXCLUDED.recording,
laststatus = EXCLUDED.laststatus,
lasterrorcode = EXCLUDED.lasterrorcode,
lasterrormessage = EXCLUDED.lasterrormessage,
laststatuschangetime = EXCLUDED.laststatuschangetime,
_iam_role_id = EXCLUDED._iam_role_id,
_account_id = EXCLUDED._account_id]
(Background on this error at: http://sqlalche.me/e/f405)
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1245, in _execute_context
self.dialect.do_execute(
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 588, in do_execute
cursor.execute(statement, parameters)
psycopg2.errors.InFailedSqlTransaction: current transaction is aborted, commands ignored until end of transaction block
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/app/introspector.py", line 5, in <module>
run_cli()
File "/app/introspector/cli/__init__.py", line 60, in run_cli
cli()
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/app/introspector/cli/account/aws.py", line 112, in import_aws_cmd
db.commit()
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 1036, in commit
self.transaction.commit()
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 503, in commit
self._prepare_impl()
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 482, in _prepare_impl
self.session.flush()
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 2479, in flush
self._flush(objects)
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 2617, in _flush
transaction.rollback(_capture_exception=True)
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
compat.reraise(exc_type, exc_value, exc_tb)
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 153, in reraise
raise value
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 2577, in _flush
flush_context.execute()
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 422, in execute
rec.execute(self)
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 586, in execute
persistence.save_obj(
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 230, in save_obj
_emit_update_statements(
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 995, in _emit_update_statements
c = cached_connections[connection].execute(
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 982, in execute
return meth(self, multiparams, params)
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/sql/elements.py", line 293, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1095, in _execute_clauseelement
ret = self._execute_context(
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1249, in _execute_context
self._handle_dbapi_exception(
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1476, in _handle_dbapi_exception
util.raise_from_cause(sqlalchemy_exception, exc_info)
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 398, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 152, in reraise
raise value.with_traceback(tb)
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1245, in _execute_context
self.dialect.do_execute(
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 588, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.InternalError: (psycopg2.errors.InFailedSqlTransaction) current transaction is aborted, commands ignored until end of transaction block
[SQL: UPDATE import_job SET end_date=%(end_date)s, error_details=%(error_details)s WHERE import_job.id = %(import_job_id)s]
[parameters: {'end_date': datetime.datetime(2021, 3, 24, 21, 11, 47, 41410, tzinfo=datetime.timezone.utc), 'error_details': '["Traceback (most recent call last):\\n File \\"/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py\\", line 1245, in _execute_context\ ... (9259 characters truncated) ... \n _iam_role_id = EXCLUDED._iam_role_id,\\n _account_id = EXCLUDED._account_id]\\n(Background on this error at: http://sqlalche.me/e/f405)\\n"]', 'import_job_id': 5}]
(Background on this error at: http://sqlalche.me/e/2j85)
I'm using Docker image e934e4551c83 and release v0.1.2.
See misunderstanding in #18 for context.
I said:
It looks like the problem is in the aws_config_configurationrecorder table.
[SQL: INSERT INTO aws_config_configurationrecorder (
I tried to rule that out by setting the service flag to import just the ec2 service.
./introspector account aws import --service ec2
But I got the same error result.
@gsoltis said:
After running an import, even if it's just a single service (the --service flag definitely needs documentation...), introspector will resync the aws_ tables from the resource tables.
Hey Goldfig team, I'm trying out this repo for a tutorial I'm writing about maintaining AWS tags.
So far, I've got my containers running, but I run into the following error when I run init
:
$ ./gf init
Checking for running goldfig
[2020-08-30 14:48:57,050] {bootstrap_db.py:85} ERROR - failed adding function from aws_logs_metricfilter_match.sql
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1246, in _execute_context
cursor, statement, parameters, context
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 588, in do_execute
cursor.execute(statement, parameters)
psycopg2.errors.UndefinedObject: language "plv8" does not exist
...
There's more to the trace here, but it's sounding like plv8
isn't installed.
I believe it's missing from the Postgres docker image, so I tried using this one in the docker-compose.yml
file and it worked: https://registry.hub.docker.com/u/clkao/postgres-plv8/
Update: Got it working with the other docker image.
My Docker and Docker Compose versions are the same ones given in the prerequisites.
(Incidentally, these are the versions installed by apt from the Ubuntu repository.)
$ lsb_release --description --short
Ubuntu 20.04.2 LTS
$ docker --version
Docker version 19.03.8, build afacb8b7f0
$ docker-compose --version
docker-compose version 1.25.0, build unknown
Yet when I follow the "getting started" steps I get an error about an unsupported version.
$ cd ~/tmp
$ curl -sSLO https://github.com/goldfiglabs/introspector/releases/latest/download/introspector_linux.zip
$ unzip -oq introspector_linux.zip
$ docker-compose up -d
ERROR: Version in "./docker-compose.yml" is unsupported. You might be seeing this error because you're using the wrong Compose file version. Either specify a supported version (e.g "2.2" or "3.3") and place your service definitions under the `services` key, or omit the `version` key and place your service definitions at the root of the file to use version 1.
For more on the Compose file format versions, see https://docs.docker.com/compose/compose-file/
What versions do I really need to use?
Using AWS IAM credentials, all is well:
$ AWS_PROFILE=saa-gen-iam ./introspector account aws import --help
Usage: introspector.py account aws import [OPTIONS]
[...]
Using AWS SSO I expect to get the same result.
Instead Introspector panics with a cryptic error because the AWS Go SDK can't get credentials.
$ AWS_PROFILE=saa-gen ./introspector account aws import --help
panic: EC2RoleRequestError: no EC2 instance role found
caused by: request canceled, context deadline exceeded
goroutine 1 [running]:
main.main()
/Users/greg/workspace/introspector/launcher/main.go:208 +0x3d9
I believe the minimum version of the Go SDK with good support for AWS SSO is v1.2.0.
https://github.com/aws/aws-sdk-go-v2/releases/tag/v1.2.0
Can you upgrade Inspector to use at least that version?
If a call to Introspector fails or the query returns a failure, the Go wrapper should return a non-zero return value to the calling shell.
๐ค 17:08:30 anglegrinder:introspector$ ./introspector asdfasdf
Usage: introspector.py [OPTIONS] COMMAND [ARGS]...
Try 'introspector.py --help' for help.
Error: No such command 'asdfasdf'.
๐ค 17:08:40 anglegrinder:introspector$ echo $?
0
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.