awslabs / amazon-athena-cross-account-catalog Goto Github PK

Are there any plans to start using GitHub Releases or a public Package Index to publish this code? We're currently consuming it and using it, unmodified, so there isn't any need for it to be under version control in our system. The only reason that the source even needs to be available is to repackage for lambda, so a GitHub Release that includes the lambda-packaged code as an artifact would be ideal, because that's a public resource we could consume into our build pipeline, directly.

Thanks!

I'll prefix this by saying that it's entirely possible this is a problem of my own making. I've followed the guide to the letter (using the cloudformation yaml provided) and whilst I can now run Athena queries across accounts I'm not able to list the tables in each database. When I try I see the following errors logged in then cross account lambda

`Calling listTables({'dbName': '', 'filter': None, 'nextToken': None, 'maxSize': -1})

[ERROR] KeyError: 'SerializationLibrary'
Traceback (most recent call last):
File "/var/task/heracles/lambda.py", line 42, in handler
result = klas().execute(api_request)
File "/var/task/heracles/handlers/getters.py", line 107, in execute
hive_table = HiveMappers.map_glue_table(databaseName, table['Name'], table)
File "/var/task/heracles/handlers/hive_mappers.py", line 81, in map_glue_table
serializationLib=glue_table['StorageDescriptor']['SerdeInfo']['SerializationLibrary'],`

I'm continuing to dig, but any advice would be much appreciated.

Hi,

I've been trying to deploy this lambda, following all the steps and reading in detail instructions and even source code but I'm unable to deploy it and make it work.

What is happening is that apparently the event that is sent to the lambda by Athena, is not in the same structure as its supposed by your code, since it does not contain the apiRequest and apiName properties. I modified the main lambda handler to do a simple print on the event and this is what I get

2020-05-28T09:33:49.267+02:00 START RequestId: XXXXX-XXXX-XXXX-XXXX-XXXXX Version: $LATEST
2020-05-28T09:33:49.268+02:00 {'@type': 'PingRequest', 'identity': {'id': 'ASIAXXXXXXXX', 'principal': 'UNKNOWN_PRINCIPAL', 'account': '99999999'}, 'catalogName': 'man_dev', 'queryId': 'XXXXXX-XXXXX-XXXXX-XXXXXX-f6169d00924c'}
2020-05-28T09:33:49.268+02:00 Calling None(None)
2020-05-28T09:33:49.269+02:00 [ERROR] NotImplementedError Traceback (most recent call last):   File "/var/task/heracles/lambda.py", line 22, in handler     klas = HandlerBase.get_class_for_api_name(api_name)   File "/var/task/heracles/handlers/handlerbase.py", line 25, in get_class_for_api_name     raise NotImplementedError
2020-05-28T09:33:49.270+02:00 END RequestId: XXXXX-XXXX-XXXX-XXXX-XXXXX
2020-05-28T09:33:49.270+02:00 REPORT RequestId: XXXXX-XXXX-XXXX-XXXX-XXXXX Duration: 2.35 ms Billed Duration: 100 ms Memory Size: 256 MB Max Memory Used: 83 MB Init Duration: 459.50 ms 

I started to debug this and it seems that is the structure actually expected by the Athena query federation SDK since it has all those PingRequest, PingResponse, etc properties and events.

Can you please let me know what I'm doing wrong or any insights on how to continue debugging my problem? This components its very important for our infrastructure and will probably set a very nice workflow for our DataScientist. We are deploying this on eu-west-1