awslabs / amazon-athena-cross-account-catalog Goto Github PK
View Code? Open in Web Editor NEW๐ Reference implementation for granting cross-account AWS Glue Data Catalog access from Amazon Athena
License: Apache License 2.0
๐ Reference implementation for granting cross-account AWS Glue Data Catalog access from Amazon Athena
License: Apache License 2.0
Are there any plans to start using GitHub Releases or a public Package Index to publish this code? We're currently consuming it and using it, unmodified, so there isn't any need for it to be under version control in our system. The only reason that the source even needs to be available is to repackage for lambda, so a GitHub Release that includes the lambda-packaged code as an artifact would be ideal, because that's a public resource we could consume into our build pipeline, directly.
Thanks!
I'll prefix this by saying that it's entirely possible this is a problem of my own making. I've followed the guide to the letter (using the cloudformation yaml provided) and whilst I can now run Athena queries across accounts I'm not able to list the tables in each database. When I try I see the following errors logged in then cross account lambda
`Calling listTables({'dbName': '', 'filter': None, 'nextToken': None, 'maxSize': -1})
[ERROR] KeyError: 'SerializationLibrary'
Traceback (most recent call last):
File "/var/task/heracles/lambda.py", line 42, in handler
result = klas().execute(api_request)
File "/var/task/heracles/handlers/getters.py", line 107, in execute
hive_table = HiveMappers.map_glue_table(databaseName, table['Name'], table)
File "/var/task/heracles/handlers/hive_mappers.py", line 81, in map_glue_table
serializationLib=glue_table['StorageDescriptor']['SerdeInfo']['SerializationLibrary'],`
I'm continuing to dig, but any advice would be much appreciated.
Hi,
I've been trying to deploy this lambda, following all the steps and reading in detail instructions and even source code but I'm unable to deploy it and make it work.
What is happening is that apparently the event that is sent to the lambda by Athena, is not in the same structure as its supposed by your code, since it does not contain the apiRequest and apiName properties. I modified the main lambda handler to do a simple print on the event and this is what I get
2020-05-28T09:33:49.267+02:00 START RequestId: XXXXX-XXXX-XXXX-XXXX-XXXXX Version: $LATEST
2020-05-28T09:33:49.268+02:00 {'@type': 'PingRequest', 'identity': {'id': 'ASIAXXXXXXXX', 'principal': 'UNKNOWN_PRINCIPAL', 'account': '99999999'}, 'catalogName': 'man_dev', 'queryId': 'XXXXXX-XXXXX-XXXXX-XXXXXX-f6169d00924c'}
2020-05-28T09:33:49.268+02:00 Calling None(None)
2020-05-28T09:33:49.269+02:00 [ERROR] NotImplementedError Traceback (most recent call last): File "/var/task/heracles/lambda.py", line 22, in handler klas = HandlerBase.get_class_for_api_name(api_name) File "/var/task/heracles/handlers/handlerbase.py", line 25, in get_class_for_api_name raise NotImplementedError
2020-05-28T09:33:49.270+02:00 END RequestId: XXXXX-XXXX-XXXX-XXXX-XXXXX
2020-05-28T09:33:49.270+02:00 REPORT RequestId: XXXXX-XXXX-XXXX-XXXX-XXXXX Duration: 2.35 ms Billed Duration: 100 ms Memory Size: 256 MB Max Memory Used: 83 MB Init Duration: 459.50 ms
I started to debug this and it seems that is the structure actually expected by the Athena query federation SDK since it has all those PingRequest, PingResponse, etc properties and events.
Can you please let me know what I'm doing wrong or any insights on how to continue debugging my problem? This components its very important for our infrastructure and will probably set a very nice workflow for our DataScientist. We are deploying this on eu-west-1
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.