Comments (6)
Going to reopen this until I finish documenting s3 usage in the developer docs.
from neurocaas.
Hi Shuonan,
The bucket itself will be specified in your "stack_config_template.json" file- the field "PipelineName" is exactly the name of the created s3 bucket the fields indicated here are paths within that bucket. You should be able to find the bucket named after your PipelineName in S3. Having run Step 6 of the developer guide, you should then be able to find directories within this bucket, with "inputs/", "configs/" and "submissions/" subdirectories, where your input data, configuration files, and submit.json files should go.
from neurocaas.
Thank you, step 6 run successfully with the following information:
Successfully packaged artifacts and wrote output template to file compiled_packaged.yaml.
Execute the following command to deploy the packaged template
sam deploy --template-file ../neurocaas/ncap_iac/ncap_blueprints/bardensr/compiled_packaged.yaml --stack-name <YOUR STACK NAME>
Deploying with following values
===============================
Stack name : bardensr
Region : None
Confirm changeset : False
Deployment s3 bucket : None
Capabilities : ["CAPABILITY_NAMED_IAM", "CAPABILITY_AUTO_EXPAND"]
Parameter overrides : {}
However, under bardensr
I wasn't able to find the 'inputs/' 'configs/' etc:
ls -a ./bardensr/
. .ipynb_checkpoints stack_config_template.json
.. compiled_packaged.yaml stack_config_template.json~
.DS_Store compiled_template.json test.ipynb
.aws-sam test_resources
am i missing something? thank you!
from neurocaas.
Hi Shuonan,
In order to list the contents of an S3 bucket, please use the aws cli as follows:
aws s3 ls [your bucket name] --recursive
You can likewise use this cli to upload/download files https://docs.aws.amazon.com/cli/latest/reference/s3/cp.html or folders https://docs.aws.amazon.com/cli/latest/reference/s3/sync.html. I will document this more clearly in the developer docs!
from neurocaas.
thank you, that was helpful, closing the issue
from neurocaas.
Closed in newest version of developer docs: "End Goal" section.
from neurocaas.
Related Issues (20)
- Bucket-to-bucket transfer does not work for users by default from cli
- Loss of access to available analysis HOT 1
- Security group error during initial blueprint setup HOT 2
- Remove session dependence of blueprints
- Expand dev permissions to allow log writes.
- Bucket Bypass extended use case
- Incorporate climate impact logging/adaptive resource allocation
- DANDI integration HOT 4
- setting up aws credentials HOT 6
- Error: Jobs do not launch without warning (label-job-create-web) HOT 1
- Developer workflow: write to logs HOT 2
- Race condition: stopping vs. terminating instances.
- Documentation: Clarify usage HOT 1
- "Select Datasets" when already selected.
- [Resolved] ValueError: key must be provided when HDF5 file contains multiple datasets HOT 1
- Developer docs installation command pip install ./ncap_iac HOT 1
- Pull Requests from outside contributors do not get the correct ref HOT 1
- Dev Mode timestamps don't show on website
- Outstanding Todos: HOT 4
- Conditional PR Deployment Dependencies
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from neurocaas.