Comments (3)
laserson
commented on September 25, 2024
One eggo is refactored, there should be a pretty clear separation of code that creates/configs a cluster, vs code that executes the actual ETL steps. My thinking now is that the "interface" will be such that the ETL code assumes some small set of ENV VARS defined and that's all.
from eggo.
laserson
commented on September 25, 2024
Also, wrt an arbitrary cluster, we will have code that configures it to make sure all the necessary deps are installed and the env vars are set. This will require the interface that we've been using, such as getting a "master" node host etc.
from eggo.
laserson
commented on September 25, 2024
No longer supporting this.
from eggo.
Related Issues (20)
- 1000 Genomes Phase 3 VCF data set to be hosted in S3 in parquet format HOT 4
- ENCODE data set to be hosted in S3 in parquet format
- Reduce number of CM/CDH restarts during cluster config
- Platinum Genomes data set
- dnload_raw operation requires the ability to `sudo -u hdfs` HOT 2
- `dnload_raw` shouldn't hardcode `ec2-user` when it `chown`s data
- Extract Hive queries into separate reusable files
- asking for password HOT 4
- AMI in eu-west-1 ? HOT 6
- tried to substitute ami-00a11e68 w/o success HOT 9
- Default parallelism set to 2 on cluster with 4 executor nodes? HOT 2
- Verify that requested AMIs are consistent with the requested region/AZ HOT 1
- us-east-1b is hardcoded into cloudformation.template HOT 2
- eggo-cluster provision - fails with return code 12 while executing HOT 1
- Remove minCount line from aws.conf
- how to set the instance type for master, manager and workers separately ? HOT 1
- No handlers could be found for logger "cm_api.resource" HOT 2
- results of "genotype count" are different - depending on used programs HOT 2
- sql query on Adam's flattened file -> Non-local session path expected to be non-null; HOT 4
- Upgrade Spark from 1.3 to 1.5 to make use of project Tungsten performance improvements. HOT 6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from eggo.