scribd / objinsync Goto Github PK
View Code? Open in Web Editor NEWContinuously synchronize directories from remote object store to local filesystem
Home Page: https://tech.scribd.com/blog/2020/breaking-up-the-dag-repo.html
License: MIT License
Continuously synchronize directories from remote object store to local filesystem
Home Page: https://tech.scribd.com/blog/2020/breaking-up-the-dag-repo.html
License: MIT License
There are cases when it could be beneficial to sync multiple directories without deleting certain contents that already exist in the target location. For example, I have a lib/
directory with custom operators, hooks, etc that needs to be synced in addition to my dags/
folder and both are under the following location: s3://<bucket>/airflow/
. Currently, if I were to run the pull command: objinsync pull s3://<bucket>/airflow/ ${AIRFLOW_HOME}/
I would delete other important files such as my airflow.cfg
file. Therefore it could be nice to implement an --exclude
flag that would preserve files or directories in the target location so the pull command could be used to directories with other files present.
Final command might look something like this:
objinsync pull s3://<bucket>/airflow/ ${AIRFLOW_HOME}/ --exclude "airflow.cfg" --exclude "some_dir/*"
In the following scenario, object FOO
doesn't sync properly:
FOO
object is uploaded to the bucket -> objinsync
add themFOO
object is deleted from the bucket -> objinsync
remove themFOO
object is uploaded again to the bucket -> objinsync
doesn't add themobjinsync
after this step, it will sync properly.FOO
object is uploaded with a modified content -> objinsync
add themI think it's because of the uidCache
that is not purged when a file is deleted.
Maybe adding a delete
in this for
works:
for f, _ := range self.filesToDelete {
os.Remove(f)
uidKey, err := uidKeyFromLocalPath(self.LocalDir, f)
if err == nil {
self.uidLock.Lock()
delete(self.uidCache, uidKey)
self.uidLock.Unlock()
}
}
Disclaimer: I'm not a Go expert
hi
we trying to deploy the objinsync solution with airflow 2, using this chart
https://artifacthub.io/packages/helm/airflow-helm/airflow
but with no success
we added the env varibels accoring to https://tech.scribd.com/blog/2020/breaking-up-the-dag-repo.html
and even i changed the chart "git sync" part
## the git-sync container image ## image: repository: ghcr.io/scribd/objinsync #k8s.gcr.io/git-sync/git-sync tag: latest
repo: "s3://airflow-stg-s3-log-bucket/airflow_home/dags" #"[email protected]:lusha_team/airflow-dags.git" branch: "dummy"
can we have more instruction how to do it ?
any idea's?
thanks
Hi,
That tool is wonderfull! Thank you. Sorry to be the first to open an issue but I can't run two instances from systemd at the same time
/opt/objinsync pull --disable-ssl -i 120s s3://airflow-us-east-1/datawarehouse/profiles /opt/airflow/images/dbt/profiles
/opt/objinsync pull --disable-ssl -i 120s s3://airflow-us-east-1/datawarehouse/dags /opt/airflow/images/dbt/dags
{"level":"info","ts":1629897460.9748888,"caller":"objinsync/main.go:87","msg":"SENTRY_DSN not found, sentry integration disabled."}
{"level":"info","ts":1629897460.9751706,"caller":"objinsync/main.go:147","msg":"Serving health check endpoints at: :8087."}
{"level":"info","ts":1629897460.9752388,"caller":"objinsync/main.go:148","msg":"Pulling from s3://airflow-us-east-1.rifiniti.com/datawarehouse/profiles to /opt/airflow/images/dbt/profiles every 2m0s..."}
{"level":"info","ts":1629897460.9752793,"caller":"objinsync/main.go:126","msg":"Pull started."}
2021/08/25 13:17:40 listen tcp :8087: bind: address already in use
Can you help?
Hi! Thanks for this tool! It looks like its exactly what I need! I'm having some issues using it though:
Any file I pull is created as an empty (0 byte) file.
I have an s3 bucket called my-bucket
which holds two files: foo.txt
and image.png
.
When I run a pull command the files get created but they are both empty:
root@gosand:/tmp# ls -l /tmp
total 0
root@gosand:/tmp# objinsync pull --once s3://my-bucket/git /tmp
{"level":"info","ts":1595423591.3152442,"caller":"objinsync/main.go:86","msg":"SENTRY_DSN not found, sentry integration disabled."}
{"level":"info","ts":1595423591.3153822,"caller":"objinsync/main.go:141","msg":"Pulling from s3://my-bucket/git to /tmp..."}
{"level":"info","ts":1595423591.3154054,"caller":"objinsync/main.go:125","msg":"Pull started."}
{"level":"info","ts":1595423591.317076,"caller":"sync/pull.go:357","msg":"Listing objects","bucket":"my-bucket","dirpath":"git"}
{"level":"info","ts":1595423591.393758,"caller":"sync/pull.go:203","msg":"Object list page contains 3 objects."}
{"level":"info","ts":1595423591.423097,"caller":"objinsync/main.go:137","msg":"Pull finished in 107.686584ms seconds."}
root@gosand:/tmp# ls -l /tmp
total 0
-rw-r--r-- 1 root root 0 Jul 22 13:13 foo.txt
-rw-r--r-- 1 root root 0 Jul 22 13:13 image.png
objinsync
on a container based off golang:1.14
.go get
.{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::my-bucket",
"arn:aws:s3:::my-bucket/git"
]
}
]
}
Is there any other info I can provide to help debug this?
Hi Team,
I am trying to make this utility work with minio and having some issues. Any help or pointers to resolve this is highly appreciated.
I have installed the minio
via docker on the same server where my airflow
is installed. I am able to access minio
properly via the publicip:9001
. I then created a bucket by the name dags
and able to upload files.
Now I want to sync the dags
volume of the airflow container with the minio
. For test purposes I am trying to run it with the following arguments but always getting error. I am having hardtime uderstanding how to fix this.
objinsync pull --once --disable-ssl --s3-endpoint http://localhost:9000 s3://dags/*.* ~/Projects/airflow/dags
{"level":"info","ts":1657540886.0054412,"caller":"objinsync/main.go:87","msg":"SENTRY_DSN not found, sentry integration disabled."}
{"level":"info","ts":1657540886.009082,"caller":"objinsync/main.go:142","msg":"Pulling from s3://dags/*.* to /home/lyridadmin/Projects/airflow/dags..."}
{"level":"info","ts":1657540886.0091188,"caller":"objinsync/main.go:126","msg":"Pull started."}
ERROR: failed to pull objects from remote store: Failed to detect AWS region: EC2MetadataRequestError: failed to get EC2 instance identity document
caused by: RequestError: send request failed
caused by: Get "http://169.254.169.254/latest/dynamic/instance-identity/document": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Tried with this as well - but getting the same error as above.
objinsync pull --once --disable-ssl --s3-endpoint http://publicip:9000 s3://dags/*.* ~/Projects/airflow/dags
Edit:
Also, how do I put the details of the accessKey and SecretKey of the Minio?
Extend to support Minio
Hi,
This is more an enhancement request rather than an issue: it would be good for objinsync to support AWS IRSA. Lowest aws-sdk-go version that supports IRSA is 1.23.13. Currently aws-sdk-go version used in objinsync is v1.20.0.
Any significant breaking changes to upgrade aws-sdk to the latest version (v1.41.6)?
Iโm running into some issues where the objinsync causes imports not to be found intermittently. I believe the reason for this that the replace conflicts with the scheduler parsing loop sometimes thus causing import not found expectations.
Thanks for the tool, it's really useful!
It would be great to publish to docker Registry using the existent GitHub workflow, tagging the image with the specific release version v#.#.#
and latest
.
Is it possible to configure the interval between synchronizations with the S3 buckets?
From the code, I have seen that it is 5 seconds, but, since I have never used the go language, I don't know how it can be configured using CLI like for the --once
or --exclude
arguments.
I would like to be able to write:
objinsync pull --interval 60 s3://bucket/keyprefix ./localdir
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.