Comments (9)
I've thought about this a lot and I don't think timestamp is a consistent and/or reliable enough mechanism to rely on for comparison.
- The time on the server may not be accurate
- When a new file is uploaded to the server, the timestamp on the local files would need to be updated to reflect that both the local/remote files are in sync. This creates additional and unnecessary work on the client.
- In a one-to-many scenario where multiple clients are pushing to a single server, when one client makes an update it will invalidate the timestamps on every other client instance.
- The time-sync itself is hacky. Basically, it would involve uploading a temporary file, checking the timestamp on the upload file using a MDTM request (which isn't supported on all FTP servers), and calculating the difference after normalizing timezone differences.
Note: The difficulties are inherent to the FTP protocol in general.
If we were doing the sync over SSH, it would be possible to calculate the MD5 for both the local and remote to compare. Unfortunately, granting full SSH access opens a whole other can of worms in terms of security.
The only reasonable solution I can think of is to generate a digest listing that maps files to their calculated MD5 hash. Such an approach would be non-trivial to implement.
The gist is, syncing files should be safe 99.99% of the time using size comparison alone. Unless you manage to make changes to a file in a way that doesn't change the size of the file, this shouldn't be an issue.
from node-ftpsync.
Ok yeah, that makes sense. Problem I'm having though, is that for my website I'm appending a hash to assets, to invalidate the cache (so for example `styles-139shw.css). This hash doesn't change in length, so any html referencing an asset that has changed won't change in file size, while the hash it's referencing did change. Which means that a change won't be detected by ftpsync.
I'm solving this manually by just removing all .html before an upload, but it feels like there should be a better, less manual solution. Maybe even something as simple as a syncAll
boolean, which would update even files of a similar filesize?
from node-ftpsync.
Oh, damn... That's probably a common use case too.
As a temporary stopgap you could look into using:
angular/angular#5123 (comment)
grunt-ftp-push always pushes all files. It's not ideal in terms of bandwidth/time but it accomplishes what you described.
I haven't looked at the codebase for this project in a while but it shouldn't be too difficult to add a syncAll
feature.
I'd like to try adding a file manifest at some point in the near future. It'll require a pretty major refactor of the code though.
from node-ftpsync.
Oh, damn... That's probably a common use case too.
Yeah, it probably is. I'm not sure what you meant with the stopgap, I'm not using angular, don't understand how I'd apply that to my use case.
Also, I'm using metalsmith and regular node scripts, so I'd like to stay with that, instead of introducing another buildtool (with grunt-ftpush). But there are probably also a lot of regular node libraries that'll do what grunt ftpush does. Thanks for the suggestion anyway!
And I was thinking, maybe instead of a syncAll it would be better to have a force
option, which accepts an array of globs. That way you can pass it something like: force: ['**.html']
, which would then overwrite all html regardless of filesize. That would be a way to fix the bug, without resorting to something drastic like pushing all files all the time.
from node-ftpsync.
Sorry, pasted the wrong link. Too much multi-tasking. The utility I meant was grunt-ftp-push:
https://www.npmjs.com/package/grunt-ftp-push
I understand the desire to get away from using grunt. I'm gradually replacing grunt with npm scripts in my projects. Unfortunately, I'm not aware of a ftp-push module that runs standalone.
That's an interesting idea. I'm not sure how to implement it. Maybe first to the match based on differences, then pass the file listing through a second filter that applies force
. The code really could use a refactor to make it act in a more functional manner.
from node-ftpsync.
Sorry, pasted the wrong link. Too much multi-tasking. The utility I meant was grunt-ftp-push:
https://www.npmjs.com/package/grunt-ftp-push
:) Yeah, that's what I suspected.
Maybe first to the match based on differences, then pass the file listing through a second filter that applies force.
I don't know what kind of mechanic would be optimal for ftp, but that definitely sounds sensible.
from node-ftpsync.
FTP is just the last step in the processing chain. Believe it or not, that's the easy part.
The complicated part filtering through the local vs remote file listing to generate a complete collection of files that need to be acted upon.
If you look at the source, the processing chain is pretty straight-forward:
- collect remote and local file listings
- diff the two to generate 3 lists (additions, updates, and removals)
- fire off FTP requests that act on the lists of changes.
from node-ftpsync.
Are there any updates on this?
I just updated my build process to add a hash to assets and now I'm facing the same problem.
from node-ftpsync.
PR #39 has landed with the functionality to do timestamp comparisons. Give it a test drive (preferably on non-production code first) and post your results.
from node-ftpsync.
Related Issues (20)
- some problem when using the lib HOT 1
- I forked your node-fptsync, and start my own branch HOT 2
- Using the client as the slave with a readonly FTP connection HOT 1
- 550 : Cannot create a file when that file already exists HOT 9
- Migrate documentation from Google Code
- Break up the documentation into separate files
- Add CONTRIBUTING.md
- 503 Bad Sequence of Commands HOT 13
- Use 'ftp-test-server' for testing
- ftpsync stops uploading after x files HOT 1
- Pass path to config.json as argument HOT 2
- TypeError: Cannot read property 'on' of undefined HOT 3
- Error 550 Can't create directory: No such file or directory HOT 8
- Ignoring remote folders ?
- Not syncing important files because filesize is the same HOT 2
- Cannot sync all files of a folder HOT 1
- DeprecationWarning: Ftp.raw[user](args): Use Ftp.raw('user args') instead HOT 3
- (node:9316) DeprecationWarning: Ftp.raw[stat](args): Use Ftp.raw('stat args') instead. HOT 3
- TypeError: ftp.raw.mkd is not a function HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from node-ftpsync.