Code Monkey home page Code Monkey logo

yajsync's Introduction

yajsync

yajsync is a port of rsync written in Java.

yajsync currently supports a minimal subset of rsync protocol version 30.0.

Currently implemented rsync options:

  • Incremental recursion (-r, --recursive)

  • Preserve owner (-o, --owner)

  • Preserve group (-g, --group)

  • Don't map uid/gid values by user/group name (--numeric-ids)

  • Preserve permissions (-p, --perms)

  • Preserve times (-t, --times)

  • Preserve symbolic links (-l, --links)

  • Transfer directories (-d, --dirs)

  • Delete extraneous files (--delete)

  • Don't skip files that match size and time (-I, --ignore-times)

  • Read daemon-access password from FILE (--password-file=FILE) or environment variable RSYNC_PASSWORD

  • Module file listings

  • Archive mode (-a, --archive)

  • Set I/O read timeout in seconds (--timeout=SECONDS)

  • Set daemon connection timeout in seconds (--contimeout=SECONDS)

Simulated options:

  • Preserve character device files and block device files (--devices)

  • Preserve named sockets and named pipes (--specials)

These will currently return an error when trying to actually read device metadata of a device file or trying to create a device file. The reason for this is the inability to handle device files in Java. We still want to support these options in order to be able to support --archive.

yajsync is compliant with at least rsync version 3.0.9.

Features

  • rsync Java API

  • Platform independent rsync server

  • Platform independent rsync client with support for both local and remote file transfers

  • Native SSL/TLS tunneling

Please be aware though that the API currently is unstable, not documented and will most probably change in the near future.

Warning

This software is still unstable and there might be data corruption bugs hiding. So use it only carefully at your own risk.

Contact

If you encounter any problems or if you have any questions or just want to provide feedback of any type, then please create a new github issue for this.

Example

Start a Server listening on localhost port 14415, with one implicitly read-only module called Downloads and one readable and writable module called Uploads:

$ cat yajsyncd.conf

# This line and all text after a `#' is a comment. Text within square
# brackets define the name of a new module. A module definition may be
# followed by any number of predefined parameter value statements on
# the form key = value. The current available module parameters are:
#
#    path          An existing path to the module (mandatory).
#    comment       Text that will be shown to the client in module listings
#                  (optional).
#    is_readable   A boolean (true or false) indicating whether files
#                  may be read below this module (optional, default is
#                  true).
#    is_writable   A boolean (true or false) indicating whether files
#                  may be written below this module (optional, default
#                  is false).
#    fs            A Java file system provider (optional).

# This is a module definition for a module called Downloads. path is
# the only mandatory module parameter. This one also provides a
# comment. All modules are implicitly readable but not writable:
[Downloads]
path = /path/to/Downloads/
comment = this text will be printed on module listings, it is optional
# is_readable = true
# is_writable = false

# Uploads is both readable and writable; it does not provide a
# comment:
[Uploads]
path = /path/to/Uploads/
is_writable = true

# Any non-default Java file system provider may be specified using the
# parameter `fs'. Here is an example using the built in Zip file
# system provider which provides transparent access to the contents of
# a zip-file (see also the client option `--fs'):
[zipfs]
fs = jar:file:/path/to/file.zip
path = /

Start the server:

$ java -Dumask=$(umask) -jar yajsync-app/target/yajsync-app-0.9.0-SNAPSHOT-full.jar server --port=14415 --config=yajsyncd.conf

Recursively upload the directory called example to Uploads:

java -Dumask=$(umask) -jar yajsync-app/target/yajsync-app-0.9.0-SNAPSHOT-full.jar client --port=14415 -r example localhost::Uploads

The same thing using the alternative syntax:

java -Dumask=$(umask) -jar yajsync-app/target/yajsync-app-0.9.0-SNAPSHOT-full.jar client -r example rsync://localhost:14415/Uploads

And finally the same thing using the original rsync client:

rsync --port=14415 -r example localhost::Uploads

Note

  • Recursive transfers always implies incremental recursion.

  • Use --charset for setting common character set (defaults to UTF-8). Note that --iconv is not supported.

  • Client local file transfers always uses rsync:s delta transfer algorithm, i.e. it does not have an option --whole-file.

  • Checksum block size is not computed in the exact same way as rsync. It is computed dynamically based on the file size and is always an even multiple of 2 and at least 512 bytes long.

  • Wild cards are not supported.

Extra features

  • (Receiver) --defer-write - defer writing into a temporary file until the content of the target file needs to be updated.

  • Support for custom Java file system providers with client option --fs and server module parameter fs.

Build instructions

Requirements:

Procedure:

git clone https://github.com/perlundq/yajsync.git
cd yajsync
mvn

Usage

Show client/server help (-h argument):

(Windows):

java -jar yajsync-app/target/yajsync-app-0.9.0-SNAPSHOT-full.jar client -h
java -jar yajsync-app/target/yajsync-app-0.9.0-SNAPSHOT-full.jar server -h

(Unix/Linux, we must inject necessary umask information as a property, assuming Bourne shell compatible syntax)

java -Dumask=$(umask) -jar yajsync-app/target/yajsync-app-0.9.0-SNAPSHOT-full.jar client -h
java -Dumask=$(umask) -jar yajsync-app/target/yajsync-app-0.9.0-SNAPSHOT-full.jar server -h

Recommended extra options to the jvm (i.e. must be placed before the -jar argument):

Turn on assertions:

-ea

Use a more memory conservative garbage collector:

-XX:+UseConcMarkSweepGC

Turn on aggressive optimisations:

-XX:+AggressiveOpts

SSL/TLS is configured externally (see JSSE documentation), but the following properties are used (options to the JVM):

-Djavax.net.ssl.keyStore=...
-Djavax.net.ssl.keyStoreAlias=...
-Djavax.net.ssl.keyStorePassword=...
-Djavax.net.ssl.trustStore=...
-Djavax.net.ssl.trustStorePassword=...

javax.net.debug is useful for debugging SSL/TLS. To see available values to javax.net.debug:

-Djavax.net.debug=help

Note: client side authorisation is not yet implemented - requires changes to server configuration.

License

Copyright (C) 1996-2011 by Andrew Tridgell, Wayne Davison, and others

Copyright (C) 2013-2016 Per Lundqvist

yajsync is licensed under GNU General Public License version 3 or later. See the file LICENSE or http://www.gnu.org/licenses/gpl.txt for the license details.

yajsync's People

Contributors

bark4mark avatar dadoonet avatar perlundq avatar usrflo avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

yajsync's Issues

Event listener

It would be great to add event listener to client API in order to track rsync progress.
-v console output may be done from listener implementation.

Exception propagation for SystemTests

Writing a system test for timeout detection I'm ending up with the logging output of the stacktrace below (which is correct and expected). In this test I'm using the YajSyncClient that does not propagate exceptions but returns the result code like the native rsync client is doing. What options do you see to enable exception propagation if YajSyncClient/Server should be testable with annotations like

@test(expected=SocketTimeoutException.class)

?

com.github.perlundq.yajsync.channels.ChannelException: java.net.SocketTimeoutException
    at com.github.perlundq.yajsync.channels.SimpleInputChannel.get(SimpleInputChannel.java:134)
    at com.github.perlundq.yajsync.channels.SimpleInputChannel.getByte(SimpleInputChannel.java:62)
    at com.github.perlundq.yajsync.channels.BufferedDuplexChannel.getByte(BufferedDuplexChannel.java:92)
    at com.github.perlundq.yajsync.channels.AutoFlushableDuplexChannel.getByte(AutoFlushableDuplexChannel.java:41)
    at com.github.perlundq.yajsync.session.SessionConfig.readLine(SessionConfig.java:112)
    at com.github.perlundq.yajsync.session.SessionConfig.receivePeerVersion(SessionConfig.java:167)
    at com.github.perlundq.yajsync.session.SessionConfig.exchangeProtocolVersion(SessionConfig.java:93)
    at com.github.perlundq.yajsync.session.ClientSessionConfig.handshake(ClientSessionConfig.java:83)
    at com.github.perlundq.yajsync.RsyncClient$ModuleListing$1.call(RsyncClient.java:247)
    at com.github.perlundq.yajsync.RsyncClient$ModuleListing$1.call(RsyncClient.java:1)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketTimeoutException
    at sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:229)
    at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
    at com.github.perlundq.yajsync.channels.net.StandardSocketChannel.read(StandardSocketChannel.java:83)
    at com.github.perlundq.yajsync.channels.SimpleInputChannel.get(SimpleInputChannel.java:122)
    ... 13 more

User authentication and flexible configuration on server side

There is some prepared code for user authentication in ClientSessionConfig.printLinesAndGetReplyStatus and some comment at ServerSessionConfig.setModule that the authentication request should be done depending on the module configuration. As far as I can see the authentication on server side is not implemented yet?

The prepared source code in ClientSessionConfig.printLinesAndGetReplyStatus hashes the user password by MD5.

Some ideas regarding the server-side implementation:

  • the current rsync way to save a clear-text password in the secrets-file besides the module configuration is not favourable. I appreciate your choice to send the MD5-hashed password to the server.
  • for being able to provide a flexible configuration I'd prefer an optional config syntax like it was used in the postfix mailserver configuration, e.g.

user = mysql:/yajsync/user-sql.cfg
OR
path = mysql:/yajsync/path-sql.cfg
OR
path = ldap:/yajsync/path-ldap.cfg
and so on.

The config-modules (mysql or ldap or text or whatever) would have a generic interface to set (a) the current module (b) the username to select the configuration value from the config-module database.

The implementation of config-modules could be separate (in separate JARs on the classpath) and each with its custom configuration, e.g. a mysql module would refer to a path-sql.cfg that could look like this:
user = mydbuser
password = mySecPas$
hosts = 127.0.0.1
dbname = yajsync
table = userconfig
select_field = path
where_field_module = module
where_field_user = user

Regarding http://linux.die.net/man/5/rsyncd.conf this flexibility would allow to
a) use one module configuration with different paths for different users
b) user specific exclude/include options if once implemented
c) user specific 'allow host' directives
...

  • considering a SSL enhancement for transport security I think users could use a server-side truststore optionally for client-authentication. I'd personally would not focus on that (maybe this part could be left to default JSSE security options).

What about these proposals?

MessageHeader ERROR_XFER

What's the meaning of
"Message MessageHeader ERROR_XFER length=108 java.nio.HeapByteBuffer[pos=0 lim=108 cap=108]" ?

java.lang.RuntimeException: TODO: not implemented: Sender.handleMessage(Message MessageHeader ERROR_XFER length=108 java.nio.HeapByteBuffer[pos=0 lim=108 cap=108])
at com.github.perlundq.yajsync.session.Sender.handleMessage(Sender.java:211)
at com.github.perlundq.yajsync.channels.TaggedInputChannel.readNextMessage(TaggedInputChannel.java:92)
at com.github.perlundq.yajsync.channels.TaggedInputChannel.readNextAvailable(TaggedInputChannel.java:62)
at com.github.perlundq.yajsync.channels.PrefetchedTaggedInputChannel.ensureMinimumPrefetched(PrefetchedTaggedInputChannel.java:161)
at com.github.perlundq.yajsync.channels.PrefetchedTaggedInputChannel.getByte(PrefetchedTaggedInputChannel.java:85)
at com.github.perlundq.yajsync.channels.IndexDecoderImpl.decodeIndex(IndexDecoderImpl.java:40)
at com.github.perlundq.yajsync.channels.RsyncInChannel.decodeIndex(RsyncInChannel.java:48)
at com.github.perlundq.yajsync.channels.AutoFlushableRsyncDuplexChannel.decodeIndex(AutoFlushableRsyncDuplexChannel.java:60)
at com.github.perlundq.yajsync.session.Sender.sendFiles(Sender.java:272)
at com.github.perlundq.yajsync.session.Sender.send(Sender.java:170)
at com.github.perlundq.yajsync.session.RsyncServerSession$1.call(RsyncServerSession.java:86)
at com.github.perlundq.yajsync.session.RsyncServerSession$1.call(RsyncServerSession.java:79)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)

Push or pull protocol

From the API, I could understand that it transfers data in one direction and pushes/uploads data from source to destination.

From the below example illustrated:
java -Dumask=$(umask) -jar yajsync-app/target/yajsync-app-0.9.0-SNAPSHOT-full.jar client --port=14415 -r example localhost::Uploads

it seems that we are uploading data from Client(from example directory) to server(in uploads directory).

Please confirm if this is true ?

protocol mismatch: rsync sender sends trailing data on nonexisting path

rsync sender sends trailing data when specifying a recursive remote file transfer from yajsync -> rsync daemon and a nonexisting path on the sender side. E.g.

yajsync -rvvv rsync://rsync.kernel.org/pub/no_such_path
FINER: deferring exception raised by task 1/2: java.util.concurrent.ExecutionException: com.github.perlundq.yajsync.session.RsyncProtocolException: Unexpectedly got 2 bytes from peer during connection tear down: [0xff, 0x01]
Mar 23, 2016 11:44:57 PM com.github.perlundq.yajsync.session.RsyncTaskExecutor exec
FINER: waiting for result from task 2/2
Mar 23, 2016 11:44:57 PM com.github.perlundq.yajsync.session.RsyncTaskExecutor exec
FINER: task 2/2 finished OK
Mar 23, 2016 11:44:57 PM com.github.perlundq.yajsync.RsyncClient$FileListing$2 call
FINE: shutting down java.util.concurrent.ThreadPoolExecutor@1c80acf[Running, pool size = 3, active threads = 1, queued tasks = 0, completed tasks = 2]
Exception in thread "main" com.github.perlundq.yajsync.session.RsyncProtocolException: Unexpectedly got 2 bytes from peer during connection tear down: [0xff, 0x01]
at com.github.perlundq.yajsync.session.Receiver.readAllMessagesUntilEOF(Receiver.java:1913)
at com.github.perlundq.yajsync.session.Receiver.call(Receiver.java:368)
at com.github.perlundq.yajsync.session.Receiver.call(Receiver.java:68)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

Connection is prematurely closed by receiver at connection teardown

Connection is prematurely closed by receiver at connection teardown. Reproduced by:

  • start yajsyncd server on port 14415:
    yajsyncd -vvvv --port=14415 --config=yajsyncd.conf
  • upload some file using rsync or yajsync:
rsync --port=14415 -r projects/coreutils localhost::upload
rsync: connection unexpectedly closed (15 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(226) [sender=3.1.1pre2]
  • yajsyncd output:
...snip...
FINE: Received index -1
Oct 23, 2014 10:33:13 AM com.github.perlundq.yajsync.session.Receiver receiveFiles
FINE: tearing down at phase ConnectionState stopped
Oct 23, 2014 10:33:13 AM com.github.perlundq.yajsync.session.Receiver receive
FINE: Receiver returned 0 errors
Oct 23, 2014 10:33:13 AM com.github.perlundq.yajsync.session.Generator processJobQueueBatched
FINE: (Generator) got 1 job(s)
Oct 23, 2014 10:33:13 AM com.github.perlundq.yajsync.session.Generator processJobQueueBatched
FINE: (Generator) processing sendSegmentDone()
Oct 23, 2014 10:33:13 AM com.github.perlundq.yajsync.session.Generator processJobQueueBatched
FINE: (Generator) awaiting next jobs...
Oct 23, 2014 10:33:13 AM com.github.perlundq.yajsync.session.Generator processJobQueueBatched
FINE: (Generator) got 1 job(s)
Oct 23, 2014 10:33:13 AM com.github.perlundq.yajsync.session.Generator processJobQueueBatched
FINE: (Generator) processing stop()
Oct 23, 2014 10:33:13 AM com.github.perlundq.yajsync.ui.YajSyncServer$7 call
SEVERE: Error: communication closed with peer: java.nio.channels.ClosedChannelException
Oct 23, 2014 10:33:13 AM com.github.perlundq.yajsync.ui.YajSyncServer$7 call
FINE: Thread exit status: ERROR

receiver: --owner is unable to handle incoming files w/ only uid and no name

$ sudo yajsync -o  rsync://kernel.org/pub/linux/kernel/next/patch-v3.2-rc7-next-20120105.xz .
...snip...
Exception in thread "main" com.github.perlundq.yajsync.session.RsyncProtocolException: unable to find mapping for uid 2000 in peer uid list {0=User (root, 0)}
        at com.github.perlundq.yajsync.session.Receiver.addUserNameToStubs(Receiver.java:374)
        at com.github.perlundq.yajsync.session.Receiver.call(Receiver.java:285)
        at com.github.perlundq.yajsync.session.Receiver.call(Receiver.java:65)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

rsync compatibility and special characters in path names

I added usrflo@42425a3 to run the same system tests with a yajsync or rsync client for simplified rsync compatibility checks.

With the following test I experienced unexpected behaviour with the rsync client. These problems do not exist with the yajsync client.

@Test
public void testClientDirCopyDotted() throws IOException
{
    Path src = _tempDir.newFolder().toPath();
    Path dst = Paths.get(src.toString() + ".dst");

    Path srcDir1 = src.resolve("dir");
    Path srcDir2 = src.resolve("dir.2");
    Path srcFile1 = srcDir1.resolve("file1");
    Path srcFile2 = srcDir2.resolve("file2");
    Files.createDirectory(srcDir1);
    Files.createDirectory(srcDir2);
    FileUtil.writeToFiles(7, srcFile1);
    FileUtil.writeToFiles(8, srcFile2);

    Files.createDirectory(dst);

    ReturnStatus status = fileCopy(src, dst, "--recursive");

    assertTrue(status.rc == 0);
}

With srcDir2 set on "dir.2" or "dir-2" the path inclusion check inside
com.github.perlundq.yajsync.filelist.Filelist.SegmentBuilder.add(FileInfo) reports an error:

/tmp/junit2086544845478406680/junit7751461368613446554.dst/junit7751461368613446554/dir should be a path prefix to: /tmp/junit2086544845478406680/junit7751461368613446554.dst/junit7751461368613446554/dir.2/file2

OR

/tmp/junit5154228388051712892/junit3843589390619568255.dst/junit3843589390619568255/dir should be a path prefix to: /tmp/junit5154228388051712892/junit3843589390619568255.dst/junit3843589390619568255/dir-2/file2

With srcDir2 set to other values like "dir2" or "dir_2" everything is ok.

There seems to be a special handling of directory names in rsync? Does this have to be addressed in YajSync? (please be aware I am working on a branch that I last merged in November)

How to dynamically send the directory that needs to be synced?

I tested the server and noticed that the synced directory needs to be predefined in yajsyncd.conf file. Is there a way that I can randomly give the directory that are not defined in the conf file?

Rsync program has the --progress options which can show the progress of the rsync, does current yajsync support this option as well?

Support Path instead of String for pathnames

There are comments like the following that declare why some pathnames have to be saved as String not Path in the current implementation. To be able to support URIs of different FileSystems there should be a dual approach:

  • allow URIs locally
  • use according String representations for remote processing

Is there anything that could interfere with such a local/remote differentiation?

/**
 * a class for rsync file information, we must keep the path name as a string
 * as opposed to storing the Path directly since the latter may fail
 * (when being the receiver) and we must keep the file list identical to the
 * peer's file list when being receiver
 */
public class FileInfo implements Comparable<FileInfo>

Bug in RsyncUrl remote path building if remote port is updated

In case a --port=... is defined in the client YajSyncClient.updateRemotePort(Path, int, RsyncUrls, RsyncUrl) is called and RsyncUrl.toRemotePathName(String, String) is executed once more. In line https://github.com/perlundq/yajsync/blob/master/src/main/com/github/perlundq/yajsync/ui/RsyncUrl.java#L179 an already built pathname is appended to the moduleName again. Supposed there was no path given the target path will be after the remote port update.

IAE when receiving existing file larger than 2 GiB

Example:

$ stat -c %s file.2G
2147483648
$ yajsync file.2G out
Warning: this software is still unstable and there might be data corruption bugs hiding. So use it only carefully at your own risk.
$ yajsync file.2G out
Warning: this software is still unstable and there might be data corruption bugs hiding. So use it only carefully at your own risk.
$ stat -c %s file.2G+1
2147483649
$ yajsync file.2G+1 out
Warning: this software is still unstable and there might be data corruption bugs hiding. So use it only carefully at your own risk.
$ yajsync file.2G+1 out
Warning: this software is still unstable and there might be data corruption bugs hiding. So use it only carefully at your own risk.
Exception in thread "main" java.lang.IllegalArgumentException: Negative position
        at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:676)
        at com.github.perlundq.yajsync.session.Receiver.matchReplica(Receiver.java:1474)
        at com.github.perlundq.yajsync.session.Receiver.combineDataToFile(Receiver.java:1371)
        at com.github.perlundq.yajsync.session.Receiver.mergeDataFromPeerAndReplica(Receiver.java:1281)
        at com.github.perlundq.yajsync.session.Receiver.matchData(Receiver.java:882)
        at com.github.perlundq.yajsync.session.Receiver.receiveFiles(Receiver.java:747)
        at com.github.perlundq.yajsync.session.Receiver.call(Receiver.java:308)
        at com.github.perlundq.yajsync.session.Receiver.call(Receiver.java:65)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

deadlock when a thread exits prematurely for a local rsync

In certain circumstances one thread may hang indefinitely if one thread exits prematurely due to an error. E.g. rsync file file/ is supposed to exit with an error, but due to this bug yajsync will also hang forever:

$ echo file > file
$ echo file2 > file2
$ yajsync file file2/
Warning: this software is still unstable and there might be data corruption bugs hiding. So use it only carefully at your own risk.
Oct 07, 2014 10:14:56 AM com.github.perlundq.yajsync.session.Receiver receive
SEVERE: unable to stat /tmp/file2/.: java.nio.file.FileSystemException: /tmp/file2/.: Not a directory
...hangs indefinitely...

This is only a problem for a local running rsync. It is caused by failing to properly close the channel(s) being used by the exiting thread - the peer thread on the other end is never notified that the thread on the other end isn't there anymore.

failing to update posix file permissions correctly even when this is supported

$ mkdir dir
$ echo file > dir/file
$ yajsync -r dir/ dir.new
Warning: this software is still unstable and there might be data corruption bugs hiding. So use it only carefully at your own risk.
$ ls -l dir dir.new
dir:
total 4
-r--r--r-- 1 perl perl 5 Oct  6 23:26 file

dir.new:
total 4
-rw------- 1 perl perl 5 Oct  6 23:28 file

in this case dir.new/file is expected to have perms 0444

Keep alive message handling

The native rsync implementation sends a MSG_DATA message in the last protocol version to keep a connection alive. This message is sent in different i/o scenarios (checks are executed at fixed places in i/o processing) depending on the time when the last packages were sent. The The following comment comes from io.c:

/* Older rsync versions used to send either a MSG_NOOP (protocol 30) or a
 * raw-data-based keep-alive (protocol 29), both of which implied forwarding of
 * the message through the sender.  Since the new timeout method does not need
 * any forwarding, we just send an empty MSG_DATA message, which works with all
 * rsync versions.  This avoids any message forwarding, and leaves the raw-data
 * stream alone (since we can never be quite sure if that stream is in the
 * right state for a keep-alive message). */
void maybe_send_keepalive(time_t now, int flags)

Bug reports like https://bugzilla.samba.org/show_bug.cgi?id=7757 or hints like the following from https://download.samba.org/pub/rsync/rsync.html reveal that the implementation in native rsync is improvable:

--delete-before [...] Deleting before the transfer is helpful if the filesystem is tight for space and removing extraneous files would help to make the transfer possible. However, it does introduce a delay before the start of the transfer, and this delay might cause the transfer to timeout (if --timeout was specified).

Considering MSG_DATA is stable and compatible with different rsync protocol versions I propose to use scheduled threads (ScheduledExecutorService.scheduleAtFixedRate) to send out MSG_DATA packages independently from sender processing. This may result in more packages than required but assures timeouts are not met as long as the sender is running.

Do you see any better way to solve this?

Optional atomic move to support differing file systems

The comment of the method FileOps.atomicMove() already tells that this method might fail:

        // this can fail in many ways, maybe we can try to recover some of them?

If the file systems in both path arguments differ the option StandardCopyOption.ATOMIC_MOVE produces an error: java.nio.file.AtomicMoveNotSupportedException: Atomic move between providers is not supported

My proposal to recover this case in https://github.com/perlundq/yajsync/blob/master/src/main/com/github/perlundq/yajsync/util/FileOps.java#L270 :

        if (tempFile.getFileSystem().equals(path.getFileSystem())) {
            Files.move(tempFile, path, StandardCopyOption.ATOMIC_MOVE);
        } else {
            Files.move(tempFile, path);
        }

Licence

Hi there

Would it be possible to licence the core component as Apache or MIT?

Regards

How to assure rsync support of "safe file lists" ?

yajsync currently supports a minimal subset of rsync protocol version 30.0, with the additional constraint that the peer must also support rsync safe file lists.

How can I assure my native rsync client supports "safe file lists" ?

$ rsync --version
rsync  version 3.1.0  protocol version 31
Copyright (C) 1996-2013 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
    64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
    socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
    append, ACLs, xattrs, iconv, symtimes, prealloc

Transport level security / SSL option

I did some research about wrapping ReadableByteChannel and
WritableByteChannel by SSL: the common way is to use http://docs.oracle.com/javase/7/docs/api/javax/net/ssl/SSLEngine.html but some developers warned this could be hard stuff ;)

A sample implementation can be found in the package org.apache.tomcat.util.net/ at http://svn.apache.org/repos/asf/tomcat/trunk/java/org/apache/tomcat/util/net/
See esp. SecureNioChannel.java, it is the ssl secured implementation of a NIO channel.

Maybe you could check this to confirm if this should fit in the current yajsync architecture.

broken pipe/deadlock when deferred updates fails

A deadlock or a broken pipe may occur if Generator tries to notify Sender after Sender and Receiver has agreed to go to the stopped connection state.

Generator will try to notify Sender if any of the deferred attribute updates fails, but Sender is gone and is no longer receiving the messages.

  1. E.g. a broken pipe occurs when doing remote recursive transfers as an unprivileged user and trying to preserve ownership:

    $ yajsync -ro localhost::etc etc.copy

    ...snip...
    WARNING: received I/O error while applying attributes on etc.copy/libvirt/qemu/networks/autostart: etc.copy/libvirt/qemu/networks/autostart: Operation not permitted
    Mar 21, 2016 4:41:12 PM com.github.perlundq.yajsync.ui.YajSyncClient remoteTransfer
    SEVERE: Error: communication closed with peer:
    com.github.perlundq.yajsync.channels.ChannelException: java.io.IOException: Broken pipe
    at com.github.perlundq.yajsync.channels.BufferedOutputChannel.send(BufferedOutputChannel.java:70)
    at com.github.perlundq.yajsync.channels.BufferedOutputChannel.flush(BufferedOutputChannel.java:79)
    at com.github.perlundq.yajsync.channels.TaggedOutputChannel.flush(TaggedOutputChannel.java:78)
    at com.github.perlundq.yajsync.channels.TaggedOutputChannel.putMessage(TaggedOutputChannel.java:55)
    at com.github.perlundq.yajsync.session.Generator$8.process(Generator.java:1014)
    at com.github.perlundq.yajsync.session.Generator$2.process(Generator.java:417)
    at com.github.perlundq.yajsync.session.Generator.processJobQueueBatched(Generator.java:355)
    at com.github.perlundq.yajsync.session.Generator.call(Generator.java:373)
    at com.github.perlundq.yajsync.session.Generator.call(Generator.java:74)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
    Caused by: java.io.IOException: Broken pipe
    at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
    at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
    at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
    at sun.nio.ch.IOUtil.write(IOUtil.java:51)
    at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)
    at com.github.perlundq.yajsync.channels.net.StandardSocketChannel.write(StandardSocketChannel.java:62)
    at com.github.perlundq.yajsync.channels.BufferedOutputChannel.send(BufferedOutputChannel.java:60)
    ... 14 more

  2. Likewise, a deadlock may occur if doing the same kind of transfer locally (yajsync -ro /etc etc.copy).
    But: A deadlock does not always occur. And it seems like this also should always result in a broken pipe, but it does not (though it uses Java pipes rather than a tcp socket).

ant build fails since 45360cc

$ ant
Buildfile: /tmp/yajsync/build.xml

init:
[mkdir] Created dir: /tmp/yajsync/build/classes
[mkdir] Created dir: /tmp/yajsync/build/jar

build:
[echo] YajSync: /tmp/yajsync/build.xml
[javac] Compiling 81 source files to /tmp/yajsync/build/classes
[javac] /tmp/yajsync/src/test/com/github/perlundq/yajsync/session/IntegerCoderTest.java:3: error: package org.junit does not exist
[javac] import static org.junit.Assert.assertEquals;
[javac] ^
...snip...

non recursive module listing requires currently nonexisting -d (--dirs) option

# already running yajsyncd on port 14415
$ rsync rsync://localhost:14415
Downloads      
Uploads        
Downloads2     
$ rsync rsync://localhost:14415/Downloads
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(605) [Receiver=3.0.9]

# yajsyncd error log:
Jan 10, 2014 7:42:35 PM com.github.perlundq.yajsync.ui.YajSyncServer$8 call
SEVERE: 
com.github.perlundq.yajsync.session.RsyncProtocolException: com.github.perlundq.yajsync.util.ArgumentParsingError: d - unknown option
        at com.github.perlundq.yajsync.session.ServerSessionConfig.handshake(ServerSessionConfig.java:123)
        at com.github.perlundq.yajsync.session.RsyncServerSession.startSession(RsyncServerSession.java:155)
        at com.github.perlundq.yajsync.ui.YajSyncServer$8.call(YajSyncServer.java:198)
        at com.github.perlundq.yajsync.ui.YajSyncServer$8.call(YajSyncServer.java:190)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:724)
Caused by: com.github.perlundq.yajsync.util.ArgumentParsingError: d - unknown option
        at com.github.perlundq.yajsync.util.ArgumentParser.getOptionForName(ArgumentParser.java:309)
        at com.github.perlundq.yajsync.util.ArgumentParser.parse(ArgumentParser.java:129)
        at com.github.perlundq.yajsync.session.ServerSessionConfig.parseArguments(ServerSessionConfig.java:261)
        at com.github.perlundq.yajsync.session.ServerSessionConfig.handshake(ServerSessionConfig.java:118)
        ... 8 more

see options.c:server_options() and options.c:parse_arguments()

xfer_dirs is initialized to -1 and if doing non-recursive module listing it is set to 1...

--bwlimit and external project dependency

In native rsync, io.c, the sleep_for_bwlimit function is commented as follows:

/* Sleep after writing to limit I/O bandwidth usage.
 *
 * @todo Rather than sleeping after each write, it might be better to
 * use some kind of averaging.  The current algorithm seems to always
 * use a bit less bandwidth than specified, because it doesn't make up
 * for slow periods.  But arguably this is a feature.  In addition, we
 * ought to take the time used to write the data into account.
 *
 * During some phases of big transfers (file FOO is uptodate) this is
 * called with a small bytes_written every time.  As the kernel has to
 * round small waits up to guarantee that we actually wait at least the
 * requested number of microseconds, this can become grossly inaccurate.
 * We therefore keep track of the bytes we've written over time and only
 * sleep when the accumulated delay is at least 1 tenth of a second. */

When adding --bwlimit to yajsync the mentioned improvement of the @todo section above (averaging) should be considered. The token bucket algorithm would fit the requirements of such an improvement.

Taking a look at https://github.com/bbeck/token-bucket I guess this would be an appropriate implementation to simply integrate bandwidth limits. Taking in account you like to keep dependencies low:

  • would you propose to re-implement such an algorithm?
  • would you propose to re-use existing code but integrate it with its license to the yajsync source code as long as being compatible?
  • would you be open for a maven dependency?

Preconditions for adding timeout options

Rsync supports connection and send/receive timeouts according to the following options:

--timeout=SECONDS
This option allows you to set a maximum I/O timeout in seconds. If no data is transferred for the specified time then rsync will exit. The default is 0, which means no timeout.
Exit-Code 30: Timeout in data send/receive

--contimeout=SECONDS
This option allows you to set the amount of time that rsync will wait for its connection to an rsync daemon to succeed. If the timeout is reached, rsync exits with an error.
Exit-Code 35: Timeout waiting for daemon connection

I'd like to discuss the ways how to implement these timeouts being non-familiar with NIO blocking/non-blocking.

In principle I could imagine to change the factories for plain/ssl sockets in the following way:

public class StandardChannelFactory implements ChannelFactory
{
    @Override
    public DuplexByteChannel open(String address, int remotePort)
        throws IOException
    {
        InetSocketAddress socketAddress = new InetSocketAddress(address,
                                                                remotePort);
        // SocketChannel socketChannel = SocketChannel.open(socketAddress);

        SocketChannel socketChannel = SocketChannel.open();
        socketChannel.socket().setSoTimeout(sndRcvTimeout);
        socketChannel.socket().connect(socketAddress, connectionTimeout);

        return new StandardSocketChannel(socketChannel);
    }
}
public class SSLChannel implements DuplexByteChannel
{
...
public static SSLChannel open(String address, int port) throws IOException
    {
        SocketFactory factory = SSLSocketFactory.getDefault();
        // Socket sock = factory.createSocket(address, port);
        InetSocketAddress socketAddress = new InetSocketAddress(address, port);
        Socket sock = factory.createSocket();
        sock.setSoTimeout(sndRcvTimeout);
        sock.connect(socketAddress, connectionTimeout);
        return new SSLChannel((SSLSocket) sock);
    }
...
}

After reading https://technfun.wordpress.com/2009/01/29/networking-in-java-non-blocking-nio-blocking-nio-and-io/ I think at least com.github.perlundq.yajsync.channels.net.StandardSocketChannel might have to be changed so read/write is executed on streams (seems like r/w works on the socket directly right now).

@perlundq: I didn't dig into this yet, may you can give some advice first?

test failure in testConnectionTimeout

cloning latest master as of today and running the tests gives me the following error on Linux x86-64 (Ubuntu 16.04) kernel 4.5.1-040501-generic, Oracle JDK 8 (build 1.8.0_91-b14, 64bit server), Apache Maven 3.3.9:

testConnectionTimeout(com.github.perlundq.yajsync.test.SystemTest): Unexpected exception, expected<java.net.SocketTimeoutException> but was<java.net.SocketException>

Running com.github.perlundq.yajsync.test.SystemTest
drwxrwxr-x           2 2016/06/24 17:27:34 ./
test            a test module
Jun 24, 2016 5:27:35 PM com.github.perlundq.yajsync.internal.session.ServerSessionConfig handshake
WARNING: failed to authenticate 
drwxrwxr-x           2 2016/06/24 17:27:35 ./
drwxrwxr-x           3 2016/06/24 17:27:35 ./
-rw-rw-r--           0 2016/06/24 17:27:35 ./file
Tests run: 32, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.672 sec <<< FAILURE!
testConnectionTimeout(com.github.perlundq.yajsync.test.SystemTest)  Time elapsed: 0.004 sec  <<< ERROR!
java.lang.Exception: Unexpected exception, expected<java.net.SocketTimeoutException> but was<java.net.SocketException>
    at org.junit.internal.runners.statements.ExpectException.evaluate(ExpectException.java:28)
    at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:274)
    at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:268)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketException: Network is unreachable
    at sun.nio.ch.Net.connect0(Native Method)
    at sun.nio.ch.Net.connect(Net.java:454)
    at sun.nio.ch.Net.connect(Net.java:446)
    at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648)
    at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:102)
    at com.github.perlundq.yajsync.net.StandardSocketChannel.open(StandardSocketChannel.java:53)
    at com.github.perlundq.yajsync.net.StandardChannelFactory.open(StandardChannelFactory.java:27)
    at com.github.perlundq.yajsync.test.SystemTest.testConnectionTimeout(SystemTest.java:1000)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at org.junit.internal.runners.statements.ExpectException.evaluate(ExpectException.java:19)
    ... 4 more

Preview of FILTER RULE implementation

Please take a look at usrflo@378754a
These classes implement http://rsync.samba.org/ftp/rsync/rsync.html, section FILTER RULES, INCLUDE/EXCLUDE PATTERN RULES and MERGE-FILE FILTER RULES.

I could integrate this filter rule processing in yajsync if you don't have other plans. The integration consists of:

  • reading option arguments
  • initialize the filter rule configuration classes according to the rsync filter rule syntax
  • check the rules on dir/file list processing

Please tell me if you are interested.

Sender throws IllegalArgumentException if source dir has only 1 component

$ yajsync -r /home target
Warning: this software is still unstable and there might be data corruption bugs hiding. So use it only carefully at your own risk.
Exception in thread "main" java.lang.IllegalArgumentException
        at sun.nio.fs.UnixPath.subpath(UnixPath.java:350)
        at sun.nio.fs.UnixPath.subpath(UnixPath.java:43)
        at com.github.perlundq.yajsync.util.PathOps.parentPath(PathOps.java:75)
        at com.github.perlundq.yajsync.util.PathOps.subtractPath(PathOps.java:81)
        at com.github.perlundq.yajsync.session.Sender.getLocalPathOf(Sender.java:1024)
        at com.github.perlundq.yajsync.session.Sender.expand(Sender.java:519)
        at com.github.perlundq.yajsync.session.Sender.expandAndSendSegments(Sender.java:626)
        at com.github.perlundq.yajsync.session.Sender.sendFiles(Sender.java:242)
        at com.github.perlundq.yajsync.session.Sender.send(Sender.java:164)
        at com.github.perlundq.yajsync.session.RsyncLocal$3.call(RsyncLocal.java:156)
        at com.github.perlundq.yajsync.session.RsyncLocal$3.call(RsyncLocal.java:152)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

There is no problem if source has more than one name component, e.g. yajsync -r /home/dir target works OK.

Simply RsyncFileAttributes.stat(Path) / support of more file attribute views

The following code snippet could be an example how to support more file attribute views beyond POSIX and Basic, for e.g. support of S_ISUID, S_ISGID and S_ISVTX and without the necessity to map via int toMode(...):

public static RsyncFileAttributes stat(Path path) throws IOException
{
    if (Files.getFileStore(path).supportsFileAttributeView("unix")) {
        Map<String,Object> attrMap = Files.readAttributes(path, "unix:*"); 
        int mode = (Integer) attrMap.get("mode");
        long size = (Long) attrMap.get("size");
        FileTime modTime = (FileTime) attrMap.get("lastModifiedTime");
        return new RsyncFileAttributes(mode, size, modTime.to(TimeUnit.SECONDS));
    }

...

Dose yajsync support incremental update ?

As I tested the yajsync , I find it update the whole file to server if the file has been changed on client,
so I am confused if I am missing some argument settings or the yajsync dose not support incremental update.

failing to copy single source file to a single destination

$ echo file > file
$ [ -e file.new ] || echo file does not exist
file does not exist
$ yajsync file file.new
Warning: this software is still unstable and there might be data corruption bugs hiding. So use it only carefully at your own risk.
Oct 06, 2014 11:10:55 PM com.github.perlundq.yajsync.session.Receiver receiveFiles
WARNING: failed to create tempfile for /tmp/file.new: /tmp/file.new/2704395247176515254.tmp

Support for MaxOS X rsync

On MacOS X, I'm trying to run:

rsync ~/tmp/test rsync://127.0.0.1:14415/Uploads

I'm getting back an error and this message on server side:

déc. 28, 2015 7:37:39 PM com.github.perlundq.yajsync.client.YajSyncServer$8 call
GRAVE: 
com.github.perlundq.yajsync.session.RsyncProtocolException: Unsupported protocol version: @RSYNCD: 29
    at com.github.perlundq.yajsync.session.SessionConfig.receivePeerVersion(SessionConfig.java:174)
    at com.github.perlundq.yajsync.session.SessionConfig.exchangeProtocolVersion(SessionConfig.java:93)
    at com.github.perlundq.yajsync.session.ServerSessionConfig.handshake(ServerSessionConfig.java:100)
    at com.github.perlundq.yajsync.RsyncServer.serve(RsyncServer.java:95)
    at com.github.perlundq.yajsync.client.YajSyncServer$8.call(YajSyncServer.java:221)
    at com.github.perlundq.yajsync.client.YajSyncServer$8.call(YajSyncServer.java:198)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

Is MacOS supported?

Thanks!

Server thread exits with an error when doing non-recursive module file listings

$ java -ea -Dumask=0002 -XX:+UseConcMarkSweepGC -XX:+AggressiveOpts -jar /home/perl/projects/yajsync/build/jar/yajsyncd.jar -vv --config=/home/perl/yajsyncd.conf --port=14415 &
$ java -ea -Dumask=0002 -XX:+UseConcMarkSweepGC -XX:+AggressiveOpts -jar build/jar/yajsync.jar --port=14415 localhost:: &>/dev/null
Feb 17, 2014 12:12:01 AM com.github.perlundq.yajsync.session.RsyncServerSession startSession
FINE: Got connection from /127.0.0.1:49296
Feb 17, 2014 12:12:01 AM com.github.perlundq.yajsync.session.ServerSessionConfig handshake
FINE: sending module listing and exiting
Feb 17, 2014 12:12:01 AM com.github.perlundq.yajsync.ui.YajSyncServer$8 call
FINE: Thread exit status: ERROR

symlinks are treated as the files they refer to instead of being skipped

$ echo file > file
$ ln -s file link
$ yajsync file link target/
Warning: this software is still unstable and there might be data corruption bugs hiding. So use it only carefully at your own risk.
$ ls -l target/
total 8
-rw-rw-r-- 1 perl perl 5 Oct  7 19:22 file
-rw-rw-r-- 1 perl perl 5 Oct  7 19:22 link

In this case link should be skipped. This bug is caused by reading contents/attributes of the file the link is referring to, rather than the link itself.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.