Comments (1)
manishbansal8843
commented on August 21, 2024
Pasting the problem here also for clarity.
We have a java 7 code base where we are using Apache commons vfs2 v2.2 which is using JSch-0.1.54 as sftp provider.
Now, the use case is to transfer files to a remote host via sftp. But, every now and then, the file upload process gets stuck. After taking the thread dump of the application, we found both the threads (t1, which sends the data to remote sftp & t2, which receives data from sftp) are in wait state forever. Below is the thread dump snapshot.
JSch session thread:
"Connect thread remote.ftp.com session" daemon prio=10 tid=0x00007f99cc243000 nid=0x144 in Object.wait() [0x00007f9985606000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.io.PipedInputStream.awaitSpace(PipedInputStream.java:273)
at java.io.PipedInputStream.receive(PipedInputStream.java:231)
- locked <0x000000043eda02d8> (a com.jcraft.jsch.Channel$MyPipedInputStream)
at java.io.PipedOutputStream.write(PipedOutputStream.java:149)
at com.jcraft.jsch.IO.put(IO.java:64)
at com.jcraft.jsch.Channel.write(Channel.java:438)
at com.jcraft.jsch.Session.run(Session.java:1459)
at java.lang.Thread.run(Thread.java:748)
Application thread used to upload file data.
"akka.actor.default-dispatcher-19" prio=10 tid=0x00007f99d4012000 nid=0xea in Object.wait() [0x00007f9988785000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at com.jcraft.jsch.Session.write(Session.java:1269)
- locked <0x0000000440a61b48> (a com.jcraft.jsch.ChannelSftp)
at com.jcraft.jsch.ChannelSftp.sendWRITE(ChannelSftp.java:2646)
at com.jcraft.jsch.ChannelSftp.access$100(ChannelSftp.java:36)
at com.jcraft.jsch.ChannelSftp$1.write(ChannelSftp.java:806)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:122)
- locked <0x0000000440aab240> (a org.apache.commons.vfs2.provider.sftp.SftpFileObject$SftpOutputStream)
at org.apache.commons.vfs2.util.MonitorOutputStream.write(MonitorOutputStream.java:104)
- locked <0x0000000440aab240> (a org.apache.commons.vfs2.provider.sftp.SftpFileObject$SftpOutputStream)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
- locked <0x0000000440aac278> (a org.apache.commons.vfs2.provider.DefaultFileContent$FileContentOutputStream)
at org.apache.commons.vfs2.util.MonitorOutputStream.write(MonitorOutputStream.java:104)
- locked <0x0000000440aac278> (a org.apache.commons.vfs2.provider.DefaultFileContent$FileContentOutputStream)
at org.apache.commons.vfs2.provider.DefaultFileContent.write(DefaultFileContent.java:741)
at org.apache.commons.vfs2.provider.DefaultFileContent.write(DefaultFileContent.java:720)
at org.apache.commons.vfs2.provider.DefaultFileContent.write(DefaultFileContent.java:691)
at org.apache.commons.vfs2.provider.DefaultFileContent.write(DefaultFileContent.java:707)
at org.apache.commons.vfs2.FileUtil.copyContent(FileUtil.java:78)
at org.apache.commons.vfs2.provider.AbstractFileObject.copyFrom(AbstractFileObject.java:289)
After looking at the codebase of Jsch library, Here is what i feel is happening.
- Application thread is uploading file data in the chunks of 4KB.
- After every chunk write, application reads the input socket for any acknowledgements until input socket buffer is empty.
During chunk writing, it checks for the ssh window size. If it is less than the payload size, we wait until remote server resizes it.(This is a place where my app thread is waiting forever) This resizing message is listened by the ssh session thread and same is updated at the channel object after which application thread proceeds for writing. - In a separate thread, session is listening for the incoming data from the remote server. Depending on the message it receives, it takes the relevant action like resizing the channel window, pass the acknowledgement message to the channel (read application) for consumption.
- Now, when a message arrives for channel consumption, it is written to the PipedOutStream which is linked to the PipedInputStream. This input stream is read by the application thread for acknowledgement messages. In case, the application thread fails to read any message, the buffer of PipedOutputstream gets full and hence it goes into wait state until application reads some data. (This is the place where session thread is waiting forever)
- Now, both the threads are depending on each other. Hence, this is kind of a deadlock.
Further, i have checked the linux machine on which this app is running, RecQ of the socket is kept on building up. Which means, socket is still alive and remote server is keep on sending 32KB packets every now and then.
sudo netstat -anpt | grep 19321
tcp6 0 0 10.14.233.97:59594 64.233.167.130:19321 TIME_WAIT -
tcp6 58256 0 10.14.233.97:58214 64.233.167.130:19321 ESTABLISHED 460144/java
tcp6 499888 0 10.14.233.97:58422 64.233.167.130:19321 ESTABLISHED 460144/java
tcp6 0 0 10.14.233.97:59622 64.233.167.130:19321 ESTABLISHED 460144/java
tcp6 0 0 10.14.233.97:59608 64.233.167.130:19321 TIME_WAIT -
tcp6 74672 0 10.14.233.97:56656 64.233.167.130:19321 ESTABLISHED 460144/java
tcp6 92688 0 10.14.233.97:56842 64.233.167.130:19321 ESTABLISHED 460144/java
from jsch.
Related Issues (20)
- It's able to write bytes up to 512KB after sftp server is down
- OpenSSH key format HOT 1
- Is this repo not ready to be updated? HOT 1
- Please delete this request
- Internet access
- Packet corrupt problem HOT 3
- java.net.SocketException: Socket is not connected (Read failed)
- poweroff and reboot command not work HOT 2
- Will Jsch consider supporting other key exchange algorithms, such as curve25519?
- Connection failed: SSH_MSG_DISCONNECT: 2 The connection is closed by SSH Server
- ProxyHTTP Unknow reason
- How to use this method
- Does "com.jcraft.jsch:0.1.55" supports ECDSA 256 algorithm ?
- 1
- Connection hangs on session.connect
- observing the performance issue with jsch
- CVE-2023-48795
- Passing + options thru on the "ls" command
- Jsch not working with Server Secure Transport not supporting [email protected] [email protected]
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from jsch.