Page History

...

Server-to-Server File Transfer (S2S) is a unique key feature of JADEYADE. It provides the possibility to set-up a number of file transfers using a single server without installing clients/agents on each source/target host.

...

YADE Implementation Architecture

...

YADE Architecture Components

JADE includes the following main components:

1. CLI (Command Line Interface) executed at shell level
2. User Application Programming Interface (API)
3. JITLJobs scheduling component of JADEYADE
4. JADE YADE Engine
5. Options Engine
6. JADE YADE Engine Interface
7. VFS (Virtual File System) Interface (Data Provider)
   • FTP
   • SFTP
   • FTPS
   • HTTP/HTTPS

     Display feature availability
     StartingFromRelease 1.7

   • WebDAV 

     Display feature availability
     StartingFromRelease 1.8

   • Local (native copy)

...

YADE Server-to-Server File Transfer

digraph "Example: Server-to-Server File Transfer" {
rankdir=LR;
edge [color="#31CEF0"];

source_server [label="Data Source (SFTP)"];
target_server  [label="Data Target (FTP)"];
jadeyade_client  [label="JADEYADE Client \n API \n JITL Job"];
jadeyade_config [label="JADEYADE Configuration"];

source_server->target_server [label="data transfer"];
source_server->jade>yade_client  [dir=both label="commands" color="grey"];
jadeyade_client->target_server  [dir=both label="commands"  color="grey"];
jadeyade_config->jade>yade_client [color="green"];
}

S2S implementation details:

1. JADE YADE uses "Streams" (or in common terminology: pipes) to transfer data from source to target.
2. JADE YADE opens an input stream from the source server and and output stream to the target server.
3. JADE YADE can use different data provider as source and target for S2S operation.
4. By default JADE YADE will transfer 32Kb chunks from input to output stream. You can increase the buffer size by setting the buffer_size parameter, but a larger buffer size may not result in higher transfer speed.
5. JADE YADE can process the content of the buffer between source and target transfer. By doing in-memory operations, even for multistage processing, JADE YADE dramatically improves the transfer speed.
   1. Calculate Hashes
   2. Reserved for further development
      1. Filtering
      2. Encoding/Decoding
      3. Encryption/Decryption
6. JADE YADE transfers data from the source server to the target server without touchdown meaning that during transfer the data is never written to the file system on the intermediate server where JADE YADE is running. JADEYADE's key feature is in-memory transfer. During transfer JADE YADE does not copy any data to the memory, but only chunks whose size is defined with the buffer_size parameter.

For more information about JADE YADE Server-to-Server File Transfer configuration see Example for JADE YADE Server-to-Server File Transfer.

JADE YADE provides a generic implementation for Virtual File Systems or Data Providers that makes it easy to implement new protocols.

...