...
- YADE should support file transfer operations with Azure Blob Storage.
- Technically speaking Azure file transfer operations preferably make use of the HTTPS protocol and a number of query parameters and headers. The implementation with YADE therefore is fairly straightforward.
- The sticking point is about what authentication methods should be supported and how they would be operatedintegrated:
- If blob containers are made publicly available and require no authentication then the YADE supports this out-of-the-box.
- If Azure authentication Azure authentication methods are applied then different implementation strategies are to be considered. Azure supports a range of authentication methods - we picked two of them for this proposal:
- Use of Shared Key authentication, see Authorize with Shared Key
- Use of Shared Access Signature authentication (SAS), see Create an Account SAS
- There are impacts on job operation depending on the authentication method that is preferred by an organization.
...
Find a preliminary comparison of authentication methods:
CapabilityArea | Shared Key | Shared Access Signature |
---|---|---|
Scope |
|
|
Access Duration |
|
|
Permissions |
|
|
Implementation |
|
|
- From the above comparison Shared Access Signatures are superior concerning more fine-grained access to resources.
- At the same time Shared Key authentication suggests ease of use as a single Shared Key can be used to authenticate any file transfer operations for an unlimited duration.
- Technically a single Shared Access Signature can be applied in a similar way allowing any file transfer operation on any blobs in a given container for a longer period, e.g. the next 100 years.
...
- Considering use of authentication methods with file transfer jobs there is a clear final statement from SOS: implementation of any authentication method is out of scope of the JobScheduler product.
- Implementation of authentication methods is within the responsibility of the userorganization that uses JobScheduler. Reasons for this include that
- users should not trust any 3rd party implementation of authentication methods,
- a number of authentication methods are offered by Azure that allow an individual choice according to security requirements of an organization,
- there is ongoing development with Azure about improvement of authentication methods.
- Implementation of file transfer operations for Azure blob storage is within the scope of the YADE.
- Implementation of authentication methods is within the responsibility of the userorganization that uses JobScheduler. Reasons for this include that
- Technically this offers the following options:
- An organization can create a single Shared Key or Shared Access Signature that represents a constant value that is made available to all YADE file transfer jobs.
- An organization can create individual Shared Keys or Shared Access Signatures that are used with groups of YADE file transfer jobs or with individual jobs.
Implementation
User Tasks: Manage Shared Keys and Shared Access Signatures
Creating Shared Keys and Shared Access Signatures
- Find examples for creating Shared Keys and SAS tokens on the fly and for using Azure Find samples for use of authentication methods and respective file transfer operations from the following articles:
- The examples are far from being perfect, however, they might be useful as a basis for a user's implementation.
SOS Tasks: YADE Integration
Implementation Strategies
- When using a single Shared Key or Shared Access Signature for all YADE file transfer jobs then
- this could be added as a constant value to the JobScheduler configuration, e.g. with a parameter to the
./config/scheduler.xml
file. - this could be added to a Credential Store that would be accessed by any YADE file transfer jobs.
- this could be added as a constant value to the JobScheduler configuration, e.g. with a parameter to the
- When using individual Shared Keys or Shared Access Signatures per YADE file transfer job then a Monitor should be implemented that is assigned on a per job basis and that can be parameterized to use the key or signature specified by a job parameter or order parameter.
- Should Shared Keys or Shared Access Signatures have to be renewed on a regular basis then this can be executed performed by a job, e.g. running daily, that is implemented by a user of JobScheduler.
YADE Changes
- Accept for both Shared Keys and Shared Access Signatures the respective configuration items:
- For both authentication methods
- Add to URL
- Resources
- container resource
- blob resource
- Example
https://<owner account specified by user>.blob.core.windows.net/<container specified by user>/<blob specified by user>
https://yade.blob.core.windows.net/yade/test.txt
- Add HTTPS headers:
x-ms-blob-type : BlockBlob
x-ms-date : <date of request>
x-ms-version : <version as specified by user>
- Add to URL
- Shared Key
- Add HTTPS authorization header:
Authorization : SharedKey <requester account specified by user>:<signature specified by user>
`
- Add HTTPS authorization header:
- Shared Access Signatures
- Add query parameters to URL for operation to list blobs in a container
https://<owner account specified by user>.blob.core.windows.net/<container specified by user>/<blob specified by user>?<query parameters>
https://yade.blob.core.windows.net/yade/test.txt?restype=container&comp=list
- Add SAS token to URL:
https://<blob storage owner account specified by user>.blob.core.windows.net/<container specified by user>/<blob specified by user>?<SAS token specified by user>
- Add query parameters to URL for operation to list blobs in a container
- For both authentication methods
- Adjust the YADE XSD Schema to allow configuration of any number of HTTP headers
- at a global level
- per profile
- on a per job basis by job/order parameters and when using the YADE standalone client by command line parameters.
- Delimitation
- The implementation is about integrating the above configuration items into HTTPS headers and query parameters.
- The implementation is not about developing a new data provider that is subject to a separate feature request.
References
Jira | ||||||||
---|---|---|---|---|---|---|---|---|
|
...