Scope
- YADE should support file transfer operations with Azure Blob Storage.
- Technically Azure file transfer operations preferably make use of the HTTPS protocol and a number of query parameters and headers. The implementation with YADE therefore is fairly straightforward.
- The sticking point is about what authentication methods should be supported and how they would be integrated:
- If blob containers are made publicly available and require no authentication then the YADE supports this out-of-the-box.
- If Azure authentication methods are applied then different implementation strategies are to be considered. Azure supports a range of authentication methods - we picked two of them for this proposal:
- There are impacts on job operation depending on the authentication method that is preferred by an organization.
Authentication Methods
Find a preliminary comparison of authentication methods:
Area | Shared Key | Shared Access Signature |
---|
Scope | - uses a single Shared Key for access at container level or at blob level
| - allows a single SAS token to be used for access to any blobs in a container
- allows individual SAS tokens to be used per blob and per file transfer operation
|
Access Duration | | - makes use of expiration dates
|
Permissions | - enables full access at container level or at blob level for any file transfer operations
| - offers permissions (read, write, delete, list) per resource type (container, object) and service (blob, queue, table, file)
- allows a single SAS token to be used that includes any permissions and file transfer operations (get, put, list)
- allows individual SAS tokens to be used specifying permissions per blob and per file transfer operation
|
Implementation | - A signature is created by hashing blob resource parameters and by specifying the blob to be accessed.
- Use of the HTTPS Authorization header for signature.
| - A signature is created by hashing blob resource parameters, the permissions that are assigned and the expiration date.
- An SAS token is generated that specifies both the plain text values of the signature and the hashed value of the signature.
- Use of HTTPS query parameters for SAS tokens. Such tokens specify the blob resource to be accessed, the permissions that are assigned, expiration period and signature.
|
- From the above comparison Shared Access Signatures are superior concerning more fine-grained access to resources.
- At the same time Shared Key authentication suggests ease of use as a single Shared Key can be used to authenticate any file transfer operations for an unlimited duration.
- Technically a single Shared Access Signature can be applied in a similar way allowing any file transfer operation on any blobs in a given container for a longer period, e.g. the next 100 years.
Operation
- Considering use of authentication methods with file transfer jobs there is a final statement from SOS: implementation of any authentication method is out of scope of the JobScheduler product.
- Implementation of authentication methods is within the responsibility of the organization that uses JobScheduler. Reasons for this include that
- users should not trust any 3rd party implementation of authentication methods,
- a number of authentication methods are offered by Azure that allow an individual choice according to security requirements of an organization,
- there is ongoing development with Azure about improvement of authentication methods.
- Implementation of file transfer operations for Azure blob storage is within the scope of the YADE.
- Technically this offers the following options:
- An organization can create a single Shared Key or Shared Access Signature that represents a constant value that is made available to all YADE file transfer jobs.
- An organization can create individual Shared Keys or Shared Access Signatures that are used with groups of YADE file transfer jobs or with individual jobs.
Implementation
User Tasks: Manage Shared Keys and Shared Access Signatures
Creating Shared Keys and Shared Access Signatures
- Find examples for creating Shared Keys and SAS tokens on the fly and for using Azure authentication methods and respective file transfer operations from the following articles:
- The examples are far from being perfect, however, they might be useful as a basis for a user's implementation.
SOS Tasks: YADE Integration
Implementation Strategies
- When using a single Shared Key or Shared Access Signature for all YADE file transfer jobs then
- this could be added as a constant value to the JobScheduler configuration, e.g. with a parameter to the
./config/scheduler.xml
file. - this could be added to a Credential Store that would be accessed by any YADE file transfer jobs.
- When using individual Shared Keys or Shared Access Signatures per YADE file transfer job then a Monitor should be implemented that is assigned on a per job basis and that can be parameterized to use the key or signature specified by a job parameter or order parameter.
- Should Shared Keys or Shared Access Signatures have to be renewed on a regular basis then this can be performed by a job, e.g. running daily, that is implemented by a user of JobScheduler.
YADE Changes
- Accept for both Shared Keys and Shared Access Signatures the respective configuration items:
- For both authentication methods
- Add to URL
- Resources
- container resource
- blob resource
- Example
https://<owner account specified by user>.blob.core.windows.net/<container specified by user>/<blob specified by user>
https://yade.blob.core.windows.net/yade/test.txt
- Add HTTPS headers:
x-ms-blob-type : BlockBlob
x-ms-date : <date of request>
x-ms-version : <version as specified by user>
- Shared Key
- Add HTTPS authorization header:
Authorization : SharedKey <requester account specified by user>:<signature specified by user>
`
- Shared Access Signatures
- Add query parameters to URL for operation to list blobs in a container
https://<owner account specified by user>.blob.core.windows.net/<container specified by user>/<blob specified by user>?<query parameters>
https://yade.blob.core.windows.net/yade/test.txt?restype=container&comp=list
- Add SAS token to URL:
https://<blob storage owner account specified by user>.blob.core.windows.net/<container specified by user>/<blob specified by user>?<SAS token specified by user>
- Adjust the YADE XSD Schema to allow configuration of any number of HTTP headers
- at a global level
- per profile
- on a per job basis by job/order parameters and when using the YADE standalone client by command line parameters.
- Delimitation
- The implementation is about integrating the above configuration items into HTTPS headers and query parameters.
- The implementation is not about developing a new data provider that is subject to a separate feature request.
References
Jira |
---|
server | SOS JIRA |
---|
columns | key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution |
---|
serverId | 6dc67751-9d67-34cd-985b-194a8cdc9602 |
---|
key | YADE-561 |
---|
|
Resources