Scope

  • YADE should support file transfer operations with Azure Blob Storage.
  • Technically Azure file transfer operations preferably make use of the HTTPS protocol and a number of query parameters and headers. The implementation with YADE therefore is fairly straightforward.
  • The sticking point is about what authentication methods should be supported and how they would be integrated:
    • If blob containers are made publicly available and require no authentication then the YADE supports this out-of-the-box.
    • If Azure authentication methods are applied then different implementation strategies are to be considered. Azure supports a range of authentication methods - we picked two of them for this proposal:
  • There are impacts on job operation depending on the authentication method that is preferred by an organization.

Authentication Methods

Find a preliminary comparison of authentication methods:

AreaShared KeyShared Access Signature
Scope
  • uses a single Shared Key for access at container level or at blob level
  • allows a single SAS token to be used for access to any blobs in a container
  • allows individual SAS tokens to be used per blob and per file transfer operation 
Access Duration
  • enables unlimited access
  • makes use of expiration dates
Permissions
  • enables full access at container level or at blob level for any file transfer operations
  • offers permissions (read, write, delete, list) per resource type (container, object) and service (blob, queue, table, file)
  • allows a single SAS token to be used that includes any permissions and file transfer operations (get, put, list)
    • allows individual SAS tokens to be used specifying permissions per blob and per file transfer operation
Implementation
  • A signature is created by hashing blob resource parameters and by specifying the blob to be accessed.
  • Use of the HTTPS Authorization header for signature.
  • A signature is created by hashing blob resource parameters, the permissions that are assigned and the expiration date.
  • An SAS token is generated that specifies both the plain text values of the signature and the hashed value of the signature.
  • Use of HTTPS query parameters for SAS tokens. Such tokens specify the blob resource to be accessed, the permissions that are assigned, expiration period and signature.


  • From the above comparison Shared Access Signatures are superior concerning more fine-grained access to resources.
  • At the same time Shared Key authentication suggests ease of use as a single Shared Key can be used to authenticate any file transfer operations for an unlimited duration.
  • Technically a single Shared Access Signature can be applied in a similar way allowing any file transfer operation on any blobs in a given container for a longer period, e.g. the next 100 years.

Operation

  • Considering use of authentication methods with file transfer jobs there is a final statement from SOS: implementation of any authentication method is out of scope of the JobScheduler product. 
    • Implementation of authentication methods is within the responsibility of the organization that uses JobScheduler. Reasons for this include that
      • users should not trust any 3rd party implementation of authentication methods,
      • a number of authentication methods are offered by Azure that allow an individual choice according to security requirements of an organization,
      • there is ongoing development with Azure about improvement of authentication methods.
    • Implementation of file transfer operations for Azure blob storage is within the scope of the YADE.
  • Technically this offers the following options:
    • An organization can create a single Shared Key or Shared Access Signature that represents a constant value that is made available to all YADE file transfer jobs.
    • An organization can create individual Shared Keys or Shared Access Signatures that are used with groups of YADE file transfer jobs or with individual jobs.

Implementation

User Tasks: Manage Shared Keys and Shared Access Signatures

Creating Shared Keys and Shared Access Signatures

SOS Tasks: YADE Integration

Implementation Strategies

  • When using a single Shared Key or Shared Access Signature for all YADE file transfer jobs then
    • this could be added as a constant value to the JobScheduler configuration, e.g. with a parameter to the ./config/scheduler.xml file.
    • this could be added to a Credential Store that would be accessed by any YADE file transfer jobs.
  • When using individual Shared Keys or Shared Access Signatures per YADE file transfer job then a Monitor should be implemented that is assigned on a per job basis and that can be parameterized to use the key or signature specified by a job parameter or order parameter.
  • Should Shared Keys or Shared Access Signatures have to be renewed on a regular basis then this can be performed by a job, e.g. running daily, that is implemented by a user of JobScheduler.

YADE Changes

  • Accept for both Shared Keys and Shared Access Signatures the respective configuration items:
    • For both authentication methods
      • Add to URL
        • Resources
        • container resource
        • blob resource
        • Example
          • https://<owner account specified by user>.blob.core.windows.net/<container specified by user>/<blob specified by user>
          • https://yade.blob.core.windows.net/yade/test.txt
      • Add HTTPS headers:
        • x-ms-blob-type : BlockBlob
        • x-ms-date : <date of request>
        • x-ms-version : <version as specified by user>
    • Shared Key
      • Add HTTPS authorization header:
        • Authorization : SharedKey <requester account specified by user>:<signature specified by user>`
    • Shared Access Signatures
      • Add query parameters to URL for operation to list blobs in a container
        • https://<owner account specified by user>.blob.core.windows.net/<container specified by user>/<blob specified by user>?<query parameters>
        • https://yade.blob.core.windows.net/yade/test.txt?restype=container&comp=list
      • Add SAS token to URL:
        • https://<blob storage owner account specified by user>.blob.core.windows.net/<container specified by user>/<blob specified by user>?<SAS token specified by user>
  • Adjust the YADE XSD Schema to allow configuration of any number of HTTP headers
    • at a global level 
    • per profile
    • on a per job basis by job/order parameters and when using the YADE standalone client by command line parameters.
  •  Delimitation
    • The implementation is about integrating the above configuration items into HTTPS headers and query parameters.
    • The implementation is not about developing a new data provider that is subject to a separate feature request.

References

YADE-561 - Getting issue details... STATUS

Resources

  • No labels