Skip to content
This repository has been archived by the owner on Jan 17, 2022. It is now read-only.
hojo edited this page Sep 26, 2012 · 1 revision

ID API

This API allows a caller to specify objects by their IDs.

Standard storage operations

ID API standard storage functions

  • dpl_post_id() Adds objects to storage using the HTTP POST method, and returns object IDs.
  • dpl_post_buffered_id() Adds objects to storage from a buffered stream using the POST method, and returns object IDs.
  • dpl_put_id() Adds objects to storage using the PUT method, and returns object IDs.
  • dpl_put_buffered_id() Adds objects to storage from a buffered stream using the PUT method, and returns object IDs.
  • dpl_get_id() Returns objects from storage.
  • dpl_get_range_id() Returns data chunks containing subportions of objects from storage.
  • dpl_get_buffered_id() Returns objects from storage in a buffered stream.
  • dpl_head_id() Returns user metadata.
  • dpl_head_all_id() Returns user and system metadata.
  • dpl_head_sysmd_id() Returns system metadata.
  • dpl_delete_id() Marks objects in storage for deletion.
  • dpl_copy_id() Copies objects from one storage location to another.
  • dpl_gen_id_from_oid() Generates unique keys from arbitrary large integers.

ID API standard storage input parameters

  • ctx The droplet context.
  • bucket Container for objects in storage.
  • subresource Component of an object in storage.
  • object_type Context type of an object in storage.
  • metadata Metadata of an object in storage.
  • sysmd System metadata of a stored object.
  • data_buf The object body.
  • data_len The object length.
  • query_params Additional variables passed through POST queries.
  • id Arbitrary large integer uniquely identifying an object.
  • condition Condition for copying an object from one bucket to another.
  • start Start of a range for a data chunk.
  • end End of a range for a data chunk.
  • header_func Callback function for every header found in the HTTP response.
  • buffer_func Callback function for every chunk of data found in the HTTP response.
  • cb_arg User-defined callback argument (closure).
  • src_bucket Source bucket for object copy operations.
  • src_id Source ID for object copy operations.
  • src_subresource Source name for component included in object copy operations.
  • dst_bucket Destination bucket for object copy operations.
  • dst_id Destination ID for object copy operations.
  • dst_subresource Destination name for component included in object copy operations.
  • metadata_directive Header describing how to process metadata for copy operations.
  • oid Arbitrary large integer.
  • storage_class Class of service indicating the number of replicas of an object in storage.

ID API standard storage output parameters

  • resource_idp Returns ID of the object.
  • connp Returns connection handles. Caller must free resource after use by calling dpl_conn_release() or dpl_conn_terminate().
  • data_bufp Returns object body. Caller must release resource after use by calling dpl_free().
  • data_lenp Returns object length.
  • metadatap Returns object metadata. Caller must release resource after use by calling dpl_dict_free().

ID API standard storage prototypes

dpl_status_t 
dpl_post_id(dpl_ctx_t *ctx, 
            char *bucket, 
            char *subresource, 
            dpl_ftype_t object_type, 
            dpl_dict_t *metadata, 
            dpl_sysmd_t *sysmd, 
            char *data_buf, 
            unsigned int data_len, 
            dpl_dict_t *query_params, 
            char **resource_idp);

dpl_status_t 
dpl_post_buffered_id(dpl_ctx_t *ctx, 
                     char *bucket, 
                     char *subresource, 
                     dpl_ftype_t object_type, 
                     dpl_dict_t *metadata, 
                     dpl_sysmd_t *sysmd, 
                     unsigned int data_len, 
                     dpl_dict_t *query_params, 
                     dpl_conn_t **connp);

dpl_status_t 
dpl_put_id(dpl_ctx_t *ctx, 
           char *bucket, 
           char *id, 
           char *subresource, 
           dpl_ftype_t object_type, 
           dpl_dict_t *metadata, 
           dpl_sysmd_t *sysmd, 
           char *data_buf, 
           unsigned int data_len);

dpl_status_t 
dpl_put_buffered_id(dpl_ctx_t *ctx, 
                    char *bucket, 
                    char *id, 
                    char *subresource, 
                    dpl_ftype_t object_type, 
                    dpl_dict_t *metadata, 
                    dpl_sysmd_t *sysmd, 
                    unsigned int data_len, 
                    dpl_conn_t **connp);

dpl_status_t 
dpl_get_id(dpl_ctx_t *ctx, 
           char *bucket, 
           char *id, 
           char *subresource, 
           dpl_ftype_t object_type, 
           dpl_condition_t *condition, 
           char **data_bufp, 
           unsigned int *data_lenp, 
           dpl_dict_t **metadatap);

dpl_status_t 
dpl_get_range_id(dpl_ctx_t *ctx, 
                 char *bucket, 
                 char *id, 
                 char *subresource, 
                 dpl_ftype_t object_type, 
                 dpl_condition_t *condition, 
                 int start, 
                 int end, 
                 char **data_bufp, 
                 unsigned int *data_lenp, 
                 dpl_dict_t **metadatap);

dpl_status_t 
dpl_get_buffered_id(dpl_ctx_t *ctx, 
                    char *bucket, 
                    char *id, 
                    char *subresource, 
                    dpl_ftype_t object_type, 
                    dpl_condition_t *condition, 
                    dpl_header_func_t header_func, 
                    dpl_buffer_func_t buffer_func, 
                    void *cb_arg);

dpl_status_t dpl_head_id(dpl_ctx_t *ctx, 
                          char *bucket, 
                          char *id, 
                          char *subresource, 
                          dpl_condition_t *condition, 
                          dpl_dict_t **metadatap);

dpl_status_t 
dpl_head_all_id(dpl_ctx_t *ctx, 
                char *bucket, 
                char *id, 
                char *subresource, 
                dpl_condition_t *condition, 
                dpl_dict_t **metadatap);

dpl_status_t 
dpl_head_sysmd_id(dpl_ctx_t *ctx, 
                char *bucket, 
                char *id, 
                char *subresource, 
                dpl_condition_t *condition, 
                dpl_sysmd_t *sysmdp, 
                dpl_dict_t **metadatap);

dpl_status_t 
dpl_delete_id(dpl_ctx_t *ctx, 
              char *bucket, 
              char *id, 
              char *subresource);

dpl_status_t 
dpl_copy_id(dpl_ctx_t *ctx, 
            char *src_bucket, 
            char *src_id, 
            char *src_subresource, 
            char *dst_bucket, 
            char *dst_id, 
            char *dst_subresource, 
            dpl_ftype_t object_type, 
            dpl_metadata_directive_t metadata_directive, 
            dpl_dict_t *metadata, 
            dpl_sysmd_t *sysmd, 
            dpl_condition_t *condition);

dpl_status_t 
dpl_gen_id_from_oid(dpl_ctx_t *ctx, 
                    uint64_t oid, 
                    dpl_storage_class_t storage_class, 
                    char **resource_idp);

SRWS-specific operations

To uniquely identify an object, its location and how many copies of that object should exist, a 20-byte (160 bits) key is divided into predefined parts. At the highest level, there are three parts:

  • Hashing/Dispersion: the first 24 bits of all keys is an MD5 hash of the payload. This makes sure that adjacent objects are dispersed across all storage nodes.
  • Payload: the next 128 bits uniquely identify an object. The payload includes 64 bits for the object ID and 32 bits for the volume ID.
  • Class of service: 8 bits, 4 bits for the class itself, and 4 bits to specify the replica number of a particular key. This ensures that a replica of an object cannot clash with the key of another object.

The SRWS-specific functions of the ID API are included to permit key generation required by the storage system.

ID API SRWS-specific functions

  • dpl_srws_gen_key() Generates keys for objects without the class-of-service bits.
  • dpl_srws_set_class() Sets the class of service (number of replicas) for objects in storage.

ID API SRWS-specific parameters

  • id (IN/OUT) Unique key as computed for an object in storage. The key must be pre-allocated in a BN_new() call.
  • oid Arbitrary 64-bit large integer unique for given volid.
  • volid 32-bit volume ID.
  • serviceid 8-bit ID for the application interfacing with the storage system.
  • specific 24-bit ID specific to the application, such as a mailbox ID for a webmail application.
  • class Class of service indicating the number of replicas of an object in storage.

ID API SRWS-specific prototypes

dpl_status_t 
dpl_srws_gen_key(BIGNUM *id, 
                 uint64_t oid, 
                 uint32_t volid, 
                 uint8_t serviceid, 
                 uint32_t specific);

dpl_status_t 
dpl_srws_set_class(BIGNUM *id, 
                   int class);