Citation
+Edmondson M (????). googleCloudStorageR: R Interface with Google Cloud Storage. -R package version . +R package version 0.5.0.
@Manual{,
title = {googleCloudStorageR: R Interface with Google Cloud Storage},
author = {Mark Edmondson},
- note = {R package version },
+ note = {R package version 0.5.0},
}
Authors
@@ -117,7 +133,7 @@Authors
-
-
Mark Edmondson. Author, maintainer. +
Mark Edmondson. Author, maintainer.
Authors
-
@@ -74,52 +83,76 @@
- Click ‘Create a new Client ID’, and choose “Installed Application”. +
- Download the client ID JSON. +
-
+
Set the client ID via
+ +googleAuthR::gar_set_client():
+ - Click ‘Create a new Client ID’, and choose “Web Application”. +
- Download the client ID JSON. +
- Add the URL of where your Shiny app will run, with no port number. e.g.
https://mark.shinyapps.io/searchConsoleRDemo/+
+ - And/Or also put in localhost or 127.0.0.1 with a port number for local testing. Remember the port number you use as you will need it later to launch the app e.g.
http://127.0.0.1:1221+
+ -
+
Set the web client ID via
+ +googleAuthR::gar_set_client():
+ - To run the app locally specifying the port number you used in step 4 e.g.
shiny::runApp(port=1221)or set a shiny option to default to it:options(shiny.port = 1221)and launch via theRunAppbutton in RStudio.
+ Running on your Shiny Server will work only for the URL from step 3.
+- Click on “APIs” +
- Select and activate the Cloud Storage JSON API +
- After loading the package via
library(googleCloudStorage), it will look to see if"https://www.googleapis.com/auth/devstorage.full_control"is set ingetOption("googleAuthR.scopes.selected")and set it if it is not, adding to the existing scopes.
+
+ -
+
Alternativly, set the
+ +googleAuthRoption for Google Cloud storage scope after the library has been loaded but before authentication.
+ - Click ‘Create a new Client ID’, and choose “Installed Application”. -
- Note your Client ID and secret. -
-
-
Add them by modifying your .Renviron file, or under the following entries:
--Sys.setenv("GCS_CLIENT_ID" = "mykey", - "GCS_CLIENT_SECRET" = "mysecretkey")
- -
-
Alternatively, modify these options after googleAuthR has been loaded:
--options("googleAuthR.client_id" = "YOUR_CLIENT_ID") -options("googleAuthR.client_secret" = "YOUR_CLIENT_SECRET")
- - Click ‘Create a new Client ID’, and choose “Web Application”. -
- Note your Client ID and secret. -
- Add the URL of where your Shiny app will run, with no port number. e.g.
https://mark.shinyapps.io/searchConsoleRDemo/-
- - And/Or also put in localhost or 127.0.0.1 with a port number for local testing. Remember the port number you use as you will need it later to launch the app e.g.
http://127.0.0.1:1221-
- -
-
Add them by modifying your .Renviron file, or under the following entries:
--Sys.setenv("GCS_WEB_CLIENT_ID" = "mykey", - "GCS_WEB_CLIENT_SECRET" = "mysecretkey")
- -
-
Alternatively, in your Shiny script modify these options:
--options("googleAuthR.webapp.client_id" = "YOUR_CLIENT_ID") -options("googleAuthR.webapp.client_secret" = "YOUR_CLIENT_SECRET")
- - To run the app locally specifying the port number you used in step 4 e.g.
shiny::runApp(port=1221)or set a shiny option to default to it:options(shiny.port = 1221)and launch via theRunAppbutton in RStudio.
- Running on your Shiny Server will work only for the URL from step 3.
-- Click on “APIs” -
- Select and activate the Cloud Storage JSON API -
- After loading the package via
library(googleCloudStorage), it will look to see if"https://www.googleapis.com/auth/devstorage.full_control"is set ingetOption("googleAuthR.scopes.selected")and set it if it is not, adding to the existing scopes.
- -
-
Alternativly, set the
-googleAuthRoption for Google Cloud storage scope after the library has been loaded but before authentication.-options(googleAuthR.scopes.selected = "https://www.googleapis.com/auth/devstorage.full_control")
-
R library for interacting with the Google Cloud Storage JSON API (api docs).
Setup
Google Cloud Storage charges you for storage (prices here).
You can use your own Google Project with a credit card added to create buckets, where the charges will apply. This can be done in the Google API Console
++Configuring your own Google Project
+The instructions below are for when you visit the Google API console (https://console.developers.google.com/apis/)
+For local use
+-
+
+For Shiny use
+-
+
+Activate API
+-
+
Setting environment variables
By default, all cloudyr packages look for the access key ID and secret access key in environment variables. You can also use this to specify a default bucket, and auto-authentication upon attaching the library. For example:
-Sys.setenv("GCS_CLIENT_ID" = "mykey",
- "GCS_CLIENT_SECRET" = "mysecretkey",
- "GCS_WEB_CLIENT_ID" = "my-shiny-key",
- "GCS_WEB_CLIENT_SECRET" = "my-shiny-secret-key",
- "GCS_DEFAULT_BUCKET" = "my-default-bucket",
- "GCS_AUTH_FILE" = "/fullpath/to/service-auth.json")Sys.setenv("GCS_DEFAULT_BUCKET" = "my-default-bucket",
+ "GCS_AUTH_FILE" = "/fullpath/to/service-auth.json")These can alternatively be set on the command line or via an Renviron.site or .Renviron file (see here for instructions).
-Authentication
-Authentication can be carried out each session via gcs_auth. The first time you run this you will be sent to a Google login prompt in your browser to allow the googleCloudStorageR project access (or the Google project you configure).
Once authenticated a file named .httr-oauth is saved to your working directory. On subsequent authentication this file will hold your authentication details, and you won’t need to go via the browser. Deleting this file, or setting new_user=TRUE will start the authentication flow again.
library(googleCloudStorageR)
-## first time this will send you to the browser to authenticate
-gcs_auth()
-
-## to authenticate with a fresh user, delete .httr-oauth or run with new_user=TRUE
-gcs_auth(new_user = TRUE)
-
-...call functions...etc...Each new R session will need to run gcs_auth() to authenticate future API calls.
Auto-authentication
-Alternatively, you can specify the location of a service account JSON file taken from your Google Project, or the location of a previously created .httr-oauth token in a system environment:
Sys.setenv("GCS_AUTH_FILE" = "/fullpath/to/auth.json")This file will then used for authentication via gcs_auth() when you load the library:
## GCS_AUTH_FILE set so auto-authentication
-library(googleCloudStorageR)
-
-## no need for gcs_auth()
-gcs_get_bucket("your-bucket")To authenticate, you specify the location of a service account JSON file taken from your Google Project:
+
Setting a default Bucket
To avoid specifying the bucket in the functions below, you can set the name of your default bucket via environmental variables or via the function gcs_global_bucket(). See the Setting environment variables section below for more details.
## set bucket via environment
-Sys.setenv("GCS_DEFAULT_BUCKET" = "my-default-bucket")
-
-library(googleCloudStorageR)
-
-## optional, if you haven't set environment argument GCS_AUTH_FILE
-## gcs_auth()
-
-## check what the default bucket is
-gcs_get_global_bucket()
-[1] "my-default-bucket"
-
-## you can also set a default bucket after loading the library for that session
-gcs_global_bucket("your-default-bucket-2")
-gcs_get_global_bucket()
-[1] "my-default-bucket-2"## set bucket via environment
+Sys.setenv("GCS_DEFAULT_BUCKET" = "my-default-bucket")
+
+library(googleCloudStorageR)
+
+## check what the default bucket is
+gcs_get_global_bucket()
+[1] "my-default-bucket"
+
+## you can also set a default bucket after loading the library for that session
+gcs_global_bucket("your-default-bucket-2")
+gcs_get_global_bucket()
+[1] "my-default-bucket-2"Downloading objects from Google Cloud storage
Once you have a Google project and created a bucket with an object in it, you can download it as below:
-library(googleCloudStorageR)
-
-## optional, if you haven't set environment argument GCS_AUTH_FILE
-## gcs_auth()
-
-## get your project name from the API console
-proj <- "your-project"
-
-## get bucket info
-buckets <- gcs_list_buckets(proj)
-bucket <- "your-bucket"
-bucket_info <- gcs_get_bucket(bucket)
-bucket_info
-
-==Google Cloud Storage Bucket==
-Bucket: your-bucket
-Project Number: 1123123123
-Location: EU
-Class: STANDARD
-Created: 2016-04-28 11:39:06
-Updated: 2016-04-28 11:39:06
-Meta-generation: 1
-eTag: Cxx=
-
-
-## get object info in the default bucket
-objects <- gcs_list_objects()
-
-## save directly to an R object (warning, don't run out of RAM if its a big object)
-## the download type is guessed into an appropriate R object
-parsed_download <- gcs_get_object(objects$name[[1]])
-
-## if you want to do your own parsing, set parseObject to FALSE
-## use httr::content() to parse afterwards
-raw_download <- gcs_get_object(objects$name[[1]],
- parseObject = FALSE)
-
-## save directly to a file in your working directory
-## parseObject has no effect, it is a httr::content(req, "raw") download
-gcs_get_object(objects$name[[1]], saveToDisk = "csv_downloaded.csv")library(googleCloudStorageR)
+
+## get your project name from the API console
+proj <- "your-project"
+
+## get bucket info
+buckets <- gcs_list_buckets(proj)
+bucket <- "your-bucket"
+bucket_info <- gcs_get_bucket(bucket)
+bucket_info
+
+==Google Cloud Storage Bucket==
+Bucket: your-bucket
+Project Number: 1123123123
+Location: EU
+Class: STANDARD
+Created: 2016-04-28 11:39:06
+Updated: 2016-04-28 11:39:06
+Meta-generation: 1
+eTag: Cxx=
+
+
+## get object info in the default bucket
+objects <- gcs_list_objects()
+
+## save directly to an R object (warning, don't run out of RAM if its a big object)
+## the download type is guessed into an appropriate R object
+parsed_download <- gcs_get_object(objects$name[[1]])
+
+## if you want to do your own parsing, set parseObject to FALSE
+## use httr::content() to parse afterwards
+raw_download <- gcs_get_object(objects$name[[1]],
+ parseObject = FALSE)
+
+## save directly to a file in your working directory
+## parseObject has no effect, it is a httr::content(req, "raw") download
+gcs_get_object(objects$name[[1]], saveToDisk = "csv_downloaded.csv")Uploading objects < 5MB
-Objects can be uploaded via files saved to disk, or passed in directly if they are data frames or list type R objects. By default, data frames will be converted to CSV via write.csv(), lists to JSON via jsonlite::toJSON.
Objects can be uploaded via files saved to disk, or passed in directly if they are data frames or list type R objects. By default, data frames will be converted to CSV via write.csv(), lists to JSON via jsonlite::toJSON.
If you want to use other functions for transforming R objects, for example setting row.names = FALSE or using write.csv2, pass the function through object_function
## upload a file - type will be guessed from file extension or supply type
-write.csv(mtcars, file = filename)
-gcs_upload(filename)
-
-## upload an R data.frame directly - will be converted to csv via write.csv
-gcs_upload(mtcars)
-
-## upload an R list - will be converted to json via jsonlite::toJSON
-gcs_upload(list(a = 1, b = 3, c = list(d = 2, e = 5)))
-
-## upload an R data.frame directly, with a custom function
-## function should have arguments 'input' and 'output'
-## safest to supply type too
-f <- function(input, output) write.csv(input, row.names = FALSE, file = output)
-
-gcs_upload(mtcars,
- object_function = f,
- type = "text/csv")## upload a file - type will be guessed from file extension or supply type
+write.csv(mtcars, file = filename)
+gcs_upload(filename)
+
+## upload an R data.frame directly - will be converted to csv via write.csv
+gcs_upload(mtcars)
+
+## upload an R list - will be converted to json via jsonlite::toJSON
+gcs_upload(list(a = 1, b = 3, c = list(d = 2, e = 5)))
+
+## upload an R data.frame directly, with a custom function
+## function should have arguments 'input' and 'output'
+## safest to supply type too
+f <- function(input, output) write.csv(input, row.names = FALSE, file = output)
+
+gcs_upload(mtcars,
+ object_function = f,
+ type = "text/csv")Upload metadata
You can pass metadata with an object via the function gcs_metadata_object().
the name you pass to the metadata object will override the name if it is also set elsewhere.
-meta <- gcs_metadata_object("mtcars.csv",
- metadata = list(custom1 = 2,
- custom_key = 'dfsdfsdfsfs))
-
-gcs_upload(mtcars, object_metadata = meta)
@@ -233,208 +260,208 @@
If the file/object is under 5MB, simple uploads are used.
For files > 5MB, resumable uploads are used. This allows you to upload up to 5TB.
If you get an interrupted connection when uploading, gcs_upload will retry 3 times, if it fails it will return a Retry object, that you can try again later from where the upload stopped. Call this via gcs_retry_upload
## write a big object to a file
-big_file <- "big_filename.csv"
-write.csv(big_object, file = big_file)
-
-## attempt upload
-upload_try <- gcs_upload(big_file)
-
-## if successful, upload_try is an object metadata object
-upload_try
-==Google Cloud Storage Object==
-Name: "big_filename.csv"
-Size: 8.5 Gb
-Media URL https://www.googleapis.com/download/storage/v1/b/xxxx
-Bucket: your-bucket
-ID: your-bucket/"test.pdf"/xxxx
-MD5 Hash: rshao1nxxxxxY68JZQ==
-Class: STANDARD
-Created: 2016-08-12 17:33:05
-Updated: 2016-08-12 17:33:05
-Generation: 1471023185977000
-Meta Generation: 1
-eTag: CKi90xxxxxEAE=
-crc32c: j4i1sQ==
-
-
-## if unsuccessful after 3 retries, upload_try is a Retry object
-==Google Cloud Storage Upload Retry Object==
-File Location: big_filename.csv
-Retry Upload URL: http://xxxx
-Created: 2016-08-12 17:33:05
-Type: csv
-File Size: 8.5 Gb
-Upload Byte: 4343
-Upload remaining: 8.1 Gb
-
-## you can retry to upload the remaining data using gcs_retry_upload()
-try2 <- gcs_retry_upload(upload_try)## write a big object to a file
+big_file <- "big_filename.csv"
+write.csv(big_object, file = big_file)
+
+## attempt upload
+upload_try <- gcs_upload(big_file)
+
+## if successful, upload_try is an object metadata object
+upload_try
+==Google Cloud Storage Object==
+Name: "big_filename.csv"
+Size: 8.5 Gb
+Media URL https://www.googleapis.com/download/storage/v1/b/xxxx
+Bucket: your-bucket
+ID: your-bucket/"test.pdf"/xxxx
+MD5 Hash: rshao1nxxxxxY68JZQ==
+Class: STANDARD
+Created: 2016-08-12 17:33:05
+Updated: 2016-08-12 17:33:05
+Generation: 1471023185977000
+Meta Generation: 1
+eTag: CKi90xxxxxEAE=
+crc32c: j4i1sQ==
+
+
+## if unsuccessful after 3 retries, upload_try is a Retry object
+==Google Cloud Storage Upload Retry Object==
+File Location: big_filename.csv
+Retry Upload URL: http://xxxx
+Created: 2016-08-12 17:33:05
+Type: csv
+File Size: 8.5 Gb
+Upload Byte: 4343
+Upload remaining: 8.1 Gb
+
+## you can retry to upload the remaining data using gcs_retry_upload()
+try2 <- gcs_retry_upload(upload_try)Updating user access to objects
You can change who can access objects via gcs_update_acl to one of READER or OWNER, on a user, group, domain, project or public for all users or authenticated users.
By default you are “OWNER” of all the objects and buckets you upload and create.
-## update access of object to READER for all public
-gcs_update_object_acl("your-object.csv", entity_type = "allUsers")
-
-## update access of object for user joe@blogs.com to OWNER
-gcs_update_acl("your-object.csv",
- entity = "joe@blogs.com",
- role = "OWNER")
-
-## update access of object for googlegroup users to READER
-gcs_update_object_acl("your-object.csv",
- entity = "my-group@googlegroups.com",
- entity_type = "group")
-
-## update access of object for all users to OWNER on your Google Apps domain
-gcs_update_object_acl("your-object.csv",
- entity = "yourdomain.com",
- entity_type = "domain",
- role = OWNER)## update access of object to READER for all public
+gcs_update_object_acl("your-object.csv", entity_type = "allUsers")
+
+## update access of object for user joe@blogs.com to OWNER
+gcs_update_acl("your-object.csv",
+ entity = "joe@blogs.com",
+ role = "OWNER")
+
+## update access of object for googlegroup users to READER
+gcs_update_object_acl("your-object.csv",
+ entity = "my-group@googlegroups.com",
+ entity_type = "group")
+
+## update access of object for all users to OWNER on your Google Apps domain
+gcs_update_object_acl("your-object.csv",
+ entity = "yourdomain.com",
+ entity_type = "domain",
+ role = OWNER)Deleting an object
Delete an object by passing its name (and bucket if not default)
-## returns TRUE is successful, a 404 error if not found
-gcs_delete_object("your-object.csv")Viewing current access level to objects
Use gcs_get_object_acl() to see what the current access is for an entity + entity_type.
## default entity_type is user
-acl <- gcs_get_object_acl("your-object.csv",
- entity = "joe@blogs.com")
-acl$role
-[1] "OWNER"
-
-## for allUsers and allAuthenticated users, you don't need to supply entity
-acl <- gcs_get_object_acl("your-object.csv",
- entity_type = "allUsers")
-acl$role
-[1] "READER"## default entity_type is user
+acl <- gcs_get_object_acl("your-object.csv",
+ entity = "joe@blogs.com")
+acl$role
+[1] "OWNER"
+
+## for allUsers and allAuthenticated users, you don't need to supply entity
+acl <- gcs_get_object_acl("your-object.csv",
+ entity_type = "allUsers")
+acl$role
+[1] "READER"Creating download links
Once a user (or group or the public) has access, they can reach that object via a download link generated by the function gcs_download_url
download_url <- gcs_download_url("your-object.csv")
-download_url
-[1] "https://storage.cloud.google.com/your-project/your-object.csv"Signed URLs
You can create temporary links for users who may not have a Google account, but still need to be private. This is achieved using the gcs_signed_url function, which you pass a meta object too.
obj <- gcs_get_object("your_file", meta = TRUE)
-
-signed <- gcs_signed_url(obj)The default is for the link to be accessible for an hour, but you can alter that:
-## a link that will expire in 24 hours (86400 seconds) from now.
-24hours_signed <- gcs_signed_url(obj, expiration_ts = Sys.time() + 86400)R Session helpers
-Versions of save.image(), save() and load() are implemented called gcs_save_image(), gcs_save() and gcs_load(). These functions save and load the global R session to the cloud.
## save the current R session including all objects
-gcs_save_image()
-
-### wipe environment
-rm(list = ls())
-
-## load up environment again
-gcs_load()Versions of save.image(), save() and load() are implemented called gcs_save_image(), gcs_save() and gcs_load(). These functions save and load the global R session to the cloud.
## save the current R session including all objects
+gcs_save_image()
+
+### wipe environment
+rm(list = ls())
+
+## load up environment again
+gcs_load()Save specific objects:
-cc <- 3
-d <- "test1"
-gcs_save("cc","d", file = "gcs_save_test.RData")
-
-## remove the objects saved in cloud from local environment
-rm(cc,d)
-
-## load them back in from GCS
-gcs_load(file = "gcs_save_test.RData")
-cc == 3
-[1] TRUE
-d == "test1"
-[1] TRUEcc <- 3
+d <- "test1"
+gcs_save("cc","d", file = "gcs_save_test.RData")
+
+## remove the objects saved in cloud from local environment
+rm(cc,d)
+
+## load them back in from GCS
+gcs_load(file = "gcs_save_test.RData")
+cc == 3
+[1] TRUE
+d == "test1"
+[1] TRUEYou can also upload .R code files and source them directly using gcs_source:
## make a R source file and upload it
-cat("x <- 'hello world!'\nx", file = "example.R")
-gcs_upload("example.R", name = "example.R")
-
-## source the file to run its code
-gcs_source("example.R")
-
-## the code from the upload file has run
-x
-[1] "hello world!"Uploading via a Shiny app
The library is also compatible with Shiny authentication flows, so you can create Shiny apps that lets users log in and upload their own data.
An example of that is shown below:
-library("shiny")
-library("googleAuthR")
-library("googleCloudStorageR")
-options(googleAuthR.scopes.selected = "https://www.googleapis.com/auth/devstorage.full_control")
-## optional, if you want to use your own Google project
-# options("googleAuthR.client_id" = "YOUR_CLIENT_ID")
-# options("googleAuthR.client_secret" = "YOUR_CLIENT_SECRET")
-
-## you need to start Shiny app on port 1221
-## as thats what the default googleAuthR project expects for OAuth2 authentication
-
-## options(shiny.port = 1221)
-## print(source('shiny_test.R')$value) or push the "Run App" button in RStudio
-
-shinyApp(
- ui = shinyUI(
- fluidPage(
- googleAuthR::googleAuthUI("login"),
- fileInput("picture", "picture"),
- textInput("filename", label = "Name on Google Cloud Storage",value = "myObject"),
- actionButton("submit", "submit"),
- textOutput("meta_file")
- )
- ),
- server = shinyServer(function(input, output, session){
-
- access_token <- shiny::callModule(googleAuth, "login")
-
- meta <- eventReactive(input$submit, {
-
- message("Uploading to Google Cloud Storage")
-
- # from googleCloudStorageR
- with_shiny(gcs_upload,
- file = input$picture$datapath,
- # enter your bucket name here
- bucket = "gogauth-test",
- type = input$picture$type,
- name = input$filename,
- shiny_access_token = access_token())
-
- })
-
- output$meta_file <- renderText({
-
- req(meta())
-
- str(meta())
-
- paste("Uploaded: ", meta()$name)
-
- })
-
- })
-)library("shiny")
+library("googleAuthR")
+library("googleCloudStorageR")
+options(googleAuthR.scopes.selected = "https://www.googleapis.com/auth/devstorage.full_control")
+## optional, if you want to use your own Google project
+# options("googleAuthR.client_id" = "YOUR_CLIENT_ID")
+# options("googleAuthR.client_secret" = "YOUR_CLIENT_SECRET")
+
+## you need to start Shiny app on port 1221
+## as thats what the default googleAuthR project expects for OAuth2 authentication
+
+## options(shiny.port = 1221)
+## print(source('shiny_test.R')$value) or push the "Run App" button in RStudio
+
+shinyApp(
+ ui = shinyUI(
+ fluidPage(
+ googleAuthR::googleAuthUI("login"),
+ fileInput("picture", "picture"),
+ textInput("filename", label = "Name on Google Cloud Storage",value = "myObject"),
+ actionButton("submit", "submit"),
+ textOutput("meta_file")
+ )
+ ),
+ server = shinyServer(function(input, output, session){
+
+ access_token <- shiny::callModule(googleAuth, "login")
+
+ meta <- eventReactive(input$submit, {
+
+ message("Uploading to Google Cloud Storage")
+
+ # from googleCloudStorageR
+ with_shiny(gcs_upload,
+ file = input$picture$datapath,
+ # enter your bucket name here
+ bucket = "gogauth-test",
+ type = input$picture$type,
+ name = input$filename,
+ shiny_access_token = access_token())
+
+ })
+
+ output$meta_file <- renderText({
+
+ req(meta())
+
+ str(meta())
+
+ paste("Uploaded: ", meta()$name)
+
+ })
+
+ })
+)
@@ -450,129 +477,72 @@
Object administration
Object administration
You can get meta data about an object by passing meta=TRUE to gcs_get_object
gcs_get_object("your-object", "your-bucket", meta = TRUE)-Explanation of Google Project access
-googleCloudStorageR has its own Google project which is used to call the Google Cloud Storage API, but does not have access to the objects or buckets in your Google Project unless you give permission for the library to access your own buckets during the OAuth2 authentication process.
No other user, including the owner of the Google Cloud Storage API project has access unless you have given them access, but you may want to change to use your own Google Project (that could or could not be the same as the one that holds your buckets).
--Configuring your own Google Project
-The instructions below are for when you visit the Google API console (https://console.developers.google.com/apis/)
-For local use
--
-
-For Shiny use
--
-
-Activate API
--
-
Installation
This package is on CRAN:
-# latest stable version
-install.packages("googleCloudStorageR")Or, to pull a potentially unstable version directly from GitHub:
-if(!require("ghit")){
- install.packages("ghit")
-}
-ghit::install_github("cloudyr/googleCloudStorageR")if(!require("ghit")){
+ install.packages("ghit")
+}
+ghit::install_github("cloudyr/googleCloudStorageR")
Links
+ +Change log All releases
+Changelog
+-googleCloudStorageR 0.3.0.9000
++googleCloudStorage 0.5.0 Unreleased +
+-
+
- Refactor authentication to favour json file and fix auth hangs (#87) +
- Prevent error if bucket name is empty string (#90) +
-
+
gcs_upload()will use file extension ofnamein its temporary file (#91)
+ - Add
gcs_copy_object()+
+ - Add
gcs_compose_objects()+
+ - Refactor
gcs_list_objects()to googleAuthR > 0.7gar_api_page()+
+
+googleCloudStorageR 0.4.0 2017-11-17 +
Major changes
@@ -118,12 +151,15 @@gcs_save_all and gcs_load_all which will zip, save/load and upload/download a directory_gcssave.yaml file to control gcs_first/last behaviour-googleCloudStorageR 0.3.0
++googleCloudStorageR 0.3.0 2017-05-27 +
Major changes
@@ -133,9 +169,9 @@gs:// style URLs for object names (#57 - thanks seandavi)gcs_get_bucket() to only expect length 1 character vectors for bucket name. (#60)gcs_object_list (#58 - thanks @G3rtjan)saveToDisk option to gcs_load (#52 - thanks @tomsing1)gcs_get_object() (#63 - thanks @nkeriks)gcs_object_list (#58 - thanks @G3rtjan)saveToDisk option to gcs_load (#52 - thanks @tomsing1)gcs_get_object() (#63 - thanks @nkeriks)prefix and delimiter in gcs_object_list to filter objects listed (#68)-googleCloudStorageR 0.2.0
++googleCloudStorageR 0.2.0 2016-09-11 +
Major changes
@@ -156,7 +193,7 @@gcs_save to store R session data in cloudgcs_load to restore session data stored with gcs_save
options(googleAuthR.rawResponse = TRUE) when using gcs_get_object
+options(googleAuthR.rawResponse = TRUE) when using gcs_get_object
object_name in gcs_get_object etc.Object Object
+ +Object.RdObject Object
+Object(acl = NULL, bucket = NULL, cacheControl = NULL, componentCount = NULL, contentDisposition = NULL, contentEncoding = NULL, contentLanguage = NULL, contentType = NULL, crc32c = NULL, customerEncryption = NULL, etag = NULL, generation = NULL, id = NULL, md5Hash = NULL, mediaLink = NULL, - metadata = NULL, metageneration = NULL, name = NULL, owner = NULL, - selfLink = NULL, size = NULL, storageClass = NULL, timeCreated = NULL, - timeDeleted = NULL, updated = NULL)+ metadata = NULL, metageneration = NULL, name = NULL, + owner = NULL, selfLink = NULL, size = NULL, storageClass = NULL, + timeCreated = NULL, timeDeleted = NULL, updated = NULL) -
Arguments
+Arguments
| new_user | -If TRUE, reauthenticate via Google login screen |
- ||
|---|---|---|---|
| no_auto | -Will ignore auto-authentication settings if TRUE |
+ json_file | +Authentication json file you have downloaded from your Google Project |
Value
- -Invisibly, the token that has been saved to the session
-Details
-If you have set the environment variable GCS_AUTH_FILE to a valid file location,
- the function will look there for authentication details.
-Otherwise it will look in the working directory for the `.httr-oauth` file, which if not present
- will trigger an authentication flow via Google login screen in your browser.
If GCS_AUTH_FILE is specified, then gcs_auth() will be called upon loading the package
- via library(googleCloudStorageR),
- meaning that calling this function yourself at the start of the session won't be necessary.
- GCS_AUTH_FILE can be either a token generated by gar_auth or
- service account JSON ending with file extension .json
The best way to authenticate is to use an environment argument pointing at your authentication file.
+Set the file location of your download Google Project JSON file in a GCS_AUTH_FILE argument
Then, when you load the library you should auto-authenticate
+However, you can authenticate directly using this function pointing at your JSON auth file.
+Examples
++# NOT RUN { +library(googleCloudStorageR) +gcs_auth("location_of_json_file.json") +# }
Contents
Compose up to 32 objects into one
+ +gcs_compose_objects.RdThis merges objects stored on Cloud Storage into one object.
+ +gcs_compose_objects(objects, destination, + bucket = gcs_get_global_bucket())+ +
Arguments
+| objects | +A character vector of object names to combine |
+
|---|---|
| destination | +Name of the new object. |
+
| bucket | +The bucket where the objects sit |
+
Value
+ +Object metadata
+ +See also
+ +Other object functions: gcs_copy_object,
+ gcs_delete_object,
+ gcs_get_object,
+ gcs_list_objects,
+ gcs_metadata_object
Examples
+++# NOT RUN { + gcs_global_bucket("your-bucket") + objs <- gcs_list_objects() + + compose_me <- objs$name[1:30] + + gcs_compose_objects(compose_me, "composed/test.json") + +# }
Copy an object
+ +gcs_copy_object.RdCopies an object to a new destination
+ +gcs_copy_object(source_object, destination_object, + source_bucket = gcs_get_global_bucket(), + destination_bucket = gcs_get_global_bucket(), rewriteToken = NULL, + destinationPredefinedAcl = NULL)+ +
Arguments
+| source_object | +The name of the object to copy, or a |
+
|---|---|
| destination_object | +The name of where to copy the object to, or a |
+
| source_bucket | +The bucket of the source object |
+
| destination_bucket | +The bucket of the destination |
+
| rewriteToken | +Include this field (from the previous rewrite response) on each rewrite request after the first one, until the rewrite response 'done' flag is true. |
+
| destinationPredefinedAcl | +Apply a predefined set of access controls to the destination object. If not NULL must be one of the predefined access controls such as |
+
Value
+ +If successful, a rewrite object.
+ +See also
+ +Other object functions: gcs_compose_objects,
+ gcs_delete_object,
+ gcs_get_object,
+ gcs_list_objects,
+ gcs_metadata_object
Create a new bucket
+ +gcs_create_bucket.RdCreate a new bucket in your project
+gcs_create_bucket(name, projectId, location = "US", - storageClass = c("MULTI_REGIONAL", "REGIONAL", "STANDARD", "NEARLINE", + storageClass = c("MULTI_REGIONAL", "REGIONAL", "STANDARD", "NEARLINE", "COLDLINE", "DURABLE_REDUCED_AVAILABILITY"), - predefinedAcl = c("projectPrivate", "authenticatedRead", "private", + predefinedAcl = c("projectPrivate", "authenticatedRead", "private", "publicRead", "publicReadWrite"), - predefinedDefaultObjectAcl = c("bucketOwnerFullControl", "bucketOwnerRead", - "authenticatedRead", "private", "projectPrivate", "publicRead"), - projection = c("noAcl", "full"), versioning = FALSE, lifecycle = NULL)+ predefinedDefaultObjectAcl = c("bucketOwnerFullControl", + "bucketOwnerRead", "authenticatedRead", "private", "projectPrivate", + "publicRead"), projection = c("noAcl", "full"), versioning = FALSE, + lifecycle = NULL) -
Arguments
+Arguments
| topic | +The pub/sub topic name |
+
|---|---|
| project | +The project-id that has the pub/sub topic |
+
| bucket | +The bucket for notifications |
+
| event_types | +What events to activate, leave at default for all |
+
Details
+ +Cloud Pub/Sub notifications allow you to track changes to your Cloud Storage objects.
+As a minimum you wil need: the Cloud Pub/Sub API activated for the project;
+sufficient permissions on the bucket you wish to monitor;
+sufficient permissions on the project to receive notifications;
+an existing pub/sub topic;
+have given your service account at least pubsub.publisher permission.
See also
+ +https://cloud.google.com/storage/docs/reporting-changes
+Other pubsub functions: gcs_delete_pubsub,
+ gcs_get_service_email,
+ gcs_list_pubsub
Examples
+++# NOT RUN { +project <- "myproject" +bucket <- "mybucket" + +# get the email to give access +gce_get_service_email(project) + +# once email has access, create a new pub/sub topic for your bucket +gcs_create_pubsub("gcs_r", project, bucket) + +# }+ +
Delete a bucket
+ +gcs_delete_bucket.RdDelete the bucket, and all its objects
+gcs_delete_bucket(bucket, ifMetagenerationMatch = NULL, ifMetagenerationNotMatch = NULL)-
Arguments
+Arguments
| object_name | -Object to be deleted |
+ Object to be deleted, or a |
||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bucket | @@ -132,9 +155,11 @@
| config_name | +A name of a configuration |
+
|---|---|
| bucket | +The bucket for notifications |
+
Details
+ +Cloud Pub/Sub notifications allow you to track changes to your Cloud Storage objects.
+As a minimum you wil need: the Cloud Pub/Sub API activated for the project;
+sufficient permissions on the bucket you wish to monitor;
+sufficient permissions on the project to receive notifications;
+an existing pub/sub topic; have given your service account at least pubsub.publisher permission.
See also
+ +https://cloud.google.com/storage/docs/reporting-changes
+Other pubsub functions: gcs_create_pubsub,
+ gcs_get_service_email,
+ gcs_list_pubsub
Get the download URL
+ +gcs_download_url.RdCreate the download URL for objects in buckets
+gcs_download_url(object_name, bucket = gcs_get_global_bucket(), public = FALSE)-
Arguments
+Arguments
| project | +The project name containing the bucket |
+
|---|
Details
+ +This service email can be different from the email in the service JSON. Give this
+pubsub.publisher permission in the Google cloud console.
See also
+ +Other pubsub functions: gcs_create_pubsub,
+ gcs_delete_pubsub,
+ gcs_list_pubsub
Set global bucket name
+ +gcs_global_bucket.RdSet a bucket name used for this R session
+gcs_global_bucket(bucket)-
Arguments
+Arguments
| bucket | +The bucket for notifications |
+
|---|
Details
+ +Cloud Pub/Sub notifications allow you to track changes to your Cloud Storage objects.
+As a minimum you wil need: the Cloud Pub/Sub API activated for the project;
+sufficient permissions on the bucket you wish to monitor;
+sufficient permissions on the project to receive notifications;
+an existing pub/sub topic; have given your service account at least pubsub.publisher permission.
See also
+ +https://cloud.google.com/storage/docs/reporting-changes
+Other pubsub functions: gcs_create_pubsub,
+ gcs_delete_pubsub,
+ gcs_get_service_email
Load .RData objects or sessions from the Google Cloud
+ +gcs_load.RdLoad R objects that have been saved using gcs_save or gcs_save_image
+gcs_load(file = ".RData", bucket = gcs_get_global_bucket(), envir = .GlobalEnv, saveToDisk = file, overwrite = TRUE)-
Arguments
+Arguments
| object_name | -Name of the object. GCS uses this version if also set elsewhere. |
+ Name of the object. GCS uses this version if also set elsewhere, or a |
+
|---|---|---|
| metadata | +User-provided metadata, in key/value pairs |
+ |
| md5Hash | +MD5 hash of the data; encoded using base64 |
+ |
| crc32c | +CRC32c checksum, as described in RFC 4960, Appendix B; encoded using base64 in big-endian byte order |
+ |
| contentLanguage | +Content-Language of the object data |
+ |
| contentEncoding | +Content-Encoding of the object data |
+ |
| contentDisposition | +Content-Disposition of the object data |
+ |
| cacheControl | +Cache-Control directive for the object data |
Value
See also
-Other object functions: gcs_delete_object,
+
Other object functions: gcs_compose_objects,
+ gcs_copy_object,
+ gcs_delete_object,
gcs_get_object,
- gcs_list_objects
gcs_list_objectsContents
Parse downloaded objects straight into R
+ +gcs_parse_download.RdWrapper for httr's content. This is the default function used in gcs_get_object
Wrapper for httr's content. This is the default function used in gcs_get_object
gcs_parse_download(object, encoding = "UTF-8")-
Arguments
+Arguments
| expiration_ts | -A timestamp of class |
+ A timestamp of class |
|||||
|---|---|---|---|---|---|---|---|
| verb | @@ -140,9 +164,9 @@
| ... | -Passed to source |
+ Passed to source |
|---|
Value
See also
-Other R session data functions: gcs_load,
+
Other R session data functions: gcs_load,
gcs_save_all, gcs_save_image,
- gcs_save
gcs_saveContents
Change access to an object in a bucket
+ +gcs_update_object_acl.RdUpdates Google Cloud Storage ObjectAccessControls
+gcs_update_object_acl(object_name, bucket = gcs_get_global_bucket(), - entity = "", entity_type = c("user", "group", "domain", "project", - "allUsers", "allAuthenticatedUsers"), role = c("READER", "OWNER"))+ entity = "", entity_type = c("user", "group", "domain", "project", + "allUsers", "allAuthenticatedUsers"), role = c("READER", "OWNER")) -
Arguments
+Arguments
| bucket | +gcs bucket |
+
|---|---|
| action | +"status", "enable", "disable", or "list" |
+
Value
+ +versioned_objects dataframe #only if "list" action
+ + +googleCloudStorageR
+ +googleCloudStorageR.RdInteract with Google Cloud Storage API in R. Part of the 'cloudyr' project.
+