rds$build_auth_token not working #324
Comments
|
Thank you for the bug report! We'll check it out in the next week. |
|
Hello @Drwhit, unfortunately I was not able to reproduce the issue on Linux or Windows in us-east-2 using |
|
I’ve tried in two different machines, local Mac and ec2 linux.
I do have multiple profiles on my local, but am able to generate the token
using the aws cli without specifying which role, presumably because my
account is an admin account.
Only one profile in my ec2 instance though. credentials are obtained by the
docker container from the instance, because the instance has been assigned
a role via the Aws console, so there is no credentials file.
…On Sat, Sep 12, 2020 at 3:41 PM David Kretch ***@***.***> wrote:
Hello @Drwhit <https://github.com/Drwhit>, unfortunately I was not able
to reproduce on Linux or Windows in us-east-2. I don't have a Mac to test
with but doubt it is platform specific. I don't have many ideas of what it
could be, but possibly do you have more than one profile or set of
credentials in your ~/.aws/config or ~/.aws/credentials?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#324 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/ADTY24EFY5FXJC34CBAEXZTSFPTIBANCNFSM4Q3KGIRA>
.
|
|
Cool, thank you. I'll try to set something like that up. |
|
I am having the same issue (although I am using the I do get a token back but it seems not to work. In my case I am using Here is the code I am trying and is giving the error: Everything works as expected if I use the AWS cli or use For example, this code calling boto3 from reticulate does work: @atheriel do you have ideas? |
|
@tnederlof thanks for raising this issue. I will revisit this before next paws.common release |
|
Thank you for taking a look @DyfanJones let me know if there is more info I can provide to help, I have some envs on my side to reproduce. |
|
I notice that the code provided above has this line: token <- rds$build_auth_token(endpoint = "XXXXXXX.us-east-2.rds.amazonaws.com", region = "us-east-2", user = "user1") but looking at the test suite here it seems like you need to add the port to the endpoint: paws/paws.common/tests/testthat/test_custom_rds.R Lines 1 to 13 in 50cd4f1 Maybe that's the source of the issue? I'm also a little confused about why you need to provide |
|
I did try the port in the endpoint and still got the same error when I used the token to connect |
|
I wouldn't expect a port to cause any issues. I have recently been working on the urlparse migrating it to cpp and the port is always attached to the host :) |
|
I believe it is losing it credentials somehow. I am prototyping some possible replacements @atheriel @tnederlof Do you mind testing out this prototype method. I want to make sure it keeps its credentials. presign <- function(request, default = paws.common:::v1_sign_request_handler) {
# build request
request <- paws.common:::build(request)
signer <- switch(request$config[["signature_version"]],
"v1" = paws.common:::v1_sign_request_handler,
"s3" = paws.common:::s3_sign_request_handler,
"s3v4" = paws.common:::s3v4_sign_request_handler,
"v4" = paws.common:::v4_sign_request_handler,
default
)
# sign request
signer(request)
}
rds_build_auth_token_v2 <- function(DBHostname, Port, DBUsername, Region=NULL) {
op <- paws.common:::new_operation(
name = "BuildAuthTokenV2", http_method = "GET",
http_path = "/", host_prefix = "", paginator = list(), stream_api = F
)
input <- list(
Action = "connect",
DBUser = DBUsername
)
output <- list()
config <- paws.common::get_config()
if (!startsWith(DBHostname, "https://")) DBHostname <- paste0("https://", DBHostname)
config$endpoint <- paste(DBHostname, Port, sep = ":")
svc <- paws.database:::.rds$service(config, op)
# create new request
request <- paws.common::new_request(svc, op, input, output)
request$expire_time <- 900
request <- presign(request, paws.common:::v4_sign_request_handler)
request$http_request$url$raw_query <- sub("&Version=[0-9]{,4}-[0-9]{,2}-[0-9]{,2}", "", request$http_request$url$raw_query)
url <- paws.common:::build_url(request$http_request$url)
return(substr(url, 9, nchar(url)))
}library(paws)
client = rds()
# add prototype method to client
client$build_auth_token_v2 <- rds_build_auth_token_v2
client$build_auth_token_v2(
DBHostname='XXXXXXX.us-east-2.rds.amazonaws.com',
Port = 5432,
DBUsername="user1"
) |
|
@DyfanJones, unfortunately, I still get the same error on my end using that resulting token. Happy to test anything else that helps. I have a boto3 example running with reticulate to make sure I can keep connecting that way. |
|
@tnederlof thanks, are you ok to run some prototype functions for me? I currently don't have an equivalent environment to test this out. |
|
@DyfanJones yes Ill keep my environment around, happy to test! |
|
@tnederlof can you try the following prototype function? rds_build_auth_token_v2 <- function(DBHostname, Port, DBUsername, Region=NULL) {
op <- paws.common:::new_operation(
name = "rds-db", http_method = "GET",
http_path = "/", host_prefix = "", paginator = list(), stream_api = F
)
input <- list(
Action = "connect",
DBUser = DBUsername
)
output <- list()
config <- paws.common::get_config()
if (!is.null(Region)) {
config$region <- Region
}
if (!startsWith(DBHostname, "https://")) DBHostname <- paste0("https://", DBHostname)
config$endpoint <- paste(DBHostname, Port, sep = ":")
svc <- paws.database:::.rds$service(config, op)
# create new request
request <- paws.common::new_request(svc, op, input, output)
request$expire_time <- 900
# build request
request <- paws.common:::build(request)
request$client_info$signing_name <- "rds-db"
request$http_request$url$raw_query <- sub(
"&Version=[0-9]{,4}-[0-9]{,2}-[0-9]{,2}", "", request$http_request$url$raw_query
)
request$http_request$header$Accept <- NULL
# sign request
request <- paws.common:::v4_sign_request_handler(request)
url <- paws.common:::build_url(request$http_request$url)
return(substr(url, 9, nchar(url)))
}I forgot the raw_query is used within the signer so I need to ensure raw_query is formatted correctly before v4 signing. |
|
@DyfanJones interesting! I still have no luck with that token unfortunately. Ill put my code below, I have verified it is using the new method being described in the script. library(odbc)
library(DBI)
library(paws)
rds_build_auth_token_v2 <- function(DBHostname, Port, DBUsername, Region=NULL) {
op <- paws.common:::new_operation(
name = "rds-db", http_method = "GET",
http_path = "/", host_prefix = "", paginator = list(), stream_api = F
)
input <- list(
Action = "connect",
DBUser = DBUsername
)
output <- list()
config <- paws.common::get_config()
if (!is.null(Region)) {
config$region <- Region
}
if (!startsWith(DBHostname, "https://")) DBHostname <- paste0("https://", DBHostname)
config$endpoint <- paste(DBHostname, Port, sep = ":")
svc <- paws.database:::.rds$service(config, op)
# create new request
request <- paws.common::new_request(svc, op, input, output)
request$expire_time <- 900
# build request
request <- paws.common:::build(request)
request$client_info$signing_name <- "rds-db"
request$http_request$url$raw_query <- sub(
"&Version=[0-9]{,4}-[0-9]{,2}-[0-9]{,2}", "", request$http_request$url$raw_query
)
# sign request
request <- paws.common:::v4_sign_request_handler(request)
url <- paws.common:::build_url(request$http_request$url)
return(substr(url, 9, nchar(url)))
}
client = rds()
# add prototype method to client
client$build_auth_token_v2 <- rds_build_auth_token_v2
token <- client$build_auth_token_v2(
DBHostname='XXXXX.us-east-2.rds.amazonaws.com',
Port = 5432,
DBUsername="xxxxx",
Region="us-east-2"
)
con <- DBI::dbConnect(odbc::odbc(),
Driver = "PostgreSQL",
Server = "xxxxxx.rds.amazonaws.com",
Database = "xxxxx",
UID = "xxxxxxx",
PWD = token,
Port = 5432,
) |
|
Do you have any cloudformation or terraform so I can reproduce the environment? I am abit puzzled why it isn't signing correctly. It should be picking up the correct headers and that 🤔 |
|
I appreciate the help, yes it would be good if you could repro, ill try to create a more simple environment with the issue that you can set up. |
|
I think I know what is going on, within botocore it looks like the host is set to lower case before it is signed. As we leave it alone it causes a miss match in signatures. I will have a better look to where the lower case is set :) |
|
Found where the url needs to be lower case before signature: |
|
Good sleuthing! Happy to test another pass at the prototype function. |
|
This one is going to be a little trickier to test: remotes::install_github("dyfanjones/paws/paws.common", ref = "cpp_url_parse_fix")The current paws.common 0.8.0 (dev) won't work as expected with paws 0.7.0. This is down to Ultimately the endpoint regex is getting changed due to the vendor switch. So please make sure you revert your paws.common to cran version after the test (apologises for the inconvenience). library(odbc)
library(DBI)
library(paws)
client = rds()
# add prototype method to client
client$build_auth_token_v2 <- paws.common:::rds_build_auth_token_v2
token <- client$build_auth_token_v2(
DBHostname='XXXXX.us-east-2.rds.amazonaws.com',
Port = 5432,
DBUsername="xxxxx",
Region="us-east-2"
)
con <- DBI::dbConnect(odbc::odbc(),
Driver = "PostgreSQL",
Server = "xxxxxx.rds.amazonaws.com",
Database = "xxxxx",
UID = "xxxxxxx",
PWD = token,
Port = 5432,
)NOTE: For other paws use make sure you revert paws.common back to version 0.7.7 (cran version) |
|
Hmmm still no luck. Just to make sure I am running the right function here is what I see for function (DBHostname, Port, DBUsername, Region = NULL)
{
op <- new_operation(name = "connect", http_method = "GET",
http_path = "/", host_prefix = "", paginator = list(),
stream_api = FALSE)
input <- list(Action = "connect", DBUser = DBUsername)
output <- list()
config <- get_config()
if (!is.null(Region)) {
config$region <- Region
}
if (!startsWith(DBHostname, "https://"))
DBHostname <- paste0("https://", DBHostname)
config$endpoint <- paste(DBHostname, Port, sep = ":")
metadata <- list(service_name = "rds", endpoints = list(),
service_id = "RDS", api_version = "2014-10-31", signing_name = "rds-db",
json_version = "", target_prefix = "")
handlers <- new_handlers("query", "v4")
svc <- new_service(metadata, handlers, config, op)
request <- new_request(svc, op, input, output)
request$expire_time <- 900
request$http_request$url$raw_query <- build_query_string(request$params)
request <- v4_sign_request_handler(request)
url <- build_url(request$http_request$url)
return(substr(url, 9, nchar(url)))
} |
|
That is correct, however it would be the change to If it still isn't working, is it possible for the cloud formation to be shared so I can build the same environment? |
|
Yes, I will work on that, although it will likely be next week for me to pull something together. |
|
Here is some testing code that will mock the datetime so that paws and boto3 build the token with the same.
remotes::install_github("dyfanjones/paws/paws.common", ref = "signature_order")import boto3
from unittest.mock import patch
import datetime
# Mock datetime
class FixedDatetime(datetime.datetime):
@classmethod
def utcnow(cls):
return cls(2025, 1, 1, 0, 0, 1, tzinfo=datetime.timezone.utc)
def mock_boto_token(DBHostname, Port, DBUsername, Region):
with patch("datetime.datetime", FixedDatetime):
token = boto3.client("rds").generate_db_auth_token(
DBHostname=DBHostname,
Port=Port,
DBUsername=DBUsername,
Region=Region,
)
return tokenlibrary(paws)
library(reticulate)
# point to boto3 script
reticulate::source_python("rds_mock_token.py")
client = rds()
# add prototype method to client
client$build_auth_token_v2 <- paws.common:::rds_build_auth_token_v2
testthat::local_mocked_bindings(
Sys.time = function() as.POSIXct("2025/01/01 00:00:01 UTC"),
.package = "paws.common"
)
host = 'XXXXX.us-east-2.rds.amazonaws.com'
port = 5432L
user = "xxxxx"
region = "us-east-2"
paws_token = client$build_auth_token_v2(
DBHostname=host,
Port=port,
DBUsername=user,
Region=region
)
boto_token = mock_boto_token(
DBHostname=host,
Port=port,
DBUsername=user,
Region=region
)
# compare queries built by both
waldo::compare(
paws_token,
boto_token
)My initial results: But I have only tested it locally and aws sagemaker. |
|
Very interesting. When I run it I get differences in the Token and X-Amz-Signature. One thing I ran into is I had to add my region to run the boto3 code ( |
|
I don't think region would have a huge impact as region is usually used to build the endpoint. For example I can change the python code to: import boto3
from unittest.mock import patch
import datetime
# Mock datetime
class FixedDatetime(datetime.datetime):
@classmethod
def utcnow(cls):
return cls(2025, 1, 1, 0, 0, 1, tzinfo=datetime.timezone.utc)
def mock_boto_token(DBHostname, Port, DBUsername, Region):
with patch("datetime.datetime", FixedDatetime):
token = boto3.client("rds", region_name = Region).generate_db_auth_token(
DBHostname=DBHostname,
Port=Port,
DBUsername=DBUsername,
)
return tokenAnd it should return the same endpoint library(paws)
library(reticulate)
# point to boto3 script
reticulate::source_python("../inst/python_mocks/rds_generate_db_auth_token.py")
client = rds()
# add prototype method to client
client$build_auth_token_v2 <- paws.common:::rds_build_auth_token_v2
testthat::local_mocked_bindings(
Sys.time = function() as.POSIXct("2025/01/01 00:00:01 UTC"),
.package = "paws.common"
)
host = 'XXXXX.us-east-2.rds.amazonaws.com'
port = 5432L
user = "xxxxx"
region = "us-east-2"
paws_token = client$build_auth_token_v2(
DBHostname=host,
Port=port,
DBUsername=user,
Region=region
)
boto_token = mock_boto_token(
DBHostname=host,
Port=port,
DBUsername=user,
Region=region
)
# compare queries built by both
waldo::compare(
paws_token,
boto_token
)
#> ✔ No differencesThere must be something I am missing in the environment 🤔 |
Thanks for your work on this package, looks like it will come in very handy.
I'm a beginner at aws authentication especially related to databases, so it could definitely be user error, but it appears to me that the rds$build_auth_token function isn't working.
When I obtain the token using the this function, I am unable to connect to the database (i've replaced sensitive information with "xxxxxxxx":
token <- rds$build_auth_token(endpoint = "xxxxxxxx.us-west-2.rds.amazonaws.com:5432", region = "us-west-2", user = "xxxxxxxx")pool <- pool::dbPool(drv = RPostgres::Postgres(), dbname="xxxxxxxx", host="xxxxxxxx.us-west-2.rds.amazonaws.com", user= "xxxxxxxx", password=token, bigint = "numeric")Error in connection_create(names(opts), as.vector(opts)) : FATAL: PAM authentication failed for user "xxxxxxxx" FATAL: pg_hba.conf rejects connection for host "xxxxxxxx", user "xxxxxxxx", database "xxxxxxxx", SSL offHowever, if I obtain the token using the following command, I am able to connect:
systoken <- system("aws rds generate-db-auth-token --hostname xxxxxxxx.us-west-2.rds.amazonaws.com --port 5432 --username xxxxxxxx")pool <- pool::dbPool(drv = RPostgres::Postgres(), dbname="xxxxxxxx", host="xxxxxxxx.us-west-2.rds.amazonaws.com", user= "xxxxxxxx", password=systoken, bigint = "numeric")Here is some system info:
Matrix products: default
BLAS: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/3.5/Resources/lib/libRlapack.dylib
locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
The text was updated successfully, but these errors were encountered: