Help Needed Urgently!! Unable to attach policy for the newly created role from ConsoleMe dashboard

[akshataashok]

Help Needed Urgently!!

Hi Team, I have launched ConsoleMe and created a role. When I am trying to update the inline policy for the newly created role I am getting the below error. I have followed docker way of installation of ConsoleMe.

Kindly suggest here.

Also when I try to save the policy I get below error.

[castrapel]

Hi @akshataashok, it looks like you're running ConsoleMe in the cloud without authentication or network security group (A.K.A. Firewall) rules. Please make sure to restrict ConsoleMe so that only you and your team can access it, and consider using either OIDC, SAML, or ALB Auth to authenticate your users, which is discussed in more detail here.

Now it looks like you're running the default Docker development configuration here which specifies your DynamoDB endpoint as http://CONSOLEME-DYNAMODB:8005. This is created by the consoleme-dynamodb network which is referenced in our docker files (here and here). If you're running this on an instance, and not locally, you might have a difficult time getting Docker Networking to work. I highly recommend you modify the configuration to use AWS DynamoDB and not local DynamoDB. Also recommend that you use Elasticache, or install Reids on the system you're deploying ConsoleMe to. You can do this simply by 1) making your DynamoDB tables in AWS , 2) Removing the references to dynamodb_server: http://CONSOLEME-DYNAMODB:8005 and redis://consoleme-redis:6379/1 or consoleme-redis.

Alternatively, it might be easier to deploy with Terraform (here). When I have time, I hope to document an ECS deployment method that will make the process clearer.

[akshataashok]

Hi,
Thank you for the details. I have updated the docker networking so that now I am able to update the dynamic configuration. Also the DynamoDB tables were empty and I have created all the necessary tables. We are able to view the roles and policies in the ConsoleMe dashboard, however after couple of minutes the page will be empty and we get the below from docker logs. We suspect that this information is not able to get cached but is available in the DB.
Also we would like to request you that if we modify any yaml files how do we run them? Is it only by bringing the containers down and up?

Kindly request you to suggest for this.
{"asctime": "2021-05-18T07:38:37Z+0000", "name": "consoleme", "processName": "MainProcess", "filename": "exceptions.py", "funcName": "init", "levelname": "ERROR", "lineno": 14, "module": "exceptions", "threadName": "MainThread", "message": "Unable to retrieve expected data.", "eventTime": "2021-05-17T23:52:33.987081-07:00", "hostname": "e19b737030d4", "timestamp": "2021-05-18T07:38:37Z+0000"}
{"asctime": "2021-05-18T07:38:37Z+0000", "name": "consoleme", "processName": "MainProcess", "filename": "init.py", "funcName": "retrieve_credential_authorization_mapping", "levelname": "ERROR", "lineno": 60, "module": "init", "threadName": "MainThread", "message": null, "function": "CredentialAuthorizationMapping.retrieve_credential_authorization_mapping", "error": "Error loading cloud credential mapping. Returning empty mapping: Unable to retrieve expected data.", "exc_info": "Traceback (most recent call last):\n File "/apps/consoleme/consoleme/lib/cloud_credential_authorization_mapping/init.py", line 50, in retrieve_credential_authorization_mapping\n self.authorization_mapping = await retrieve_json_data_from_redis_or_s3(\n File "/apps/consoleme/consoleme/lib/cache.py", line 169, in retrieve_json_data_from_redis_or_s3\n raise DataNotRetrievable("Unable to retrieve expected data.")\nconsoleme.exceptions.exceptions.DataNotRetrievable: Unable to retrieve expected data.", "eventTime": "2021-05-17T23:52:33.987081-07:00", "hostname": "e19b737030d4", "timestamp": "2021-05-18T07:38:37Z+0000"}
{"asctime": "2021-05-18T07:38:37Z+0000", "name": "consoleme", "processName": "MainProcess", "filename": "base.py", "funcName": "authorization_flow", "levelname": "ERROR", "lineno": 447, "module": "base", "threadName": "MainThread", "message": "No eligible roles detected for user. But letting them continue", "function": "Basehandler.authorization_flow", "ip": "127.0.0.1", "request_path": "/api/v2/user_profile", "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36", "request_id": "e2131767-14d4-4c0f-8c72-b3dd8e090f94", "eventTime": "2021-05-17T23:52:33.987081-07:00", "hostname": "e19b737030d4", "timestamp": "2021-05-18T07:38:37Z+0000"}
{"asctime": "2021-05-18T07:38:37Z+0000", "name": "consoleme", "processName": "MainProcess", "filename": "base.py", "funcName": "authorization_flow", "levelname": "DEBUG", "lineno": 457, "module": "base", "threadName": "MainThread", "message": "Successfully authorized user.", "function": "Basehandler.authorization_flow", "ip": "127.0.0.1", "request_path": "/api/v2/user_profile", "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36", "request_id": "e2131767-14d4-4c0f-8c72-b3dd8e090f94", "eligible_roles": 0, "eligible_accounts": 0, "eventTime": "2021-05-17T23:52:33.987081-07:00", "hostname": "e19b737030d4", "timestamp": "2021-05-18T07:38:37Z+0000"}
2021-05-18 07:38:37,793 - INFO - tornado.access - [web.py:2239 - log_request() ] - 200 GET /api/v2/user_profile (127.0.0.1) 4.81ms

[castrapel]

There's initialize_dynamodb_oss.py that can set up your dynamo tables for you, and initialize_redis_oss.py that can help with pre-populating your Redis cache.

The static configuration files shouldn't change too often after you have a stable configuration. Stopping and restarting the containers is the easiest way to get the latest configuration changes. However, your Dynamic configuration changes should automatically load within 1 minute of making the change.

The error you are seeing indicates that ConsoleMe couldn't find the cached credential authorization mapping in Redis. If you have configuration values set for "generate_and_store_credential_authorization_mapping.s3.bucket" and "generate_and_store_credential_authorization_mapping.s3.file", it will also check that location. You can set these configuration values and try reinitializing your redis cache (initialize_redis_oss.py), which should save the file to S3 and it will be retrievable in your web request.

But what I don't know is why your Redis data is disappearing. When you restart your docker containers, you're literally just running docker-compose -f docker-compose-dockerhub.yaml -f docker-compose-dependencies.yaml down, and then up? Do you think your Redis data is lost when you are bringing the containers down and up?

Did you manually create your role credential authorization mapping? If so, the celery task here runs every 5 minutes and will overwrite it.

[akshataashok]

Hi,
Thank you for your updates but I have couple of queries here and it would be great if you could assist here. All that I have done is , I have cloned the ConsoleMe repo, inside an EC2 instance. Installed all the prerequisites and ran docker-compose -f docker-compose-dockerhub.yaml -f docker-compose-dependencies.yaml up -d.
Then I was facing docker networking issue between the containers and I fixed it where I was able to solve the connectivity issues. Also I got error stating there wasn't any DynamoDB table for writing. Hence I created couple of DB tables manually as per https://github.com/Netflix/consoleme/blob/master/scripts/initialize_dynamodb_oss.py.

Then I started getting the roles and policies data visible in ConsoleMe dashboard. However I noticed that the data is available in the dashboard only for ~10mins after which the page becomes blank. If I restart the containers by running docker-compose -f docker-compose-dockerhub.yaml -f docker-compose-dependencies.yaml down
docker-compose -f docker-compose-dockerhub.yaml -f docker-compose-dependencies.yaml up -d again I will get the list of roles and policies in the dashboard which again disappears after couple of minutes.

But I could say the data is available in the DB which we could see and there's no data loss.
Regarding the question "Did you manually create your role credential authorization mapping? If so, the celery task here runs every 5 minutes and will overwrite it." I haven't created any configuration manually.

My doubt here is as you mentioned we have
https://github.com/Netflix/consoleme/blob/master/consoleme/celery_tasks/celery_tasks.py#L1308 and https://github.com/Netflix/consoleme/blob/master/scripts/initialize_dynamodb_oss.py. But if I do any modifications to these or any other yaml files how do I execute them to get them reflected. As of now I only restart the containers.
**
I have not modified any static files all that I have done is updated the dynamic configuration. Also I am unable to do any changes in the configuration for role credential authorization mapping like I am not getting which is exact file where I need to do changes and how do I run these scripts to make them implemented.** or restarting the containers is the only way of doing it?
Please help me with the right steps.

[akshataashok]

Hi,
Would like to bring to your notice that the Celery container is going down after couple of minutes which I suspect should be the reason for the page going blank. As long as this container is up and running we could access the roles and when this container goes off data will not be visible in the dashboard.

b0774dff8e50 consoleme/consoleme "bash -c ' python sc…" 3 hours ago Up 4 minutes 8081/tcp consoleme_consoleme-celery_1

[castrapel]

Hmm, could you print the container logs for the celery container when it crashes?

[akshataashok]

Hello,
I have attached here the celery container logs for your reference. Thank you
b0774dff8e50a5feb5c0ca9382336eaead4c30fac98ad60445429e5d56a19cde-json.log

On a high level the initial error says the below:
2021-05-21 08:47:17,464 - WARNING - root - [retrieve_or_decode_configuration.py:56 - () ] - Neither the CONSOLEME_CONFIG_B64 or CONSOLEME_CONFIG_S3 environment variables are defined. This is required to retrieve ConsoleMe's configuration via S3, or decode it from Base64. Exiting
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 169, in _new_conn
conn = connection.create_connection(
File "/usr/local/lib/python3.8/site-packages/urllib3/util/connection.py", line 96, in create_connection
raise err
File "/usr/local/lib/python3.8/site-packages/urllib3/util/connection.py", line 86, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
Since I am using docker mode of installation with the default settings, ideally I am referring to example_config/example_config_development.yaml.
So please do let me know if I need to do any changes to any of these configuration files and which commands to rerun to be able to resolve this issue.

[dilbwagsingh]

Hey @akshataashok, although your logs show a different error than mine. I also had a similar issue with celery stopping after a while. Can you just ensure if your EC2 is not publicly accessible. Link to article - redis/redis#6420