Skip to content

Commit

Permalink
Update README.md
Browse files Browse the repository at this point in the history
  • Loading branch information
himanshu219 authored Mar 29, 2024
1 parent 1ddabda commit 5d01992
Showing 1 changed file with 39 additions and 11 deletions.
50 changes: 39 additions & 11 deletions BlockBlobReader/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,24 +4,52 @@ This contains the function to read from an Azure Blob Storage, then forward data
## About the Configuration Process
Sumo provides an Azure Resource Management (ARM) template to build most of the components in the pipeline. The template creates:

An event hub to which Azure Event Grid routes create block blobs events.
A Service Bus for storing tasks.
Three Azure functions—TaskProducer, TaskConsumer, and DLQTaskConsumer—that are responsible for sending monitoring data to Sumo.
A storage account to which the Azure functions write their log messages about successful and failed transmissions.
You download the Sumo-provided ARM template, upload the template to the Azure Portal, set the parameters that identify the URL of your Sumo HTTP source and and the connection string of for the Azure Storage Account (where Azure services export their logs), and deploythe template. After deployment, you create an Event Grid subscription with a Azure Storage Account as publisher and the event hub created by the ARM template as the subscriber.
* An event hub to which Azure Event Grid routes create block blobs events.
* A Service Bus for storing tasks.
* Three Azure functions — TaskProducer, TaskConsumer, and DLQTaskConsumer that are responsible for sending monitoring data to Sumo.
* A storage account to which the Azure functions write their log messages about successful and failed transmissions.

For more details checkout the [documentation](https://help.sumologic.com/Send-Data/Collect-from-Other-Data-Sources/Azure_Blob_Storage/Collect_Logs_from_Azure_Blob_Storage)

![Blob Storage Data Collection Pipeline](https://s3.amazonaws.com/appdev-cloudformation-templates/AzureBlobStorageCollection.png)
![Block Blob Storage Data Collection Pipeline](https://s3.amazonaws.com/appdev-cloudformation-templates/AzureBlobStorageCollection.png)

## For Developers

### Code structure

Currently ARM template is integrated with github and for each functions build folder is present in `BlockBlobReader/target` directory

## Building the function
Currently ARM template is integrated with github and for each functions
* BlockBlobReader/target/producer_build/BlobTaskProducer - Function for Creating tasks(json object with start and end bytes).
* BlockBlobReader/target/consumer_build/BlobTaskConsumer - Function for Downloading block blobs and ingesting to Sumo
* BlockBlobReader/target/dlqprocessor_build/BlobTaskConsumer - Function for retrying failed tasks.

## For Developers
### Updating target directory

Make all the code changes in `BlockBlobReader/src` directory, once all the changes are completed, run below command to update target directory.

`npm run build`
This command copies required files in BlockBlobReader/target/ directory

Integrations tests are in BlockBlobReader/tests folder and unit tests are in sumo-function-utils/tests folder
This command copies required files in `BlockBlobReader/target` directory

Integrations tests are in `BlockBlobReader/tests` folder and unit tests are in `sumo-function-utils/tests` folder

### Run Unit Test

Integrations tests are in `BlockBlobReader/tests` folder and unit tests are in sumo-`function-utils/tests` folder

```console

export AZURE_SUBSCRIPTION_ID=`<Your azure subscription id, to obtain it refer docs https://learn.microsoft.com/en-us/azure/azure-portal/get-subscription-tenant-id#find-your-azure-subscription>`
export AZURE_CLIENT_ID=`Your application id which you can get after registering application. Refer https://learn.microsoft.com/en-us/entra/identity-platform/quickstart-register-app#register-an-application`
export AZURE_CLIENT_SECRET=`Generate client secret by referring docs https://learn.microsoft.com/en-us/entra/identity-platform/quickstart-register-app#add-credentials`
export AZURE_TENANT_ID=`You tenant id, to obtain it refer docs https://learn.microsoft.com/en-us/azure/azure-portal/get-subscription-tenant-id#find-your-microsoft-entra-tenant`
export AZURE_DEFAULT_REGION=`eastus`
export SUMO_ACCESS_ID=`<Generate access id and access key https://help.sumologic.com/docs/manage/security/access-keys/#create-your-access-key>`
export SUMO_ACCESS_KEY=`<Generate access id and access key https://help.sumologic.com/docs/manage/security/access-keys/#create-your-access-key>`
export SUMO_DEPLOYMENT=`Enter one of the allowed values au, ca, de, eu, fed, in, jp, us1 or us2. Visit https://help.sumologic.com/APIs/General-API-Information/Sumo-Logic-Endpoints-and-Firewall-Security`

```

Execute below command under `BlockBlobReader/tests` directory

`python test_blobreader.py`

0 comments on commit 5d01992

Please sign in to comment.