Skip to content

Commit

Permalink
Nico shishkin patch 3 (#2449)
Browse files Browse the repository at this point in the history 
* MS Graph update

* Update microsoft-graph.md

---------

Co-authored-by: “nico-shishkin” <“[email protected]”>
  • Loading branch information
@nico-shishkin
nico-shishkin and “nico-shishkin” authored Aug 9, 2023
1 parent 388956a commit 5f6aa34
Showing 1 changed file with 82 additions and 104 deletions.
186 changes: 82 additions & 104 deletions _source/logzio_collections/_security-sources/microsoft-graph.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,8 @@ data-source: Microsoft Graph
data-for-product-source: Cloud SIEM
templates: ["no-template"]
open-source:
- title: Logzio-MSGraph
github-repo: microsoft-graph
- title: Microsoft Graph API fetcher
github-repo: logzio-api-fetcher
contributors:
- yyyogev
- imnotashrimp
Expand All @@ -20,144 +20,122 @@ shipping-tags:
order: 930
---

You can ship logs available from the Microsoft Graph APIs with Logzio-MSGraph.
Logzio-MSGraph is a self-hosted application.
You can ship logs available from the Microsoft Graph APIs with logzio-api-fetcher.

Logzio-MSGraph supports these APIs:

* Azure Active Directory audit logs
* Azure Active Directory sign-in logs

There are many other APIs available through Microsoft Graph.
If you don't see your API in the list,
please [open an issue](https://github.com/logzio/microsoft-graph/issues/new) at GitHub to request it.

#### To integrate Microsoft Graph and Logz.io

<div class="tasklist">

##### Register a new app in Azure Active Directory

In the Azure portal, go to [App registration](https://portal.azure.com/#blade/Microsoft_AAD_RegisteredApps/ApplicationsListBlade)
and select **New registration** from the top menu.

Name your app and click **Register**.

##### Create a client secret

Choose **Certificates & secrets** from the side menu,
and click on **New client secret**.

Add a **Description**.
We recommend something specific, such as "secret for Logzio-MSGraph integration".

In the **Expires** list, choose **Never**.

Click **Add**.

Copy the value of the generated secret to your text editor.
You'll need this later.
##### Pull the Docker image of the Logz.io API fetcher

<!-- info-box-start:info -->
You won't be able to retrieve the secret's value after you leave this page.
{:.info-box.note}
<!-- info-box-end -->

##### Set the app's permissions

Choose **API permissions** from the side menu,
and click **Add a permission**.

Select **Microsoft Graph > Application permissions**.

Select these items:
```shell
docker pull logzio/logzio-api-fetcher
```

* **AuditLog.Read.All**
* **Directory.Read.All**

Click **Add permissions**.
##### Create a local directory for this integration

Click **Grant admin consent for Default Directory**, and then click **Yes** to confirm.
You will need a dedicated directory to use it as mounted directory for the Docker container of the Logz.io API fetcher.

<!-- info-box-start:info -->
Only Azure administrators can grant consent for Default Directory. If the _Grant admin consent_ button is disabled, ask your Azure admin to update the setting for you.
{:.info-box.note}
<!-- info-box-end -->
```shell
mkdir logzio-api-fetcher
cd logzio-api-fetcher
```

##### Create a configuration file

Create a configuration yaml file (`logzio-msgraph-config.yaml`) for Logzio-MSGraph.

For a complete list of options, see the configuration parameters below.👇
In the directory created in the previous step, create a file `config.yaml` using the example configuration below:

```yaml
senderParams:
accountToken: "<<LOG-SHIPPING-TOKEN>>"
listenerUrl: "<<LISTENER-HOST>>"

azureADClient:
pullIntervalSeconds: 300
tenantId: "<<AD_TENANT_ID>>"
clientId: "<<APP_CLIENT_ID>>"
clientSecret: "<<APP_CLIENT_SECRET_VALUE>>"

logLevel: INFO
logzio:
url: https://<<LISTENER-HOST>>:8071
token: <<LOG-SHIPPING-TOKEN>>

oauth_apis:
- type: api_fetcher
name: azure_test
credentials:
id: <<AZURE_AD_SECRET_ID>>
key: <<AZURE_AD_SECRET_VALUE>>
token_http_request:
url: https://login.microsoftonline.com/<<AZURE_AD_TENANT_ID>>/oauth2/v2.0/token
body: client_id=<<AZURE_AD_CLIENT_ID>>
&scope=https://graph.microsoft.com/.default
&client_secret=<<AZURE_AD_SECRET_VALUE>>
&grant_type=client_credentials
headers:
method: POST
data_http_request:
url: https://graph.microsoft.com/v1.0/auditLogs/signIns
headers:
json_paths:
data_date: createdDateTime
settings:
time_interval: 1
days_back_fetch: 30

```

###### Parameters
| Parameter | Description | Required/Default |
|---|---|---|
| senderParams.accountToken | Your Logz.io account token. {% include log-shipping/log-shipping-token.html %} | Required |
| senderParams.listenerUrl | Listener URL. {% include log-shipping/listener-var.html %} | `listener.logz.io` |
| senderParams.fromDisk | If `true`, logs are stored on disk until they're shipped. (See **If from-disk=true** below). If `false`, logs persist in memory until they're shipped. (See **If from-disk=false** below. | `true` |
| senderParams.senderDrainIntervals | How often the sender should drain the queue, in seconds. | `30` |
| azureADClient.tenantId | Azure Active Directory tenant ID. You can find this in the _Overview_ section of the app you registered in step 1. | Required |
| azureADClient.clientId | Application client ID. You can find this in the _Overview_ section of the app you registered in step 1. | Required |
| azureADClient.clientSecret | Value of the Application Client Secret you created in step 2. | Required |
| azureADClient.pullIntervalSeconds | Time interval, in seconds, to pull the logs with the Graph API. | `300` |
| logLevel | Log level for Logizo-MSGraph to omit. Can be one of: `OFF`, `ERROR`, `WARN`, `INFO`, `DEBUG`, `TRACE`, `ALL`. | `INFO` |
| targetApi | Specifies types of api lists to run, each API provider has its own list. Must contain at least 1 list with 1 api in the list. Current supported providers: ADApis, ASCApis. | Required |

###### If fromDisk=true

| Parameter | Description | Required/Default |
|---|---|---|
senderParams.fileSystemFullPercentThreshold | Threshold percentage of disk space at which to stop queueing. If this threshold is reached, all new logs are dropped until used space drops below the threshold. Set to `-1` to ignore threshold. | `98` |
| senderParams.gcPersistedQueueFilesIntervalSeconds | Time interval, in seconds, to clean sent logs from the disk. | `30` |
| senderParams.diskSpaceCheckInterval | Time interval, in milliseconds, to check for disk space.| `1000` |

###### If fromDisk=false
| URL | {% include log-shipping/listener-var.md %} | Required |
| TOKEN | Your Logz.io account token. {% include log-shipping/log-shipping-token.html %} | Required |
| type | The type of the OAuth API. Currently we support the following types: azure_graph, general. | Required |
| name | The name of the OAuth API. Please make names unique. | Required |
| credentials.id | The OAuth API credentials id. | Required |
| credentials.key | The OAuth API credentials key. | Required |
| http_request.method | The HTTP method. Can be GET or POST. | Required |
| http_request.url | The OAuth API url. Make sure the url is without `?` at the end. | Required |
| http_request.headers | Pairs of key and value the represents the headers of the HTTP request. | Optional |
| http_request.body | The body of the HTTP request. Will be added to HTTP POST requests only. | Optional |
| token_http_request.method | The HTTP method. Can be GET or POST. | Required |
| token_http_request.url | The OAuth API token request url. Make sure the url is without `?` at the end. | Required |
| token_http_request.headers | Pairs of key and value the represents the headers of the HTTP request. | Optional |
| token_http_request.body | The body of the HTTP request. Will be added to HTTP POST requests only. | Optional |
| json_paths.data_date | The json path to the data's date value inside the response of the OAuth API. | Required |
| settings.time_interval | The OAuth API time interval between runs. | Required |
| settings.days_back_fetch | The max days back to fetch from the OAuth API. | Optional. Default value is 14 days.|
| filters | Pairs of key and value of parameters that can be added to the OAuth API url. Make sure the keys and values are valid for the OAuth API. | Optional |
| custom_fields | Pairs of key and value that will be added to each data and be sent to Logz.io. | Optional |

##### Create a Last Start Dates text file

Create an empty text file named last_start_dates.txt in the same directory as the config file:

| Parameter | Description | Required/Default |
|---|---|---|
| senderParams.inMemoryQueueCapacityInBytes | The amount of memory, in bytes, Logzio-MSGraph can use for the memory queue. Set to `-1` for unlimited bytes. | `1024 * 1024 * 100` |
| senderParams.logsCountLimit | The number of logs in the memory queue before dropping new logs. Set to `-1` to configure the sender to not limit the queue by logs count. | `-1` |
```shell
$ touch last_start_dates.txt
```

##### Download and run Logzio-MSGraph
After every successful iteration of an API, the last start date of the next iteration will be written to last_start_dates.txt. Each line starts with the API name and ends with the last start date.

You can launch Logzio-MSGraph in a Docker container or as a standalone Java app.
If you stopped the container, you can continue from the exact place you stopped, by adding the date to the API filters in the configuration.

In a Docker container:
##### Run the Docker container

```shell
docker run -d -v $(pwd)/logzio-msgraph-config.yaml:/config.yaml logzio/logzio-msgraph
docker run --name logzio-api-fetcher \
-v "$(pwd)":/app/src/shared \
logzio/logzio-api-fetcher
```

Or to run as a standalone Java app,
download the latest jar from the [release page](https://github.com/logzio/microsoft-graph/releases).
Then run:
##### Stop the Docker container

When you stop the container, the code will run until the iteration is completed. To make sure it will finish the iteration on time, please give it a grace period of 30 seconds when you run the `docker stop` command.

```shell
java -jar logzio-msgraph.jar logzio-msgraph-config.yaml
docker stop -t 30 logzio-api-fetcher
```

Logs collected by this integration will have the type `Microsoft-Graph`

##### Check Logz.io for your logs

Give your logs some time to get from your system to ours, and then open [Open Search Dashboards](https://app.logz.io/#/dashboard/osd).
Give your logs some time to get from your system to ours,
and then open [Open Search Dashboards](https://app.logz.io/#/dashboard/osd). You can filter for data of your custom field type value or type `api_fetcher` to see the incoming Microsoft Graph logs.

If you still don't see your logs, see [log shipping troubleshooting]({{site.baseurl}}/user-guide/log-shipping/log-shipping-troubleshooting.html).
If you still don't see your logs,
see [log shipping troubleshooting]({{site.baseurl}}/user-guide/log-shipping/log-shipping-troubleshooting.html).

</div>

0 comments on commit 5f6aa34

Please sign in to comment.