Skip to content

Commit

Permalink
Update notebooks with settings file
Browse files Browse the repository at this point in the history
  • Loading branch information
@mehrinkiani
mehrinkiani committed Feb 6, 2024
1 parent bfa8784 commit 79f6c5b
Show file tree
Hide file tree
Showing 4 changed files with 294 additions and 114 deletions.
120 changes: 85 additions & 35 deletions notebooks/keras_fashion_mnist.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
"name": "stdout",
"output_type": "stream",
"text": [
"modelscan, version 0.5.0\n"
"modelscan, version 0.5.1\n"
]
}
],
Expand Down Expand Up @@ -48,6 +48,37 @@
"from utils.tensorflow_fashion_mnist_model import train_model, get_predictions"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Optional Settings File for ModelScan\n",
"\n",
"ModelScan scan settings can be configured using a settings file. \n",
"\n",
"- To create a configurable settings file use: `modelscan create-settings-file` (creates a `modelscan-settings.toml` file in current directory). \n",
"\n",
"- The location and name of a settings file can also be specified with `modelscan create-settings-file -l ../path-to/settings-file.toml` \n",
"\n",
"Configurations:\n",
"- A settings file would allow enabling/disabling particular scans such as H5LambdaDetectScan, and PickleUnsafeOpScan. \n",
"\n",
"- A settings file can also be configured to set particular severity level (CRITICAL, HIGH, MEDIUM, or LOW) for unsafe operators. \n",
"\n",
"- Reporting format for ModelScan results can also be specified in a settings file.\n",
"\n",
"If you would like to create a configurable settings file, uncomment the code in the next cell and run it. "
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"#!modelscan create-settings-file -l my-modelscan-settings.toml"
]
},
{
"cell_type": "markdown",
"metadata": {},
Expand All @@ -59,43 +90,43 @@
},
{
"cell_type": "code",
"execution_count": 4,
"execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 1/10\n",
"1875/1875 [==============================] - 2s 1ms/step - loss: 0.4986 - accuracy: 0.8258\n",
"1875/1875 [==============================] - 1s 726us/step - loss: 0.4961 - accuracy: 0.8248\n",
"Epoch 2/10\n",
"1875/1875 [==============================] - 2s 800us/step - loss: 0.3741 - accuracy: 0.8649\n",
"1875/1875 [==============================] - 1s 689us/step - loss: 0.3736 - accuracy: 0.8641\n",
"Epoch 3/10\n",
"1875/1875 [==============================] - 1s 772us/step - loss: 0.3391 - accuracy: 0.8766\n",
"1875/1875 [==============================] - 1s 718us/step - loss: 0.3351 - accuracy: 0.8782\n",
"Epoch 4/10\n",
"1875/1875 [==============================] - 1s 755us/step - loss: 0.3125 - accuracy: 0.8852\n",
"1875/1875 [==============================] - 1s 716us/step - loss: 0.3147 - accuracy: 0.8829\n",
"Epoch 5/10\n",
"1875/1875 [==============================] - 2s 965us/step - loss: 0.2955 - accuracy: 0.8908\n",
"1875/1875 [==============================] - 1s 713us/step - loss: 0.2960 - accuracy: 0.8908\n",
"Epoch 6/10\n",
"1875/1875 [==============================] - 2s 905us/step - loss: 0.2808 - accuracy: 0.8963\n",
"1875/1875 [==============================] - 1s 715us/step - loss: 0.2797 - accuracy: 0.8963\n",
"Epoch 7/10\n",
"1875/1875 [==============================] - 2s 1ms/step - loss: 0.2696 - accuracy: 0.9004\n",
"1875/1875 [==============================] - 1s 715us/step - loss: 0.2672 - accuracy: 0.9008\n",
"Epoch 8/10\n",
"1875/1875 [==============================] - 2s 895us/step - loss: 0.2587 - accuracy: 0.9039\n",
"1875/1875 [==============================] - 1s 723us/step - loss: 0.2580 - accuracy: 0.9044\n",
"Epoch 9/10\n",
"1875/1875 [==============================] - 2s 925us/step - loss: 0.2479 - accuracy: 0.9068\n",
"1875/1875 [==============================] - 1s 733us/step - loss: 0.2470 - accuracy: 0.9067\n",
"Epoch 10/10\n",
"1875/1875 [==============================] - 2s 893us/step - loss: 0.2406 - accuracy: 0.9112\n",
"313/313 [==============================] - 0s 555us/step - loss: 0.3496 - accuracy: 0.8790\n",
"1875/1875 [==============================] - 1s 700us/step - loss: 0.2406 - accuracy: 0.9097\n",
"313/313 [==============================] - 0s 480us/step - loss: 0.3465 - accuracy: 0.8782\n",
"\n",
"Model trained with test accuracy: 0.8790000081062317\n"
"Model trained with test accuracy: 0.8781999945640564\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/Users/mehrinkiani/mambaforge/envs/py310/lib/python3.10/site-packages/keras/src/engine/training.py:3000: UserWarning: You are saving your model as an HDF5 file via `model.save()`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')`.\n",
"/Users/mehrinkiani/mambaforge/envs/modelscan/lib/python3.10/site-packages/keras/src/engine/training.py:3000: UserWarning: You are saving your model as an HDF5 file via `model.save()`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')`.\n",
" saving_api.save_model(\n"
]
}
Expand All @@ -119,16 +150,16 @@
},
{
"cell_type": "code",
"execution_count": 5,
"execution_count": 6,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"1/1 [==============================] - 0s 42ms/step\n",
"1/1 [==============================] - 0s 34ms/step\n",
"\n",
"The model predicts: ['Ankle boot', 'Pullover', 'Trouser'] with probabilities: [ 98.948 99.926 100. ]\n",
"The model predicts: ['Ankle boot', 'Pullover', 'Trouser'] with probabilities: [ 95.851 99.163 100. ]\n",
"\n",
"The true labels are ['Ankle boot', 'Pullover', 'Trouser']\n"
]
Expand All @@ -153,15 +184,22 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"# Scan the Safe Model\n",
"# Scan Safe Model\n",
"\n",
"\n",
"The scan results include information on the files scanned, and any issues if found. For the safe model scanned, modelscan finds no model serialization attacks, as expected.\n",
"\n",
"### ModelScan Settings File\n",
"- If you have created a settings file with default name and location (`modelscan-settings.toml`) it would automatically be used when scanning a model. \n",
"\n",
"- If you want to use a specific name and/or location of settings file, please specify it with `--settings-file` as outlined in the next cell. \n",
"\n",
"The scan results include information on the files scanned, and any issues if found. For the safe model scanned, modelscan finds no model serialization attacks, as expected."
"- If you have not created any settings file, ModelScan will scan using default settings. "
]
},
{
"cell_type": "code",
"execution_count": 6,
"execution_count": 7,
"metadata": {},
"outputs": [
{
Expand All @@ -170,15 +208,19 @@
"text": [
"No settings file detected at /Users/mehrinkiani/Documents/modelscan/notebooks/modelscan-settings.toml. Using defaults. \n",
"\n",
"Scanning /Users/mehrinkiani/Documents/modelscan/notebooks/KerasModels/safe_model.h5 using modelscan.scanners.H5LambdaDetectScan model scan\n",
"\n",
"\u001b[34m--- Summary ---\u001b[0m\n",
"\n",
"\u001b[32m No issues found! 🎉\u001b[0m\n"
"\u001b[32m No issues found! 🎉\u001b[0m\n",
"\n",
"\u001b[34m--- Skipped --- \u001b[0m\n",
"\n",
"Total skipped: \u001b[1;36m1\u001b[0m - run with --show-skipped to see the full list.\n"
]
}
],
"source": [
"#!modelscan -p ./KerasModels/safe_model.h5 --settings-file my-modelscan-settings.toml\n",
"!modelscan -p ./KerasModels/safe_model.h5"
]
},
Expand All @@ -193,7 +235,7 @@
},
{
"cell_type": "code",
"execution_count": 7,
"execution_count": 8,
"metadata": {},
"outputs": [
{
Expand Down Expand Up @@ -222,7 +264,7 @@
},
{
"cell_type": "code",
"execution_count": 8,
"execution_count": 9,
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -244,7 +286,7 @@
},
{
"cell_type": "code",
"execution_count": 9,
"execution_count": 10,
"metadata": {},
"outputs": [
{
Expand All @@ -255,9 +297,9 @@
"aws_secret_access_key=<aws_secret_key>\n",
"aws_access_key_id=<access_key_id>\n",
"aws_secret_access_key=<aws_secret_key>\n",
"1/1 [==============================] - 0s 35ms/step\n",
"1/1 [==============================] - 0s 26ms/step\n",
"\n",
"The model predicts: ['Ankle boot', 'Pullover', 'Trouser'] with probabilities: [ 98.948 99.926 100. ]\n",
"The model predicts: ['Ankle boot', 'Pullover', 'Trouser'] with probabilities: [ 95.851 99.163 100. ]\n",
"\n",
"The true labels are ['Ankle boot', 'Pullover', 'Trouser']\n"
]
Expand All @@ -284,16 +326,23 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"# Scan the Unsafe Model\n",
"# Scan Unsafe Model\n",
"\n",
"The scan results include information on the files scanned, and any issues if found. In this case, a critical severity level issue is found in the unsafe model scanned. \n",
"\n",
"modelscan also outlines the found operator(s) and module(s) deemed unsafe. "
"ModelScan also outlines the found operator(s) and module(s) deemed unsafe. \n",
"\n",
"### ModelScan Settings File\n",
"- If you have created a settings file with default name and location (`modelscan-settings.toml`) it would automatically be used when scanning a model. \n",
"\n",
"- If you want to use a specific name and/or location of settings file, please specify it with `--settings-file` as outlined in the next cell. \n",
"\n",
"- If you have not created any settings file, ModelScan will scan using default settings."
]
},
{
"cell_type": "code",
"execution_count": 10,
"execution_count": 11,
"metadata": {},
"outputs": [
{
Expand Down Expand Up @@ -327,6 +376,7 @@
}
],
"source": [
"#!modelscan -p KerasModels/unsafe_model.h5 --settings-file my-modelscan-settings.toml\n",
"!modelscan -p KerasModels/unsafe_model.h5"
]
},
Expand All @@ -349,7 +399,7 @@
},
{
"cell_type": "code",
"execution_count": 11,
"execution_count": 12,
"metadata": {},
"outputs": [
{
Expand All @@ -359,7 +409,7 @@
"No settings file detected at /Users/mehrinkiani/Documents/modelscan/notebooks/modelscan-settings.toml. Using defaults. \n",
"\n",
"Scanning /Users/mehrinkiani/Documents/modelscan/notebooks/KerasModels/unsafe_model.h5 using modelscan.scanners.H5LambdaDetectScan model scan\n",
"\u001b[1m{\u001b[0m\u001b[32m\"modelscan_version\"\u001b[0m: \u001b[32m\"0.5.0\"\u001b[0m, \u001b[32m\"timestamp\"\u001b[0m: \u001b[32m\"2024-01-25T17:56:36.776966\"\u001b[0m, \n",
"\u001b[1m{\u001b[0m\u001b[32m\"modelscan_version\"\u001b[0m: \u001b[32m\"0.5.1\"\u001b[0m, \u001b[32m\"timestamp\"\u001b[0m: \u001b[32m\"2024-02-06T11:00:43.503049\"\u001b[0m, \n",
"\u001b[32m\"input_path\"\u001b[0m: \n",
"\u001b[32m\"/Users/mehrinkiani/Documents/modelscan/notebooks/KerasModels/unsafe_model.h5\"\u001b[0m, \n",
"\u001b[32m\"total_issues\"\u001b[0m: \u001b[1;36m1\u001b[0m, \u001b[32m\"summary\"\u001b[0m: \u001b[1m{\u001b[0m\u001b[32m\"total_issues_by_severity\"\u001b[0m: \u001b[1m{\u001b[0m\u001b[32m\"LOW\"\u001b[0m: \u001b[1;36m0\u001b[0m, \u001b[32m\"MEDIUM\"\u001b[0m: \n",
Expand Down Expand Up @@ -389,7 +439,7 @@
],
"metadata": {
"kernelspec": {
"display_name": "Python 3.10.13 ('py310')",
"display_name": "Python 3.10.13 ('modelscan')",
"language": "python",
"name": "python3"
},
Expand All @@ -408,7 +458,7 @@
"orig_nbformat": 4,
"vscode": {
"interpreter": {
"hash": "bd638e2064d9001d4ca93bc8e56e039dad230900dd235e8a6196f1614960903a"
"hash": "4dc76e75b8580a33bdc5a5a988b23a4143f70625528e602e356063408d11834c"
}
}
},
Expand Down
Loading

0 comments on commit 79f6c5b

Please sign in to comment.