Make it possible to point to a local deployment of Fuzz Introspector (#…

…145)

--> This builds on top of
#141 so will rebase once that
has landed.

Adds ability to use a local deployment of Fuzz Introspector's web api
instead of using https://introspector.oss-fuzz.com/

This has several benefits. Deploying a local version and pointing
oss-fuzz-gen to this will:
- For larger experiments avoid any timeout issues incurred by too many
requests to `introspector.oss-fuzz.com`. Local will have no issues
responding to requests.
- Testing new things will go faster as it's not necessary to deploy a
new version of the API by way of `introspector.oss-fuzz.com` before
oss-fuzz-gen can use it.
- If for some reason some limitations or regressions are landed on
`introspector.oss-fuzz.com` then oss-fuzz-gen testing against
introspector isn't blocked because using the local version (where the
regression can be fixed) gives the exact same features as
`introspector.oss-fuzz.com`.
- it's easier to test against data from different dates.
`introspector.oss-fuzz.com` is updated once a day against the latest
data from oss-fuzz. However, a local version can create a DB against
oss-fuzz's state at any given date.


To test this:

1. Follow the instructions here to set up a local webapp:
https://github.com/ossf/fuzz-introspector/tree/main/tools/web-fuzzing-introspection#launching-a-local-version

2. Run in oss-fuzz-gen: `python3 -m data_prep.introspector tinyxml2 -m 5
-o test5 -l`

---------

Signed-off-by: David Korczynski <[email protected]>

data_prep/introspector.py

@@ -37,13 +37,28 @@
 
 TIMEOUT = 10
 MAX_RETRY = 5
-INTROSPECTOR_ENDPOINT = 'https://introspector.oss-fuzz.com/api'
-INTROSPECTOR_CFG = f'{INTROSPECTOR_ENDPOINT}/annotated-cfg'
-INTROSPECTOR_FUNCTION = f'{INTROSPECTOR_ENDPOINT}/far-reach-but-low-coverage'
-INTROSPECTOR_SOURCE = f'{INTROSPECTOR_ENDPOINT}/function-source-code'
-INTROSPECTOR_XREF = f'{INTROSPECTOR_ENDPOINT}/all-cross-references'
-INTROSPECTOR_TYPE = f'{INTROSPECTOR_ENDPOINT}/type-info'
-INTROSPECTOR_FUNC_SIG = f'{INTROSPECTOR_ENDPOINT}/function-signature'
+INTROSPECTOR_ENDPOINT = ''
+INTROSPECTOR_CFG = ''
+INTROSPECTOR_FUNCTION = ''
+INTROSPECTOR_SOURCE = ''
+INTROSPECTOR_XREF = ''
+INTROSPECTOR_TYPE = ''
+INTROSPECTOR_FUNC_SIG = ''
+
+
+def _set_introspector_endpoints(endpoint):
+  """Sets URLs for Fuzz Introspector endpoints to local or remote endpoints."""
+  global INTROSPECTOR_ENDPOINT, INTROSPECTOR_CFG, INTROSPECTOR_FUNCTION, INTROSPECTOR_SOURCE, INTROSPECTOR_XREF, INTROSPECTOR_TYPE, INTROSPECTOR_FUNC_SIG
+
+  INTROSPECTOR_ENDPOINT = endpoint
+  logging.info(f'Fuzz Introspector endpoint set to {INTROSPECTOR_ENDPOINT}')
+
+  INTROSPECTOR_CFG = f'{INTROSPECTOR_ENDPOINT}/annotated-cfg'
+  INTROSPECTOR_FUNCTION = f'{INTROSPECTOR_ENDPOINT}/far-reach-but-low-coverage'
+  INTROSPECTOR_SOURCE = f'{INTROSPECTOR_ENDPOINT}/function-source-code'
+  INTROSPECTOR_XREF = f'{INTROSPECTOR_ENDPOINT}/all-cross-references'
+  INTROSPECTOR_TYPE = f'{INTROSPECTOR_ENDPOINT}/type-info'
+  INTROSPECTOR_FUNC_SIG = f'{INTROSPECTOR_ENDPOINT}/function-signature'
 
 
 def _construct_url(api: str, params: dict) -> str:
@@ -469,6 +484,11 @@ def _parse_arguments() -> argparse.Namespace:
                       type=str,
                       default='',
                       help='Output directory.')
+  parser.add_argument('-e',
+                      '--endpoint',
+                      type=str,
+                      default='https://introspector.oss-fuzz.com/api',
+                      help='Fuzz Introspecor API endpoint.')
 
   args = parser.parse_args()
   return args
@@ -481,6 +501,8 @@ def _parse_arguments() -> argparse.Namespace:
   if args.out:
     os.makedirs(args.out, exist_ok=True)
 
+  _set_introspector_endpoints(args.endpoint)
+
   try:
     oss_fuzz_checkout.clone_oss_fuzz()
     oss_fuzz_checkout.postprocess_oss_fuzz()