Skip to content

Commit

Permalink
Improve test_http_input execution speed (#720)
Browse files Browse the repository at this point in the history
* rename test_http_input acceptance test
* improve test speed by mocking the threaded http server
* remove `simulate` from client calls
  • Loading branch information
dtrai2 authored Dec 6, 2024
1 parent ac2b2f8 commit 96f9a26
Show file tree
Hide file tree
Showing 4 changed files with 79 additions and 92 deletions.
1 change: 0 additions & 1 deletion logprep/connector/file/input.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,6 @@
import queue
import threading
import zlib
from logging import Logger
from typing import Callable, TextIO

from attrs import define, field, validators
Expand Down
27 changes: 14 additions & 13 deletions logprep/connector/http/input.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,10 +23,10 @@
host: 0.0.0.0
port: 9000
endpoints:
/firstendpoint: json
/firstendpoint: json
/second*: plaintext
/(third|fourth)/endpoint: jsonl
The endpoint config supports regex and wildcard patterns:
* :code:`/second*`: matches everything after asterisk
* :code:`/(third|fourth)/endpoint` matches either third or forth in the first part
Expand All @@ -38,7 +38,7 @@
add basic authentication for a specific endpoint. The format of this file would look like:
.. code-block:: yaml
:caption: Example for credentials file
:caption: Example for credentials file
:linenos:
input:
Expand All @@ -49,7 +49,7 @@
/second*:
username: user
password: secret_password
You can choose between a plain secret with the key :code:`password` or a filebased secret
with the key :code:`password_file`.
Expand All @@ -60,20 +60,20 @@
- basic auth must only be used with strong passwords
- basic auth must only be used with TLS encryption
- avoid to reveal your plaintext secrets in public repositories
Behaviour of HTTP Requests
^^^^^^^^^^^^^^^^^^^^^^^^^^
* :code:`GET`:
* Responds always with 200 (ignores configured Basic Auth)
* When Messages Queue is full, it responds with 429
* :code:`POST`:
* Responds with 200 on non-Basic Auth Endpoints
* Responds with 401 on Basic Auth Endpoints (and 200 with appropriate credentials)
* When Messages Queue is full, it responds wiht 429
* :code:`ALL OTHER`:
* Responds with 405
"""

Expand Down Expand Up @@ -353,7 +353,7 @@ class Config(Input.Config):
:title: Uvicorn Webserver Configuration
:location: uvicorn_config
:suggested-value: uvicorn_config.access_log: true, uvicorn_config.server_header: false, uvicorn_config.data_header: false
Additionally to the below it is recommended to configure `ssl on the metrics server endpoint
<https://www.uvicorn.org/settings/#https>`_
Expand All @@ -378,8 +378,8 @@ class Config(Input.Config):
"""Configure endpoint routes with a Mapping of a path to an endpoint. Possible endpoints
are: :code:`json`, :code:`jsonl`, :code:`plaintext`. It's possible to use wildcards and
regexps for pattern matching.
.. autoclass:: logprep.connector.http.input.PlaintextHttpEndpoint
:noindex:
.. autoclass:: logprep.connector.http.input.JSONLHttpEndpoint
Expand Down Expand Up @@ -430,6 +430,7 @@ def __init__(self, name: str, configuration: "HttpInput.Config") -> None:
)
schema = "https" if ssl_options else "http"
self.target = f"{schema}://{host}:{port}"
self.app = None
self.http_server = None

def setup(self):
Expand Down Expand Up @@ -462,9 +463,9 @@ def setup(self):
self.metrics,
)

app = self._get_asgi_app(endpoints_config)
self.app = self._get_asgi_app(endpoints_config)
self.http_server = http.ThreadingHTTPServer(
self._config.uvicorn_config, app, daemon=False, logger_name="HTTPServer"
self._config.uvicorn_config, self.app, daemon=False, logger_name="HTTPServer"
)
self.http_server.start()

Expand Down
File renamed without changes.
Loading

0 comments on commit 96f9a26

Please sign in to comment.