How to effectively track real-time model metrics?

[leepand]

This is a great project, but how can we effectively track model metrics in real-time projects? I have noticed that using experiment.persist(if_exists="replace") during persistence can be time-consuming, which may cause delays in real-time model services.

[elehcimd]

Hi @leepand, Thank you for testing it! You can use the streaming API to monitor metrics in real-time: https://mltraq.com/advanced/datastream/ . Let me know if it works for your use case, happy to delve deeper into it.

[leepand]

Thank you for your prompt response. However, the streaming API may not align with our use case. Our project aims to provide services in the following manner. Here is a simple example:

from flask import Flask, request
app = Flask(name)

prediction function

def ValuePredictor(to_predict_list):
to_predict = np.array(to_predict_list).reshape(1, 12)
loaded_model = pickle.load(open("model.pkl", "rb"))
result = loaded_model.predict(to_predict)
return result[0]

@app.route('/result', methods = ['POST'])
def result():
if request.method == 'POST':
to_predict_list = request.form.to_dict()
to_predict_list = list(to_predict_list.values())
to_predict_list = list(map(int, to_predict_list))
result = ValuePredictor(to_predict_list)
if int(result)== 1:
prediction ='Income more than 50K'
else:
prediction ='Income less that 50K'

     **with experiment.run() as run:
           run.fields.x = to_predict_list
           run.fields.request_id="21146110_1"
    experiment.persist(if_exists="replace")
   ............**

[leepand]

Currently, we achieve real-time persistence of our key-value (KV) streaming data using this method (https://github.com/grantjenks/python-diskcache), which meets our performance requirements. It would be perfect if we could also utilize MLtraq to handle the corresponding table data.

[elehcimd]

Hah! MLtraq targets experimentation, and it's not designed to serve models/tracking around it. It seems you're only lacking its serialization capabilities. You can keep using python-diskcache, with MLtraq handling serialization/deserialization. Here's a quick example:

from mltraq.storage.serialization import serialize, deserialize
from mltraq import Bunch
import pandas as pd

data = Bunch()
data.table = pd.DataFrame({'c1': [1,2,3], 'c2': [4,5,6]})
data.request_id = "21146110_1"
blob = serialize(data)
print(deserialize(blob).request_id)

The supported data types are documented at https://mltraq.com/advanced/storage

[leepand]

Got it. Thank you for your suggestion.