How best to handle sessions behind web API

[oehm-smith]

Howdy, i'm newish to Python development though have been a JavaScript / NodeJS / Java dev for many years.

I've developed a stand alone .py app that works a treat.

Would this library work for web apps that sit behind an RESTful API? You'd need to manage the state somehow. Would creating a new AgentMemory be enough? And how do you maintain a reference to each one - would dump() and load() be quick enough (say in an SQLLite DB). Or is there a better way?

Cheers!

[KennyVaneetvelde]

You guessed it! This is what dump() and load() were designed for.

Since the general assumption is that an API itself is stateless (in other words: an API does not track a conversation in-memory) you have a few options.

Option 1 (This is what the APIs of OpenAI, Groq, ... do)

• dump() the history & return it together with the response from the API, this is what for example the OpenAI API does.
• The user sends back the history together with a new message, the API now does a load()

Option 2:

Basically the same as above, but you keep track of the state in an sqllite DB and instead of sending the history to&from the user you save/load it from the DB and give the user some kind of a user ID that you can use to track instead.

Both have some implications
With option 1, the user could modify the history, this can be good or bad, depending on what you want to achieve or what you want to block off, but then again, option 1 doesn't need you to keep track of anything at all in your backend, so that's nice.

In either case, you'll be using dump() & load().

Keep in mind though: The system prompt is handled separately from the rest of the history, since typically you don't want to give away your system prompt. You still can do so, if you have a use case for it, though. There is just no simple dump/load mechanism for it, but I don't think that's ever needed since initializing your system prompt is easy enough.

I haven't gotten around to creating an API example yet, but it's coming!

Hopefully this was helpful to you! Let me know if it worked or if you're stuck or whatever!

[KennyVaneetvelde]

Microservices make me sad 😢
They are overkill in 99% of all use cases (hence why many companies are starting to have a second look at monolithic approaches)

Anywaaays tiny rant out of the way, nowadays if I need to build a backend in Python, I always go for FastAPI+SQLalchemy+Pydantic - SQLlite is viable here as its definitely not a toy database anymore, but if it's a customer-facing application I'll probably go for a digitalocean droplet and install MySQL on it, this does mean you have to take care of your own backups n stuff, but you can also opt for a managed DB (those go from $15/month but you don't have to worry about much)

So yeah, all depends on what you need...

I do highly recommend digitalocean, if you make a new account, you get $200 to play around with for 2 months straight on anything you like, so it allows you to experiment quite a bit!

And of course, you can also check out firebase, and instead of a full API, use firebase functions, It's going more toward the microservices side again but as far as that whole shebang goes, firebase is still OK and not too overwhelming

But personally, I switched from firebase to digitalocean, and occasionally I'll only use firebase still for users/auth

[oehm-smith]

Nice tips, thanks.

I'm still not clear on the services part of my question.

Java EE, .Net or JS frameworks like NestJS support services by managing the lifecycle of components, including that they essentially sit there and poll for events and manage running all the services as if in a continuous loop.

Does python have an equivalent, or are people happy to write their own:

while True:
    if some_escape_code then:
        break
    ...
done

(Or use generators with their .close())

I understand that for what we are discussing here, your architecture is sufficient. I was just hoping to get some opinions from a seasoned Python dev given I'm trying to break in to this domain (AI will start taking away Software Engineering roles, but AI in itself will be a source of employment for some time to come, and Python is the language most synonymous with AI ML).

[oehm-smith]

I see that FastAPI has a dependency injection mechanism and any injected services can be written to maintain state. But like you suggest, a separate DB would be the best way to manage the app state properly.

Further to this, it has Lifespans for managing lifecyle - https://fastapi.tiangolo.com/advanced/events/#lifespan

[oehm-smith]

Oh, and to dot the last eye and cross the last tee - the while True: loop is achieved by placing the app under fastApi's control:

fastapi dev app/routes.py

[KennyVaneetvelde]

Ahh I see I don't really have that much experience with those specific service frameworks but yes, FastAPI's lifecycle events are very useful!

But indeed an API should be stateless generally