Atom: where should we try to go

[MatthieuDartiailh]

Here are some of my opinions on things we could try to change in Atom. They may not all be feasible and are definitively backward incompatible for some. I would like to have some opinions on those or other things I did not list:

Members

De-correlate validation and access rules

I have found annoying on multiple occasion that there is strong link, at least
in the Member subclasses, between how we validate a value and how it can be
accessed. Those should be orthogonal concerns.

De-correlate validation and coercion

To me coercion should be an action that is available any time the validation
fails and not be strongly linked to the type of validation we carry.

De-correlate type check and value checking

Another point I have been considering if it would be possible to split the type
checking part of the logic and the value checking part of the logic. This would
allow to simply the range validation mode for example.

Removing Member subclasses

All in all I am in favor of removing all Member subclasses since I consider
that basing any decision on the class of a member instead of inspecting the
modes is wrong. To make this feasible it may desirable to improve the interface
to set and get members, in particular by providing a more strongly typed
interface.

Removing dead members

Some members seem of little value to me and removing them may mean we get more
space for some modes.

• Delegate: I am not even sure this member actually work
• Property: this one is weird and it is really hard to reason about notifications
• Signal: this one is rather orthogonal to the other members and even in Enaml
  it is not used that uses a different class of the same name.

Atom UX

Make Atom more similar to a dataclass

When Atom was created dataclasses were not a thing, but they are now quite popular
and provide some nice default behavior and integrate very well with IDE.

For example, we could get autocompletion for __init__ without actually writing
an implementation. Similarly having the possibility generate optimized implementation
for eq, hash, etc may be interesting.

This would also require to improving support for creating Atom subclasses
using type annotations. Currently we can infer members but not tune default
values when a callable is required, specify metadata or select access restrictions
(ReadOnly, Constant).

Support for Generic[T] runtime checking when T is known

Since the strong point of Atom is perform runtime validation it would be valuable
to specialize this validation for Generic subclasses. For example in the following
we should be able to enforce that we get an int.

class A(Atom, Generic[T]):
    a: T

i = A[int]()
i.a = "a"  # Should raise

Support for immutable mapping

One nice feature in Atom is the ability to have immutable object. However often
one needs to store a Mapping and currently we cannot enforce immutability since
we can only use dict.

Notifications

Reduce cross-talk between observers

Currently an exception occurring in any observer handler prevent the others
from running. I have often been bitten by this in enaml where the whole UI
is broken because of an unrelated exception.

Preventing observer to raise does not seem very practical to me, however
if we limit ourselves to Python 3.11+ we could use ExceptionGroup to pack
together teh exception raised in all observers.

Custom container with user defined notification logic

While building UI models, I often needed container able to notify that a
change occurred. ContainerList is bad way to do this from my point of view
because it is often too granular.

What I would like to be able to do is define custom container and have a
built-in way to connect member observers to notification originating from the
container. For such a mechanism to be meaningful we would need to provide
a default payload for container notifications.

? make it possible to have un-observable Atom ?

Because the observers take some space on each atom instance it may be interesting
to be able to disable the notification system. This could be achieve using
a new base class.

[frmdstryr]

Could you provide examples and/or thoughts on how de-correlating would work?

Removing Member subclasses

From a performance I'd rather delete all modes and make dedicated member types for each.. since all the code I've seen never modifies them anyways after initially setting them but each mode requires jumptables.

Reduce cross-talk

It wouldn't be hard to have a custom exception that collects the errors. Since the change_types flag wastes space (due to padding) it could be extended to pass in a separate mode on how errors should be handled.

user defined notification logic

I think this can be done with events/signals?

un-observable Atom

The observer pool pointer could be moved into the "slots" and be handled by a custom member. Or notification could be delegated to members themselves (since the pool currently has to copy each member name).

[MatthieuDartiailh]

Could you provide examples and/or thoughts on how de-correlating would work?

This is related to the idea of moving away from having member subclasses that artificially tie together modality of access and types.

Removing Member subclasses

From a performance I'd rather delete all modes and make dedicated member types for each.. since all the code I've seen never modifies them anyways after initially setting them but each mode requires jumptables.

Having dedicated subclasses would lead to quite a bit of duplication and make customization harder IMO.

Reduce cross-talk

It wouldn't be hard to have a custom exception that collects the errors. Since the change_types flag wastes space (due to padding) it could be extended to pass in a separate mode on how errors should be handled.

It requires some care to be sure we do not leak traceback that keep alive many objects. But I agree we "just" need to do it.

user defined notification logic

I think this can be done with events/signals?

My point here is that I would like a member like ContainerList (i.e. that tie/untie the observers to the container stored on the member) but for user defined container. Such a container could internally use an Event or Signal but the automatic management requires some work in Atom.

un-observable Atom

The observer pool pointer could be moved into the "slots" and be handled by a custom member. Or notification could be delegated to members themselves (since the pool currently has to copy each member name).

Currently an Atom object with a single member occupies 64 bytes while an equivalent object using slots (directly or through dataclass) uses 40. We need extra information so there will always be an overhead but I would like to reduce it and the observing bit felt that something that could be optional.

[frmdstryr]

Just found https://github.com/python/cpython/blob/main/Objects/object_layout.md it looks like every object has extra stuff in the pre-header.

Something is still off with the size of the ObserverPoool pointer because the catom layout looks the same as python 3.12.

I'd expect there to only be an 8 byte difference instead of 24. It exactly matches the broken test in the PR I opened earlier today.

[MatthieuDartiailh]

One thing I noticed in that document is the space related to weakref. It seems it is now built-in in all the objects (without requiring a dedicated slot). If it is so there is no need anymore for magic to enable it for atom objects and atomref may be retired in the future if the new mechanism has a similar efficiency.

[frmdstryr]

I noticed that as well. Perhaps atomref could deprecated and just replaced with weakref.ref.

One thing I thought about last night that could potentially save a lot of space (at least for enaml-web) would be to have an option to make Atom somehow lazily & dynamically allocate room for slots as needed.

Eg. Instead of allocating an array of all slots up front, make the slots point to a dict or something that only lazily adds room for slots if member differs from the the default value (based on a member tag setting). I have like 20 slots but often only 2 or 3 get changed and the rest are just pointers to empty strings doing nothing but wasting space. But I still need notifications so I can't use something other than Atom...