How to use selections?

[Eisbrenner]

Hi, I like to plot a selection of my data, but I can't figure out how that works with the viz.heatmap.

So the following works

df = vaex.open("df.hdf5")
df = df[df.depth > 200]
df = df[df.maxage > 24 * 60 * 60 * 365.25 * 6]
df.viz.heatmap(
    x="i",
    y="j",
    what=np.log(vaex.stat.count() + 1),
    limits="99.7%",
)

but I think this runs through the data 3 times? Maybe not, but the docs may tell me not to do it this way? I really have trouble with the docs on this topic.

Anyway, I saw the select method and I think the docs argue that this should be better (?), but I can't get it to work:

df = vaex.open("df.hdf5")
df.select(df.maxage > 24 * 60 * 60 * 365.25 * 6, name="maxage")
df.select(df.depth > 200, name="depth")
# fails from here on
df.viz.heatmap(x="i",y="j",selection=["(maxage)&(depth)"],what=np.log(vaex.stat.count() + 1),limits="99.7%")
df.viz.heatmap(x="i",y="j",selection=[True],what=np.log(vaex.stat.count() + 1),limits="99.7%")
df.viz.heatmap(x="i",y="j",selection=True,what=np.log(vaex.stat.count() + 1),limits="99.7%")
df["(maxage)&(depth)"].viz.heatmap(x="i",y="j",what=np.log(vaex.stat.count() + 1),limits="99.7%")
df.viz.heatmap(x="i",y="j",selection=["(df.maxage > 24 * 60 * 60 * 365.25 * 6)&(df.depth > 200)"],what=np.log(vaex.stat.count() + 1),limits="99.7%")

the last one seems completely miss guided, it seems in that case maxage and depth are not defined anymore.

[KeyError: "Unknown variables or column: '(df.maxage > 24 * 60 * 60 * 365.25 * 6)&(df.depth > 200)'"]()

The others mostly throw an error related to some recursion

[[... skipping similar frames: NodeVisitor.visit at line 407 (652 times), Expression.dependencies at line 680 (327 times), Expression.variables at line 708 (327 times), Selection.dependencies at line 51 (326 times), NodeTransformer.generic_visit at line 492 (326 times), Expression.variables.<locals>.record at line 697 (326 times), translate at line 603 (326 times), Translator.visit_Name at line 506 (326 times)]]()
# and
[RecursionError: maximum recursion depth exceeded]()

So, does one use df.select here? Should I not use that?

Best
Ezra

[JovanVeljanoski]

Hi, here are some examples, i trust examples would be more valuable than explaining things in words:

# Get example data
import vaex
df = vaex.example()


# option 1: using selections
df.select('x > 5')
df.viz.heatmap('x', 'y', selection='default', f='log1p')

# option 2: using selections with custom names
df.select('x > 5', name='my_sel')
df.viz.heatmap('x', 'y', selection='my_sel', f='log1p')

# option 3: with a string expression directly in the viz method
df.viz.heatmap('x', 'y', selection='y < 0', f='log1p')

# option 4: as a vaex expression directly in the viz method
df.viz.heatmap('x', 'y', selection='y < 0', f='log1p')

There are some advanced plotting examples in the docs.

Hope this helps.
J.

[JovanVeljanoski]

It is quite hard to give an example for everything that vaex can do, since there are so many features.. I guess that is why these discussion board exist.

Good point about docstrings tho, I will try to fix this as soon as I can.

[JovanVeljanoski]

Of course, you do not have to use df.select(). If you find it better for your use-case to build a boolean expression some other way with some other logic, and pass that expression to the selection argument in any method that supports it, that is fine!

[Eisbrenner]

I actually find the df.select(condition, mode='and/or/xor') quite neat, so I will try that instead of the string concatenation.

[JovanVeljanoski]

The aggregators have a selection argument (e.g. vaex.agg.mean(.., selection='default').

If you want to apply it on the entire dataframe, then better to just use a filter, i.e. df[df.x > 5].groupby('y')

[Eisbrenner]

Thanks for answering that too, I thought I had the answer and edited the question but have continued to struggle since then!