Adding spherized profiles

[gwaybio]

I spherize all plates of batch 1 data using all DMSO profiles as a reference. I apply feature selection to the full dataframe of concatenated level 4a data, and output the spherized data.

I also needed to change the name of a script from profile.py to profile_cells.py. This solves the issues I described in #59 (comment)

Merge steps

• I sanity check that all batch 2 plates were processed (prob should have done in Adding batch 2 data #58 ...)
• I make some minor modifications to this PR
• @shntnu reviews and we merge this PR once he approves

[gwaybio]

Note, I didn't perform any sanity checks in this data

[gwaybio]

I added batch two spherized profiles after merging #60

[gwaybio]

In #48 (comment), @shntnu noted:

cytomining/pycytominer#128. If the default value of epsilion=1e-6 is fine, then we needn't fix that issue right now. How would we know whether it is fine or not? I suppose we can just do it very crudely and empirically for now: do the results improve similar to what we've seen in past analysis by Ted et al.?

If they do, then epsilion=1e-6 is fine and there's nothing to be done here.
If they don't improve then we need to think harder about the plan
Update: I just noticed #60 so you're all set to figure out whether there's anything to be done here.

I agree that an empirical test that reproduces the improvement Ted saw in regards to non-spherized vs. spherized data would make us all set. However, I don't think we do it in this pull request, and not even in this repo. Instead, @AdeboyeML can take these profiles and run them through the pipeline he created in https://github.com/broadinstitute/lincs-profiling-comparison.

So, I propose the following:

1. I sanity check that all batch 2 plates were processed (prob should have done in Adding batch 2 data #58 ...)
2. I make some minor modifications to this PR
3. @shntnu reviews and we merge this PR once he approves
4. @AdeboyeML runs these data through his replicate reproducibility assessment pipeline
5. If we see similar improvement as Ted, then we're good. If not, we need to adjust epsilon to what Mohammad used (this value is around somewhere...)

---

edit, i'll add the PR-specific steps to the beginning of this PR

[gwaybio]

@shntnu - this is now ready for your eyes, when you get a chance

[shntnu]

I've not read the whole thing yet, but I wonder if you should mention the existence of spherized data somewhere here.

[gwaybio]

ah, good point

[shntnu]

LGTM

Did to mean to propose the PR merge after this step, not before?

@AdeboyeML runs these data through his replicate reproducibility assessment pipeline

[gwaybio]

Did to mean to propose the PR merge after this step, not before?

we merge first, then Adeniyi checks using data from the merge

[shntnu]

we merge first, then Adeniyi checks using data from the merge

Got it. My concern was bloating the repo in case you need to reprocess. But I trust your judgment in figuring of what order works best.

Excited to have this in the repo!!

PS – if you do need to end up replacing, I'd recommend actually deleting the files as I did here
jump-cellpainting/pilot-cpjump1-data#9 (comment)

[gwaybio]

PS – if you do need to end up replacing, I'd recommend actually deleting the files as I did here
jump-cellpainting/pilot-cpjump1-data#9 (comment)

Awesome, this is good to keep in mind.

We might at some point also consider moving from gitLFS to dvc. It was super easy to get setup, and plays very nicely with AWS. I did this in the grit-benchmark repo ( in broadinstitute/grit-benchmark#28)

In the most recent commit, I added a bunch of comments to two different README files. We might want to edit them before first official release, but we can open a new, documentation-focused PR then. I am going to merge!

[shntnu]

The notebook says

Here, we load in all normalized profiles (level 4a) data across all plates and apply a spherize transform using the DMSO profiles as the background distribution.

but it should say

Here, we load in all normalized profiles (level 4a) data across all plates, apply the standard set of feature selection operations, and then apply a spherize transform using the DMSO profiles as the background distribution.

It's not worth updating anything; I'm just adding a note here for ourselves.

[gwaybio]

good catch. I added #77 so we can make sure to improve (I agree it is not urgent, but someone new could start there (good practice for editing a file using github 😄 ))