Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New cyto tool: create cell locations file #257

Merged
merged 83 commits into from
Apr 7, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
83 commits
Select commit Hold shift + click to select a range
ec7a79d
Init
shntnu Feb 24, 2023
2ee9909
cmdline tool
shntnu Feb 24, 2023
ed132a9
Cleanup
shntnu Feb 24, 2023
8662db1
use fire
shntnu Feb 25, 2023
80698e6
output is optional
shntnu Feb 25, 2023
2da6618
fix sqlite fixture
shntnu Feb 25, 2023
ba54ab2
refactor
shntnu Feb 25, 2023
43b4f58
files are optional
shntnu Feb 26, 2023
63c1b07
Typo
shntnu Feb 26, 2023
2b49280
cleanup
shntnu Feb 26, 2023
69323b0
drop comments
shntnu Feb 26, 2023
7c36fb5
create test fixtures
shntnu Feb 26, 2023
612fc9c
checks
shntnu Feb 26, 2023
2162126
fix paths
shntnu Feb 26, 2023
d64eb11
Use fixtures
shntnu Feb 26, 2023
6e1afa8
yield
shntnu Feb 26, 2023
3cf8bdd
download file if needed
shntnu Feb 26, 2023
f1b581a
typo
shntnu Feb 26, 2023
36c9dca
Update req
shntnu Feb 26, 2023
2df741c
use boto3 session
shntnu Feb 27, 2023
adf635a
Test s3 locations
shntnu Feb 27, 2023
bc2f01d
skip s3 test
shntnu Feb 27, 2023
36de960
dtypes
shntnu Feb 27, 2023
6185578
Add mike's snipper
shntnu Feb 27, 2023
6070491
use mike's format
shntnu Mar 2, 2023
55a4f74
tests pass
shntnu Mar 2, 2023
4716143
use method
shntnu Mar 3, 2023
a77364f
add alternatives
shntnu Mar 3, 2023
7fdd393
refactor mike's code
shntnu Mar 4, 2023
5595abc
cleanup
shntnu Mar 4, 2023
8ad1ccf
better tests
shntnu Mar 4, 2023
4016815
cleanup
shntnu Mar 5, 2023
492f29b
overwrite is an option, other cleanup
shntnu Mar 5, 2023
ea21a45
Update pycytominer/cyto_utils/cell_locations.py
shntnu Mar 27, 2023
487ef2d
add docs
shntnu Mar 27, 2023
599a600
Add deps
shntnu Mar 27, 2023
3549060
Move to setup
shntnu Mar 27, 2023
a628fe5
add fire to deps
shntnu Mar 27, 2023
a6cd67d
use pathlib
shntnu Mar 27, 2023
9c232c9
Merge branch 'cytomining:master' into ss_cell_locations
shntnu Mar 28, 2023
8415627
Add pip install .[cell_locations] (+formatting)
shntnu Mar 28, 2023
b111399
Update docs + fix typo in actions
shntnu Mar 28, 2023
287cc17
Use as module
shntnu Mar 28, 2023
fbdf9aa
Formatting
shntnu Mar 28, 2023
5fa59cf
add cell_locations
shntnu Mar 28, 2023
945e5ce
Merge in SQL
shntnu Mar 28, 2023
e987ccf
Update README.md
shntnu Mar 30, 2023
4f067fa
Update .github/workflows/codecov.yml
shntnu Mar 30, 2023
3d711f7
Update .github/workflows/python-app.yml
shntnu Mar 30, 2023
cda5f76
Update pycytominer/tests/test_data/cell_locations_example_data/test_c…
shntnu Mar 30, 2023
f65b36d
Update pycytominer/tests/test_data/cell_locations_example_data/test_c…
shntnu Mar 30, 2023
03995b8
Update pycytominer/tests/test_cyto_utils/test_cell_locations.py
shntnu Mar 30, 2023
c817626
Update pycytominer/cyto_utils/cell_locations.py
shntnu Mar 30, 2023
72896b9
Update pycytominer/cyto_utils/cell_locations.py
shntnu Mar 30, 2023
59816ff
Update pycytominer/tests/test_cyto_utils/test_cell_locations.py
shntnu Mar 30, 2023
7df9c68
Update pycytominer/tests/test_cyto_utils/test_cell_locations.py
shntnu Mar 30, 2023
52d7589
Address various comment
shntnu Apr 1, 2023
8eaa9ee
To address this warning below:
shntnu Apr 1, 2023
683048d
use sqlalchemy
shntnu Apr 1, 2023
03bc671
More comments and switch to boto3.client
shntnu Apr 1, 2023
1a433e3
Be explicit about anon; fix indentation bug
shntnu Apr 2, 2023
892e627
explicit types
shntnu Apr 2, 2023
76e1f49
Address various comment
shntnu Apr 2, 2023
cf7c94a
Move gitignore entries to the top level
shntnu Apr 2, 2023
5e9ae36
rename files, add docs
shntnu Apr 2, 2023
a55d74a
Upgrade to python 3.10
shntnu Apr 2, 2023
3e5afdd
refactor _load_single_cell
shntnu Apr 2, 2023
8d70fd0
fix type
shntnu Apr 2, 2023
e77c6cf
test on highest build version
shntnu Apr 2, 2023
972e720
explain warning
shntnu Apr 5, 2023
9b4a0bf
s3 is an attribute
shntnu Apr 5, 2023
e05056d
compact check
shntnu Apr 5, 2023
6f2e3b3
Update README.md
shntnu Apr 5, 2023
1e8f1e6
fix typo + more docs
shntnu Apr 5, 2023
3e16209
black cells.py
shntnu Apr 5, 2023
d7b345c
trim code
shntnu Apr 5, 2023
3b5388e
black
shntnu Apr 5, 2023
53b214b
Skip test
shntnu Apr 5, 2023
56d71a8
docs
shntnu Apr 5, 2023
fdebb84
dtypes
shntnu Apr 5, 2023
c037ef9
skip test
shntnu Apr 5, 2023
24caa04
Fix test
shntnu Apr 5, 2023
e68ab4b
Add docs
shntnu Apr 5, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 23 additions & 24 deletions .github/workflows/codecov.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,33 +2,32 @@ name: Code coverage

on:
push:
branches: [ master ]
branches: [master]
pull_request:
branches: [ master ]
branches: [master]

jobs:
run:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@master
- name: Setup Python
uses: actions/setup-python@master
with:
python-version: 3.7
- name: Generate coverage report
run: |
pip install pytest
pip install pytest-cov
pip install -r requirements.txt
pip install .[collate]
pytest --cov=./ --cov-report=xml
- name: Upload coverage to Codecov
uses: codecov/codecov-action@v1
with:
file: ./coverage.xml
files: ./coverage1.xml,./coverage2.xml
directory: ./coverage/reports/
flags: unittests
name: codecov-umbrella
fail_ci_if_error: false
path_to_write_report: ./coverage/codecov_report.gz
- uses: actions/checkout@master
- name: Setup Python
uses: actions/setup-python@master
with:
python-version: 3.9
shntnu marked this conversation as resolved.
Show resolved Hide resolved
- name: Generate coverage report
run: |
pip install pytest
pip install pytest-cov
pip install .[collate,cell_locations]
shntnu marked this conversation as resolved.
Show resolved Hide resolved
pytest --cov=./ --cov-report=xml
- name: Upload coverage to Codecov
uses: codecov/codecov-action@v1
with:
file: ./coverage.xml
files: ./coverage1.xml,./coverage2.xml
directory: ./coverage/reports/
flags: unittests
name: codecov-umbrella
fail_ci_if_error: false
path_to_write_report: ./coverage/codecov_report.gz
38 changes: 20 additions & 18 deletions .github/workflows/python-app.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,33 +5,35 @@ name: Python build

on:
push:
branches: [ master ]
branches: [master]
pull_request:
branches: [ master ]
branches: [master]

jobs:
build:

runs-on: ${{ matrix.os }}
strategy:
matrix:
python-version: [3.7, 3.8, 3.9]
os: [ubuntu-latest, macos-latest]
env:
OS: ${{ matrix.os }}
OS: ${{ matrix.os }}
# This is needed to avoid a warning from SQLAlchemy
# https://sqlalche.me/e/b8d9
# We can remove this once we upgrade to SQLAlchemy >= 2.0
SQLALCHEMY_SILENCE_UBER_WARNING: "1"
shntnu marked this conversation as resolved.
Show resolved Hide resolved
steps:
- uses: actions/checkout@v2
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v2
with:
python-version: ${{ matrix.python-version }}
- uses: actions/checkout@v2
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v2
with:
python-version: ${{ matrix.python-version }}

- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install pytest
if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
pip install .[collate]
- name: Test with pytest
run: |
pytest
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install pytest
pip install .[collate,cell_locations]
- name: Test with pytest
run: |
pytest
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -9,4 +9,5 @@ build
*.sqlite
pycytominer/tests/test_data/collate/backend/**/*.csv
!pycytominer/tests/test_data/collate/backend/**/*master.csv
!pycytominer/tests/test_data/cell_locations_example_data/*.sqlite

40 changes: 39 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,11 +45,12 @@ Since the project is actively being developed, with new features added regularly
# Example:
pip install git+git://github.com/cytomining/pycytominer@2aa8638d7e505ab510f1d5282098dd59bb2cb470
```

### CSV collation

If running your images on a cluster, unless you have a MySQL or similar large database set up then you will likely end up with lots of different folders from the different cluster runs (often one per well or one per site), each one containing an `Image.csv`, `Nuclei.csv`, etc.
In order to look at full plates, therefore, we first need to collate all of these CSVs into a single file (currently SQLite) per plate.
We currently do this with a library called [cytominer-database](https://github.com/cytomining/cytominer-database).
We currently do this with a library called [cytominer-database](https://github.com/cytomining/cytominer-database).

If you want to perform this data collation inside pycytominer using the `cyto_utils` function `collate` (and/or you want to be able to run the tests and have them all pass!), you will need `cytominer-database==0.3.4`; this will change your installation commands slightly:

Expand All @@ -62,6 +63,43 @@ pip install "pycytominer[collate] @ git+git://github.com/cytomining/pycytominer@

If using `pycytominer` in a conda environment, in order to run `collate.py`, you will also want to make sure to add `cytominer-database=0.3.4` to your list of dependencies.

## Creating a cell locations lookup table

The `CellLocation` class offers a convenient way to augment a [LoadData](https://cellprofiler-manual.s3.amazonaws.com/CPmanual/LoadData.html) file with X,Y locations of cells in each image.
The locations information is obtained from a single cell SQLite file.

To use this functionality, you will need to modify your installation command, similar to above:

```bash
# Example for general case commit:
pip install "pycytominer[cell_locations] @ git+git://github.com/cytomining/pycytominer"
```

Example using this functionality:

```bash
shntnu marked this conversation as resolved.
Show resolved Hide resolved
metadata_input="s3://cellpainting-gallery/test-cpg0016-jump/source_4/workspace/load_data_csv/2021_08_23_Batch12/BR00126114/test_BR00126114_load_data_with_illum.parquet"
single_single_cell_input="s3://cellpainting-gallery/test-cpg0016-jump/source_4/workspace/backend/2021_08_23_Batch12/BR00126114/test_BR00126114.sqlite"
augmented_metadata_output="~/Desktop/load_data_with_illum_and_cell_location_subset.parquet"

python \
shntnu marked this conversation as resolved.
Show resolved Hide resolved
-m pycytominer.cyto_utils.cell_locations_cmd \
--metadata_input ${metadata_input} \
--single_cell_input ${single_single_cell_input} \
--augmented_metadata_output ${augmented_metadata_output} \
add_cell_location

# Check the output
shntnu marked this conversation as resolved.
Show resolved Hide resolved

python -c "import pandas as pd; print(pd.read_parquet('${augmented_metadata_output}').head())"

# It should look something like this (depends on the width of your terminal):

# Metadata_Plate Metadata_Well Metadata_Site ... PathName_OrigRNA ImageNumber CellCenters
# 0 BR00126114 A01 1 ... s3://cellpainting-gallery/cpg0016-jump/source_... 1 [{'Nuclei_Location_Center_X': 943.512129380054...
# 1 BR00126114 A01 2 ... s3://cellpainting-gallery/cpg0016-jump/source_... 2 [{'Nuclei_Location_Center_X': 29.9516027655562...
```

## Usage

Using pycytominer is simple and fun.
Expand Down
Loading