Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DM-39808: Add dynamically generated RC2 and rc2_subset deepCoadd source injection pipelines to drp_pipe. #93

Merged
merged 3 commits into from
Nov 27, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 43 additions & 1 deletion SConstruct
Original file line number Diff line number Diff line change
@@ -1,4 +1,46 @@
# -*- python -*-
import os
from lsst.sconsUtils import scripts
from lsst.sconsUtils.state import env
from lsst.sconsUtils.utils import libraryLoaderEnvironment
from SCons.Script import Default

# Python-only package
scripts.BasicSConstruct("drp_pipe", disableCc=True, noCfgFile=True)
# Force shebang and policy to come first so the file first appears in the bin
# directory before it is used. This is required to run on macos.
targetList = (
"version",
"shebang",
"policy",
) + scripts.DEFAULT_TARGETS
scripts.BasicSConstruct(
"drp_pipe", disableCc=True, noCfgFile=True, defaultTargets=targetList
)
PKG_ROOT = env.ProductDir("drp_pipe")

# make deepCoadd injection pipelines for rc2_subset and RC2
rc2_subset_injected_deepCoadd = env.Command(
target=os.path.join(
PKG_ROOT, "pipelines", "HSC", "DRP-RC2_subset+injected_deepCoadd.yaml"
),
source=os.path.join(PKG_ROOT, "pipelines", "HSC", "DRP-RC2_subset.yaml"),
action=" ".join(
[
libraryLoaderEnvironment(),
"make_injection_pipeline -t deepCoadd -r $SOURCE -f $TARGET --overwrite",
]
),
)
RC2_injected_deepCoadd = env.Command(
jtmccann marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As a variable, this should probably have been rc2_injected_deepCoadd. Apologies for not catching it before the merge. It's not a serious issue, but something one of us can address in a future ticket next time we're poking around in this package.

target=os.path.join(
PKG_ROOT, "pipelines", "HSC", "DRP-RC2+injected_deepCoadd.yaml"
),
source=os.path.join(PKG_ROOT, "pipelines", "HSC", "DRP-RC2.yaml"),
action=" ".join(
[
libraryLoaderEnvironment(),
"make_injection_pipeline -t deepCoadd -r $SOURCE -f $TARGET --overwrite",
]
),
)
Default([rc2_subset_injected_deepCoadd, RC2_injected_deepCoadd])
24 changes: 12 additions & 12 deletions pipelines/HSC/DRP-RC2_subset.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -205,14 +205,14 @@ subsets:
- fgcmBuildFromIsolatedStars
- fgcmFitCycle
- fgcmOutputProducts
description: >
description: |
Subset that includes all FGCM tasks.
simpleSingleFrame:
subset:
- isr
- characterizeImage
- calibrate
description: >
description: |
A simplified single frame pipeline to make the quantum graph more readable.
multiVisit:
subset: []
Expand All @@ -231,7 +231,7 @@ subsets:
- consolidateVisitSummary
- skyCorr
- isolatedStarAssociation
description: >
description: |
Set of tasks for complete single frame processing. Because this includes
skyCorr, it requires multiple detectors per visit. Although they duplicate
tasks from the main "stepN" subsets from drp_pipe's DRP-RC2.yaml, we
Expand Down Expand Up @@ -269,7 +269,7 @@ subsets:
- calibrate
- writePreSourceTable
- transformPreSourceTable
description: >
description: |
Per-detector tasks that can be run together to start the DRP pipeline.
These should never be run with 'tract' or 'patch' as part of the data ID
expression if any later steps will also be run, because downstream steps
Expand All @@ -280,7 +280,7 @@ subsets:
- consolidatePreSourceTable
- consolidateVisitSummary
- skyCorr
description: >
description: |
Per-visit tasks that can be run together, but only after the 'step1'.
These should never be run with 'tract' or 'patch' as part of the data ID
expression. skyCorr and FGCM require full visits and 'tract' and 'patch'
Expand All @@ -296,7 +296,7 @@ subsets:
subset:
- isolatedStarAssociation
- gbdesAstrometricFit
description: >
description: |
Per-tract tasks.
gbdesAstrometricFit should be run with explicit 'tract' constraints
essentially all the time, because otherwise quanta will be created for
Expand All @@ -307,7 +307,7 @@ subsets:
- fgcmBuildFromIsolatedStars
- fgcmFitCycle
- fgcmOutputProducts
description: >
description: |
FGCM tasks that must be run with no pipetask multiprocessing
and no data query
nightlyStep2d:
Expand All @@ -322,7 +322,7 @@ subsets:
- nsrcMeasVisit
- TE3
- TE4
description: >
description: |
Per-visit tasks. These should never be run with tract constraints
nightlyStep3:
subset:
Expand Down Expand Up @@ -369,7 +369,7 @@ subsets:
- psfPhotRepStar2
- psfPhotRepStar3
- psfPhotRepStar4
description: >
description: |
Tasks that can be run together, but only after the 'step1' and 'step2'
subsets.
These should be run with explicit 'tract' constraints essentially all the
Expand All @@ -385,7 +385,7 @@ subsets:
nightlyStep4:
subset:
- forcedPhotCcd
description: >
description: |
Tasks that can be run together, but only after the 'step1', 'step2'
and 'step3' subsets
These detector-level tasks should not be run with
Expand All @@ -399,7 +399,7 @@ subsets:
- wPerp
- skyObjectMean
- skyObjectStd
description: >
description: |
Tasks that can be run together, but only after the 'step1', 'step2',
'step3', and 'step4' subsets.
This step includes patch-level aggregation Tasks. These should be
Expand All @@ -414,7 +414,7 @@ subsets:
- analyzeObjectTableSurveyCore
- catalogMatchTract
- analyzeMatchedVisitCore
description: >
description: |
Core (most important), coadd-level plots/metrics from analysis_tools.

Must be run after 'step3' (writeObjectTable). Cannot be run with any
Expand Down
1 change: 1 addition & 0 deletions ups/drp_pipe.table
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ setupRequired(analysis_drp)
setupRequired(analysis_tools)
setupRequired(drp_tasks)
setupRequired(ap_association)
setupRequired(source_injection)
jtmccann marked this conversation as resolved.
Show resolved Hide resolved

# The following is boilerplate for all packages.
# See https://dmtn-001.lsst.io for details on LSST_LIBRARY_PATH.
Expand Down