Search Results - Finding my ALN and Finding all ALN (#2941)

* Table scrolling, hide modal button, big reformats * Move resultcount out of horizontal scroll, padding * Search results ALNs - my/all ALN columns * A big bundle of linting * Undo djlint autoformat. * Tests for ALN search. Basic & finding_my/all_aln * Break ALN tests into their own class, and into more pieces. * Linting! Yippee! * Bools in ALN test cases - explicit for readability * Comment consistency * End date autofill fix * Search instructions link. * Use a separate PG DB for Census data (#2573) * Add bucket and DB * Lint * Formatting * Duplicate changes to compose-web * Fix naming errors * Try separate database * WIp * Get config test to pass * Make same changes in docker-compose-web --------- Co-authored-by: Alex Steel <[email protected]> * Add functions and tests for deleting `Access` entries and creating corresponding `DeletedAccess` entries (#2849) * Add functions and tests for deleting Access entries and creating corresponding DeletedAccess entries. * Add migration file. * Actually use ACCESS_ROLES in Access. * Type hinting corrections. * Actually check in access_roles.py. * Make tests more readable. * Make tests more readable. * Account for role when using remove_email_from_submission_access. * Add test as per PR comment. * Break up audit views (#2862) * First pass at moving views. * Move imports around to fit new setup. * 2861 update logic to use elecaudits model (#2863) * #2861 Update model mappings tfor federal awards worbook creation * #Refactor code to make it more re-usable * #2861 Code improvement * #2861 More util functions and tests * #2861 Generate workbooks to ensure the updated code is still working as expected * Linting * #2861 Regenerated workbooks to only include dbkeys that are available in census table * #2861 Regenerated workbooks to only include dbkeys that are available in census tables * #2861 Improved logic in light of data discoveries * Added to commit by mistake * #2861 Moved shared logic to a common area * Update backend/census_historical_migration/test_excel_creation.py Co-authored-by: Dan Swick <[email protected]> * Linting * Fix for failing test --------- Co-authored-by: Dan Swick <[email protected]> * Command to run census migration by year (#2868) * Slicing last 2 year digits for dynamic_import * Adding new command * Renaming command to run_migration_for_year * Renaming load_data to load_historic_data_for_year * Renaming loader to historic_data_loader * Lint * Docstrings * Teaking user creation * Utilizing results dict * Lint * Fixing historic_data_migrator --------- Co-authored-by: Phil Dominguez <“[email protected]”> * Sk/hist migration failure models (#2845) * Added FAILED_SACS and CHANGE_RECORDS models * Routing CHANGE_RECORDS to fac db * Initial commit * Migration for FAILED_SACS, CHANGE_RECORDS tables * Registering ELECAUDITHEADER, FAILED_SACS, CHANGE_RECORDS models * Test cases for ELECAUDITHEADER, FAILED_SACS, CHANGE_RECORDS models * Formatted with black * Modified FailedSacs and ChangeRecords table per JMM's request * Modified FailedSacs and ChangeRecords table per JMM's request * Modified FailedSacs and ChangeRecords table per JMM's request * Modified FailedSacs and ChangeRecords table per JMM's request * Modified FailedSacs and ChangeRecords table per JMM's request * Modified FailedSacs and ChangeRecords table per JMM's request * Updating migrations * Updating migrations * Updating migrations * Updated migrations * Modifications based on meeting * Modifications based on meeting * Modifications based on meeting * Modifications based on meeting * Modifications based on meeting * Modifications based on meeting * Modifications based on meeting * Modifications based on meeting * Modifications based on ADR * Modifications based on ADR * Modifications based on ADR * Modifications based on ADR * Implemented changes per ADR review * Implemented changes per ADR review * Implemented changes per ADR review * Implemented changes per ADR review * Implemented changes per ADR review * Implemented changes per ADR review * Implemented changes per ADR review * Implemented changes per ADR review * Implemented changes per ADR review * Implemented changes per ADR review * Implemented changes per ADR review * Fixed lint issue * Redid migrations * Replaced sac with record_id * Replaced sac with record_id * Registering all models * Update backend/dissemination/models.py Co-authored-by: Hassan D. M. Sambo <[email protected]> * Redid migrations * Renamed record_id as report_id * Renamed record_id as report_id * Added databases to route to census-to-gsafac db --------- Co-authored-by: Hassan D. M. Sambo <[email protected]> * 2880 update logic in secondary auditors generator to use census models (#2881) * #2861 Update model mappings tfor federal awards worbook creation * #Refactor code to make it more re-usable * #2861 Code improvement * #2861 More util functions and tests * #2861 Generate workbooks to ensure the updated code is still working as expected * Linting * #2861 Regenerated workbooks to only include dbkeys that are available in census table * #2861 Regenerated workbooks to only include dbkeys that are available in census tables * #2861 Improved logic in light of data discoveries * Added to commit by mistake * #2861 Moved shared logic to a common area * #2880 Updated logic to use census models * #2880 Regenerated workbooks to ensure logic works as intended * Updated migrator command to use or create default user * Updated wrong mapping * More update to improve the api test * search: filter by auditee state (#2888) * 2864 update logic in findings generator to use census models (#2866) * #2861 Update model mappings tfor federal awards worbook creation * #Refactor code to make it more re-usable * #2861 Code improvement * #2861 More util functions and tests * #2861 Generate workbooks to ensure the updated code is still working as expected * Linting * #2861 Regenerated workbooks to only include dbkeys that are available in census table * #2861 Regenerated workbooks to only include dbkeys that are available in census tables * #2861 Improved logic in light of data discoveries * Added to commit by mistake * #2864 Moved shared logic to common area * #2864 Moved shared logic to common area * #2864 Updated logic to use census models and tables * #2864 Regenerated workbooks since this is crutial to ensure the code is still behaving as intended * #2861 Moved shared logic to a common area * Sk/2874 retain null values, 2886 test cases for census models (#2887) * Replace NaN with empty string * Added test cases for census models * Change auditee/auditor certifying official (#2891) * Add GET side of page for changing Auditor Certifying Official. * Add POST side of page for changing Auditor Certifying Official. * Remove stray import. * Add page for changing Auditee Certifying Official. * Use variable for role in template instead of hard-coding. * Typo correction. * 2870 update logic in corrective action plan generator to use census models (#2872) * #2861 Update model mappings tfor federal awards worbook creation * #Refactor code to make it more re-usable * #2861 Code improvement * #2861 More util functions and tests * #2861 Generate workbooks to ensure the updated code is still working as expected * Linting * #2861 Regenerated workbooks to only include dbkeys that are available in census table * #2861 Regenerated workbooks to only include dbkeys that are available in census tables * #2861 Improved logic in light of data discoveries * Added to commit by mistake * #2861 Moved shared logic to a common area * #2870 Updated logic to use census models * #2870 Updated workbooks to ensure the code still behaves as intended * Bug fixed in api_check routine * Fixed bogus row count * 2867 update logic in findings text generator to use census models (#2869) * #2861 Update model mappings tfor federal awards worbook creation * #Refactor code to make it more re-usable * #2861 Code improvement * #2861 More util functions and tests * #2861 Generate workbooks to ensure the updated code is still working as expected * Linting * #2861 Regenerated workbooks to only include dbkeys that are available in census table * #2861 Regenerated workbooks to only include dbkeys that are available in census tables * #2861 Improved logic in light of data discoveries * Added to commit by mistake * #2864 Moved shared logic to common area * #2864 Moved shared logic to common area * #2864 Updated logic to use census models and tables * #2864 Regenerated workbooks since this is crutial to ensure the code is still behaving as intended * #2861 Moved shared logic to a common area * #2867 Updated logic to use census models and tables * #2867 Regenerated workbooks to ensure the code is still behaving as intended * 2873 update logic in additional ein generator to use census models (#2875) * #2861 Update model mappings tfor federal awards worbook creation * #Refactor code to make it more re-usable * #2861 Code improvement * #2861 More util functions and tests * #2861 Generate workbooks to ensure the updated code is still working as expected * Linting * #2861 Regenerated workbooks to only include dbkeys that are available in census table * #2861 Regenerated workbooks to only include dbkeys that are available in census tables * #2861 Improved logic in light of data discoveries * Added to commit by mistake * #2864 Moved shared logic to common area * #2864 Moved shared logic to common area * #2864 Updated logic to use census models and tables * #2864 Regenerated workbooks since this is crutial to ensure the code is still behaving as intended * #2861 Moved shared logic to a common area * #2867 Updated logic to use census models and tables * #2867 Regenerated workbooks to ensure the code is still behaving as intended * #2873 Updated logic to use Census models * #2873 Generated workbooks to ensure the logic works as intended * Updated test workbooks * More bug fix in the test table generation logic * 2883 update logic in notes to sefa workbook generator to use census models (#2899) * #2883 Updated logic to use Census models * #2883 Updated workbooks * Update backend/census_historical_migration/workbooklib/notes_to_sefa.py Co-authored-by: Phil Dominguez <[email protected]> * Update backend/census_historical_migration/test_core_xforms.py Co-authored-by: Phil Dominguez <[email protected]> --------- Co-authored-by: Phil Dominguez <[email protected]> * Django admin improvements. (#2907) * 2895: New cmd for paginated census migration (#2905) * Creating paginated command * Updating readme * Lint * Using django paginator * Using page_size instead of batchSize * Readme tweak * Readme tweak * Update backend/census_historical_migration/README.md --------- Co-authored-by: Phil Dominguez <“[email protected]”> Co-authored-by: Hassan D. M. Sambo <[email protected]> * 2901 update logic in general information to use census models (#2906) * #2901 Updated logic to use Census models * Code improvement * Update backend/census_historical_migration/sac_general_lib/general_information.py Co-authored-by: Phil Dominguez <[email protected]> * Update backend/census_historical_migration/sac_general_lib/audit_information.py * Linting --------- Co-authored-by: Phil Dominguez <[email protected]> * Initial manage submission page (#2904) * WIP check-in for manage submission page. * Tests, some minor refactoring. * Django lint tweak. * Remove unnecessary util function. * Search results ALNs - my/all ALN columns * A big bundle of linting * Undo djlint autoformat. * Tests for ALN search. Basic & finding_my/all_aln * Add instructions for production ssh to deploying.md. (#2937) * Break ALN tests into their own class, and into more pieces. * Linting! Yippee! * Bools in ALN test cases - explicit for readability * Stray code block line deletion (#2939) * Add instructions for production ssh to deploying.md. * Remove stray code block start line. * Comment consistency * End date autofill fix * Search instructions link. --------- Co-authored-by: JeanMarie Mariadassou <[email protected]> Co-authored-by: Alex Steel <[email protected]> Co-authored-by: Tadhg O'Higgins <[email protected]> Co-authored-by: Hassan D. M. Sambo <[email protected]> Co-authored-by: Dan Swick <[email protected]> Co-authored-by: Phil Dominguez <[email protected]> Co-authored-by: Phil Dominguez <“[email protected]”> Co-authored-by: Sudha Kumar <[email protected]> Co-authored-by: Tim Ballard <[email protected]>
GSA-TTS · Dec 11, 2023 · f3b0b2e · f3b0b2e
1 parent b548f65
commit f3b0b2e
Show file tree

Hide file tree

Showing 4 changed files with 362 additions and 142 deletions.
diff --git a/backend/dissemination/search.py b/backend/dissemination/search.py
@@ -1,6 +1,6 @@
 from django.db.models import Q
 
-from dissemination.models import General, FederalAward
+from dissemination.models import General, FederalAward, Finding
 
 
 def search_general(
@@ -17,7 +17,11 @@ def search_general(
 ):
     query = Q()
 
-    query.add(_get_aln_match_query(alns), Q.AND)
+    # 'alns' gets processed before the match query function, as they get used again after the main search.
+    if alns:
+        split_alns, agency_numbers = _split_alns(alns)
+        query.add(_get_aln_match_query(split_alns, agency_numbers), Q.AND)
+
     query.add(_get_names_match_query(names), Q.AND)
     query.add(_get_uei_or_eins_match_query(uei_or_eins), Q.AND)
     query.add(_get_start_date_match_query(start_date), Q.AND)
@@ -31,27 +35,20 @@ def search_general(
 
     results = General.objects.filter(query).order_by("-fac_accepted_date")
 
+    if alns:
+        results = _attach_finding_my_aln_and_finding_all_aln_fields(
+            results, split_alns, agency_numbers
+        )
+
     return results
 
 
-def _get_aln_match_query(alns):
+def _split_alns(alns):
     """
-    Create the match query for ALNs.
-    Takes: A list of (potential) ALNs.
-    Returns: A query object matching on relevant report_ids found in the FederalAward table.
-
-    # ALNs are a little weird, because they are stored per-award in the FederalAward table. To search on ALNs, we:
-    # 1. Split the given ALNs into a set with their prefix and extention.
-    # 2. Search the FederalAward table for awards with matching federal_agency_prefix and federal_award_extension.
-    #    If there's just a prefix, search on all prefixes.
-    #    If there's a prefix and extention, search on both.
-    # 3. Add the report_ids from the identified awards to the search params.
+    Split an ALN query string into two sets.
+        1. split_alns: {(federal_agency_prefix, federal_award_extension), ...}
+        2. agency_numbers: {('federal_agency_prefix'), ...}
     """
-
-    if not alns:
-        return Q()
-
-    # Split each ALN into (prefix, extention)
     split_alns = set()
     agency_numbers = set()
     for aln in alns:
@@ -66,28 +63,16 @@ def _get_aln_match_query(alns):
                 # Otherwise, the individual elements go in unpaired.
                 split_alns.update([tuple(split_aln)])
 
-    # Search for relevant awards
-    report_ids = _get_aln_report_ids(split_alns)
+    return split_alns, agency_numbers
 
-    for agency_number in agency_numbers:
-        matching_awards = FederalAward.objects.filter(
-            federal_agency_prefix=agency_number
-        ).values()
-        if matching_awards:
-            for matching_award in matching_awards:
-                report_ids.update([matching_award.get("report_id")])
-    # Add the report_id's from the award search to the full search params
-    alns_match = Q()
-    for report_id in report_ids:
-        alns_match.add(Q(report_id=report_id), Q.OR)
-    return alns_match
 
-
-def _get_aln_report_ids(split_alns):
+def _get_aln_report_ids(split_alns, agency_numbers):
     """
-    Given a set of split ALNs, find the relevant awards and return their report_ids.
+    Given a set of ALNs and a set agency numbers, find the relevant awards and return a set of report_ids.
+    Utilizing sets helps to avoid duplicate reports.
     """
     report_ids = set()
+    # Matching on a specific ALN, such as '12.345'
     for aln_list in split_alns:
         matching_awards = FederalAward.objects.filter(
             federal_agency_prefix=aln_list[0], federal_award_extension=aln_list[1]
@@ -97,9 +82,66 @@ def _get_aln_report_ids(split_alns):
                 # Again, adding in a string requires [] so the individual
                 # characters of the report ID don't go in... we want the whole string.
                 report_ids.update([matching_award.get("report_id")])
+    # Matching on a whole agency, such as '12'
+    for agency_number in agency_numbers:
+        matching_awards = FederalAward.objects.filter(
+            federal_agency_prefix=agency_number
+        ).values()
+        if matching_awards:
+            for matching_award in matching_awards:
+                report_ids.update([matching_award.get("report_id")])
+
     return report_ids
 
 
+def _attach_finding_my_aln_and_finding_all_aln_fields(
+    results, split_alns, agency_numbers
+):
+    """
+    Given the results QuerySet (full of 'General' objects) and an ALN query string,
+    return the modified QuerySet, where each 'General' object has two new fields.
+
+    The process:
+    1. Get findings that fall under the given reports.
+    2. For each finding, get the relevant award. This is to access its ALN.
+    3. For each finding/award pair, they are either:
+       a. Under one of my ALNs, so we update finding_my_aln to True.
+       b. Under any other ALN, so we update finding_all_aln to True.
+    """
+    for result in results:
+        result.finding_my_aln = False
+        result.finding_all_aln = False
+        matching_findings = Finding.objects.filter(report_id=result.report_id)
+
+        for finding in matching_findings:
+            matching_award = FederalAward.objects.get(
+                report_id=result.report_id, award_reference=finding.award_reference
+            )
+            prefix = matching_award.federal_agency_prefix
+            extension = matching_award.federal_award_extension
+
+            if ((prefix, extension) in split_alns) or (prefix in agency_numbers):
+                result.finding_my_aln = True
+            else:
+                result.finding_all_aln = True
+
+    return results
+
+
+def _get_aln_match_query(split_alns, agency_numbers):
+    """
+    Given split ALNs and agency numbers, return the match query for ALNs.
+    """
+    # Search for relevant awards
+    report_ids = _get_aln_report_ids(split_alns, agency_numbers)
+
+    # Add the report_id's from the award search to the full search params
+    alns_match = Q()
+    for report_id in report_ids:
+        alns_match.add(Q(report_id=report_id), Q.OR)
+    return alns_match
+
+
 def _get_names_match_query(names):
     """
     Given a list of (potential) names, return the query object that searches auditee and firm names.