Add pe_source module from pe-reports #1

aloftus23 · 2023-07-06T21:07:58Z

🗣 Description

Move the existing pe-source module in pe-reports to this repo.

💭 Motivation and context

Splitting the repo will allow other applications to install just the pe-source module without any of the report/email functionality in pe-reports.

🧪 Testing

Updates pytests and passes pre-commits.

✅ Pre-approval checklist

This PR has an informative and human-readable title.
Changes are limited to a single goal - eschew scope creep!
All future TODOs are captured in issues, which are referenced
in code comments.
All relevant type-of-change labels have been added.
I have read the CONTRIBUTING document.
These code changes follow cisagov code standards.
All relevant repo and/or project documentation has been updated
to reflect the changes in this PR.
Tests have been added and/or modified to cover the changes in this PR.
All new and existing tests pass.

… versions For example, pandas 1.5.1

aloftus23 · 2023-07-18T21:07:29Z

Ready for review

aloftus23 · 2023-07-24T12:31:30Z

@dav3r This PR is ready. Let me know if you have any questions

dav3r

I'll try to review the rest of this later this week. I noted a few items that caught my eye.

setup.py

Co-authored-by: dav3r <[email protected]>

aloftus23 · 2023-07-26T20:02:27Z

I'll try to review the rest of this later this week. I noted a few items that caught my eye.

Thanks. Addressed your initial review. Will look out for the rest.

dav3r

I made it a little bit further through my review, but there's a lot of code in this PR. I'll have to continue at a later date.

.DS_Store

.flake8

.gitignore

README.md

setup.py

src/pe_source/__init__.py

src/pe_source/data/database.ini

tests/.DS_Store

src/pe_source/pe_scripts.py

jsf9k

I have some change requests and suggestions for improvement.

.github/workflows/build.yml

jsf9k · 2023-08-01T18:15:02Z

.gitignore

+pe_reports_logging.log
+src/pe_source/data/dnstwist_output.txt
+
+


Remove unnecessary blank line:

Suggested change

Addressed here f58ca92

jsf9k · 2023-08-01T18:15:58Z

.pre-commit-config.yaml

          - types-setuptools
+          - types-python-dateutil==2.8.19


Why not put this dependency in alphabetical order like the rest?

Fixed here: 5d1b723

jsf9k · 2023-08-01T18:29:33Z

src/pe_source/pe_scripts.py

+"""A tool for gathering pe source data.
+
+Usage:
+    pe-source DATA_SOURCE [--log-level=LEVEL] [--orgs=ORG_LIST] [--cybersix-methods=METHODS] [--soc_med_included]


Underscores are not used for any of the other options:

Suggested change

pe-source DATA_SOURCE [--log-level=LEVEL] [--orgs=ORG_LIST] [--cybersix-methods=METHODS] [--soc_med_included]

pe-source DATA_SOURCE [--log-level=LEVEL] [--orgs=ORG_LIST] [--cybersix-methods=METHODS] [--soc-med-included]

Fixed here: d8e46a6 and 3a2e262

jsf9k · 2023-08-01T18:29:58Z

src/pe_source/pe_scripts.py

+                                    If not specified, all will run. Valid values are "alerts",
+                                    "credentials", "mentions", "topCVEs". E.g. alerts,mentions.
+                                    [default: all]
+  -sc --soc_med_included            Include social media posts from cybersixgill in data collection.


Underscores are not used for any of the other options:

Suggested change

-sc --soc_med_included Include social media posts from cybersixgill in data collection.

-sc --soc-med-included Include social media posts from cybersixgill in data collection.

Fixed in d8e46a6

jsf9k · 2023-08-01T18:55:42Z

src/pe_source/data/dnsmonitor/source.py

+    for org_index, org_row in org_names_df.iterrows():
+        for domain_index, domain_row in domain_df.iterrows():
+            if org_row["domain_name"] == domain_row["domainName"]:
+                domain_df.at[domain_index, "org"] = org_row["org"]


You could possibly rewrite this to avoid the double for loop and speed it up. Just something to think about.

Good call. Made adjustments here: c362c8e

jsf9k · 2023-08-01T18:58:58Z

src/pe_source/data/helpers/redact_pii.py

+import scrubadub
+import scrubadub.detectors.date_of_birth
+
+# List of unique regexes to identify each state's Drivers License format in a larger string


Are all 50 states not listed because the states not listed use a person's SSN as the identifier?

jsf9k · 2023-08-01T19:04:59Z

src/pe_source/data/shodan/shodan_search.py

+    for i, ip_chunk in enumerate(ip_chunks):
+        count = i + 1
+        try_count = 1
+        while try_count < 7:


Magic numbers like 7 should be avoided; use variables so you can easily change the value later if necessary.

Fixed here: 10f0e6d

src/pe_source/cybersixgill.py

dav3r

Got through a few more files - will continue to chip away at this review when I am able to.

src/pe_source/cybersixgill.py

src/pe_source/data/dnsmonitor/root_domains_dnsmonitor.csv

src/pe_source/data/dnsmonitor/source.py

Was pulled from another repo. Not needed here

coveralls · 2023-09-24T18:11:34Z

Pull Request Test Coverage Report for Build 5592380198

Details

0 of 0 changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-100.0%) to 0.0%

Totals
Change from base Build 5977993120:	-100.0%
Covered Lines:	0
Relevant Lines:	0

💛 - Coveralls

Remove .DS_Store

Co-authored-by: dav3r <[email protected]> Co-authored-by: Shane Frasier <[email protected]>

Co-authored-by: Shane Frasier <[email protected]> Co-authored-by: dav3r <[email protected]>

Co-authored-by: Shane Frasier <[email protected]>

…urce into AL-copy-pe-reports

Change --soc_med_included to --soc-med-included change -csg to -c and -sc to -s to consistently keep the flags one character

Co-authored-by: dav3r <[email protected]>

…urce into AL-copy-pe-reports

Co-authored-by: dav3r <[email protected]>

…urce into AL-copy-pe-reports

Add comment explaining for cybersixgill.py and add variable for retry count

Copy pe_source module for pe_reports repo

2bad91e

aloftus23 self-assigned this Jul 6, 2023

aloftus23 and others added 12 commits July 17, 2023 15:04

Create new setup.py and update import directories

9c23a77

Run linter

7289af6

Fix pre-commit checks

ad3d391

Update README.md with examples

ab4e5a7

fix readme linter

f489fa6

Fix pytests

83ea436

fix lint

75f5e54

Fix dnstwist script

77951cb

fix lint

c42e28d

Don't tesst on python3.7 packages since it will fail with new package…

3dbd17c

… versions For example, pandas 1.5.1

Also remove python 3.6

99e738b

fix build.yml file

0bcb398

aloftus23 marked this pull request as ready for review July 18, 2023 21:06

aloftus23 requested review from dav3r, felddy, jsf9k and mcdonnnj as code owners July 18, 2023 21:06

dav3r requested changes Jul 25, 2023

View reviewed changes

setup.py Outdated Show resolved Hide resolved

setup.py Outdated Show resolved Hide resolved

setup.py Outdated Show resolved Hide resolved

setup.py Outdated Show resolved Hide resolved

aloftus23 and others added 2 commits July 26, 2023 15:57

Remove python 3.6/3.7 and add 3.10/3.11 to setup.py

0a5e6a4

Correct comment and alphabetize package_data

3e7eab1

Co-authored-by: dav3r <[email protected]>

dav3r requested changes Jul 31, 2023

View reviewed changes

dav3r reviewed Aug 1, 2023

View reviewed changes

src/pe_source/pe_scripts.py Outdated Show resolved Hide resolved

jsf9k requested changes Aug 1, 2023

View reviewed changes

dav3r reviewed Aug 1, 2023

View reviewed changes

src/pe_source/cybersixgill.py Outdated Show resolved Hide resolved

src/pe_source/cybersixgill.py Outdated Show resolved Hide resolved

dav3r reviewed Aug 2, 2023

View reviewed changes

src/pe_source/cybersixgill.py Show resolved Hide resolved

src/pe_source/cybersixgill.py Outdated Show resolved Hide resolved

src/pe_source/data/dnsmonitor/root_domains_dnsmonitor.csv Outdated Show resolved Hide resolved

src/pe_source/data/dnsmonitor/source.py Outdated Show resolved Hide resolved

aloftus23 mentioned this pull request Aug 10, 2023

Update DNSTwist to use the pe-reports module and run on all P&E orgs cisagov/crossfeed#1988

Open

12 tasks

Remove E712 ignore

ccbeaac

Was pulled from another repo. Not needed here

aloftus23 requested a review from jasonodoom as a code owner August 18, 2023 15:25

Change pe_scripts to pe_source

b23a404

aloftus23 and others added 23 commits September 24, 2023 14:12

Delete .DS_Store

e25ecd0

Remove .DS_Store

Fix .gitignore

f58ca92

Fix formatting in README

3ed5c3d

Co-authored-by: dav3r <[email protected]> Co-authored-by: Shane Frasier <[email protected]>

Fix setup.py mistakes

036eac1

Co-authored-by: dav3r <[email protected]> Co-authored-by: Shane Frasier <[email protected]>

Fix commenting in __init__

412515b

Co-authored-by: Shane Frasier <[email protected]> Co-authored-by: dav3r <[email protected]>

Remove ubuntu-10.04 from build.yml

6c3d078

Co-authored-by: Shane Frasier <[email protected]>

Alphabetize .pre-commit configs

5d1b723

Merge branch 'AL-copy-pe-reports' of https://github.com/cisagov/pe-so…

3bbb6a2

…urce into AL-copy-pe-reports

Fix options in docopt and readme

d8e46a6

Change --soc_med_included to --soc-med-included change -csg to -c and -sc to -s to consistently keep the flags one character

Alphabetize database.ini

d67d2d7

Delete tests/.DS_Store

e63fb27

Re-add CPython to setup.oy

ac7c6ea

Fix comment in src/pe_source/cybersixgill.py

00275f0

Co-authored-by: dav3r <[email protected]>

Simplify pe_org log in src/pe_source/cybersixgill.py

95615a1

Co-authored-by: dav3r <[email protected]>

Run linter

3d24f0b

Merge branch 'AL-copy-pe-reports' of https://github.com/cisagov/pe-so…

f66a62e

…urce into AL-copy-pe-reports

Use ipaddress python library to determine v6 vs v4

37e73ba

Sort root_domains_dnsmonitor by domain

53e8e28

Sort cybersixgill.py sites

f618d12

Co-authored-by: dav3r <[email protected]>

Fix docopt for soc-med-included

3a2e262

Merge branch 'AL-copy-pe-reports' of https://github.com/cisagov/pe-so…

02fa638

…urce into AL-copy-pe-reports

Address magic numbers

10f0e6d

Add comment explaining for cybersixgill.py and add variable for retry count

Replace double for-loop with merge in dnsmonitor/source

c362c8e

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add pe_source module from pe-reports #1

Add pe_source module from pe-reports #1

aloftus23 commented Jul 6, 2023 •

edited

Loading

aloftus23 commented Jul 18, 2023

aloftus23 commented Jul 24, 2023

dav3r left a comment

aloftus23 commented Jul 26, 2023

dav3r left a comment

jsf9k left a comment

jsf9k Aug 1, 2023

aloftus23 Sep 24, 2023

jsf9k Aug 1, 2023

aloftus23 Sep 24, 2023

jsf9k Aug 1, 2023

aloftus23 Sep 24, 2023

jsf9k Aug 1, 2023

aloftus23 Sep 24, 2023

jsf9k Aug 1, 2023

aloftus23 Sep 24, 2023

jsf9k Aug 1, 2023

jsf9k Aug 1, 2023

aloftus23 Sep 24, 2023

dav3r left a comment

coveralls commented Sep 24, 2023 •

edited

Loading

		pe_reports_logging.log
		src/pe_source/data/dnstwist_output.txt

	pe-source DATA_SOURCE [--log-level=LEVEL] [--orgs=ORG_LIST] [--cybersix-methods=METHODS] [--soc_med_included]
	pe-source DATA_SOURCE [--log-level=LEVEL] [--orgs=ORG_LIST] [--cybersix-methods=METHODS] [--soc-med-included]

	-sc --soc_med_included Include social media posts from cybersixgill in data collection.
	-sc --soc-med-included Include social media posts from cybersixgill in data collection.

Add pe_source module from pe-reports #1

Are you sure you want to change the base?

Add pe_source module from pe-reports #1

Conversation

aloftus23 commented Jul 6, 2023 • edited Loading

🗣 Description

💭 Motivation and context

🧪 Testing

✅ Pre-approval checklist

aloftus23 commented Jul 18, 2023

aloftus23 commented Jul 24, 2023

dav3r left a comment

Choose a reason for hiding this comment

aloftus23 commented Jul 26, 2023

dav3r left a comment

Choose a reason for hiding this comment

jsf9k left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dav3r left a comment

Choose a reason for hiding this comment

coveralls commented Sep 24, 2023 • edited Loading

Pull Request Test Coverage Report for Build 5592380198

Details

💛 - Coveralls

aloftus23 commented Jul 6, 2023 •

edited

Loading

coveralls commented Sep 24, 2023 •

edited

Loading