This repository has been archived by the owner on Jun 20, 2023. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 388
Unofficial antivirus definitions using Fangfrisch #212
Open
gchamon
wants to merge
50
commits into
bluesentry:master
Choose a base branch
from
waycarbon:feature/fangfrisch-extra-defs
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
50 commits
Select commit
Hold shift + click to select a range
8e394f9
Update Dockerfile
gchamon 9c16448
fix dockerfile missing so
gchamon 94609bf
Merge pull request #1 from waycarbon/fix/dockerfile-missing-so
gchamon d45cbb7
implement fangfrisch usage
gchamon 3a898f2
update dockerfile and update to reference correct fangfrisch bin
gchamon fce2f29
installing awscliv2 and add correct pythonpath for fangfrisch
gchamon abbeee1
revert change to /opt/app
gchamon efc71b6
add aws-cli to final lambda
gchamon e21edc6
add lib64 python packages
gchamon ea1bc0c
centralizing pip installation with --target argument
gchamon ed91b06
removing cp statement made unecessary with --target argument for pip
gchamon 5df78a7
downgrade awscli to v1
gchamon ca235e4
return with copying python3.7 site-packages for extra missing deps
gchamon 3e9b2e3
update urllib3
gchamon cf30411
remove python3.7 site packages, as it doesn't really exist
gchamon 43baf43
run aws from bin folder
gchamon f59ec40
add pythonpath to awscli invocation
gchamon cb5f840
separate fangfrisch requirements.txt; optimized docker build image
gchamon 02fef58
sed in-place the shebang for correct python interpreter for lambdas
gchamon 52d4137
fangfrisch.conf absolute path
gchamon af499af
add missing fangfrisch.conf file
gchamon 0180c60
check shell return; sync without progress
gchamon b253208
use update_defs_from_s3 to download extra definitions
gchamon 1ec5169
ditch awscli for custom down/upload functions
gchamon af87467
use sed to reflect AV_DEFINITION_PATH in fangfrisch.conf
gchamon d1c3ac8
revert changes to md5 from s3 tags function
gchamon 89d6c80
changing fangfrisch.conf in /tmp folder
gchamon 177d3dc
add hardened database with false-positives avoiding configs
gchamon 1ac4456
minor fix for logging download definition
gchamon ffa3c6b
merging clamdscan for scan.conf support and lower scan timers
gchamon 6c1d36d
fix sqlite db persistence for fangfrisch
gchamon 97037a0
optimize build times
gchamon 64fbd46
optimize dockerfile
gchamon 7962ef5
fix libpcre.so not being added to zip
gchamon 310d112
compress missing files into a single log line
gchamon 6fc216a
fix orphan config in freshclam.conf
gchamon 22d0b4e
fix config in freshclam.conf; separate freshclam and scan conf runs
gchamon bd196a9
logging freshclam output as a list of strings
gchamon e6102f9
compressing not downloading and md5 matches into single line
gchamon d4ebd2d
fix freshclam output split
gchamon 0e95de6
download freshclam defs before running fangfrisch
gchamon bb8fe9f
breaking updater if freshclam panics
gchamon d0209e0
downloading ALL extra files definitions, even if same prefix
gchamon 02cc4fa
only print older files and md5 matches if they exist
gchamon 2ba7b19
fix infection deletion log
gchamon 5ac7593
using aws lambda stage image for fangfrisch to avoid shebang rewrite
gchamon 2b76b8c
uploading all extra definition files
gchamon 111af04
refactored upload defs to merge all files into single list
gchamon c6eb891
fix typo
gchamon 750f7f0
only add extra definition files to databases to upload if necessary
gchamon File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,50 +1,119 @@ | ||
| FROM public.ecr.aws/lambda/python:3.7 AS cli_deps | ||
|
|
||
| COPY requirements-cli.txt requirements-cli.txt | ||
| RUN mkdir -p /opt/app/cli \ | ||
| && pip3 install --requirement requirements-cli.txt --target /opt/app/cli \ | ||
| && rm -rf /root/.cache/pip | ||
|
|
||
| FROM amazonlinux:2 | ||
|
|
||
| # Set up working directories | ||
| RUN mkdir -p /opt/app | ||
| RUN mkdir -p /opt/app/build | ||
| RUN mkdir -p /opt/app/bin/ | ||
|
|
||
| # Copy in the lambda source | ||
| WORKDIR /opt/app | ||
| COPY ./*.py /opt/app/ | ||
| COPY requirements.txt /opt/app/requirements.txt | ||
| RUN mkdir -p \ | ||
| /opt/app \ | ||
| /opt/app/build \ | ||
| /opt/app/bin \ | ||
| /opt/app/python_deps \ | ||
| /opt/app/cli | ||
|
|
||
| # Install packages | ||
| RUN yum update -y | ||
| RUN yum install -y cpio python3-pip yum-utils zip unzip less | ||
| RUN yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm | ||
|
|
||
| # This had --no-cache-dir, tracing through multiple tickets led to a problem in wheel | ||
| RUN pip3 install -r requirements.txt | ||
| RUN rm -rf /root/.cache/pip | ||
| RUN yum update -y \ | ||
| && amazon-linux-extras install epel -y \ | ||
| && yum install -y \ | ||
| cpio \ | ||
| yum-utils \ | ||
| tar.x86_64 \ | ||
| gzip \ | ||
| zip \ | ||
| python3-pip \ | ||
| shadow-utils.x86_64 \ | ||
| && yum clean all \ | ||
| && rm -rf /var/cache/yum | ||
|
|
||
| # Download libraries we need to run in lambda | ||
| WORKDIR /tmp | ||
| RUN yumdownloader -x \*i686 --archlist=x86_64 clamav clamav-lib clamav-update json-c pcre2 libprelude gnutls libtasn1 lib64nettle nettle | ||
| RUN rpm2cpio clamav-0*.rpm | cpio -idmv | ||
| RUN rpm2cpio clamav-lib*.rpm | cpio -idmv | ||
| RUN rpm2cpio clamav-update*.rpm | cpio -idmv | ||
| RUN rpm2cpio json-c*.rpm | cpio -idmv | ||
| RUN rpm2cpio pcre*.rpm | cpio -idmv | ||
| RUN rpm2cpio gnutls* | cpio -idmv | ||
| RUN rpm2cpio nettle* | cpio -idmv | ||
| RUN rpm2cpio lib* | cpio -idmv | ||
| RUN rpm2cpio *.rpm | cpio -idmv | ||
| RUN rpm2cpio libtasn1* | cpio -idmv | ||
|
|
||
| # Copy over the binaries and libraries | ||
| RUN cp /tmp/usr/bin/clamscan /tmp/usr/bin/freshclam /tmp/usr/lib64/* /opt/app/bin/ | ||
| RUN yumdownloader -x \*i686 --archlist=x86_64 \ | ||
| clamav \ | ||
| clamav-lib \ | ||
| clamav-update \ | ||
| clamav-scanner-systemd \ | ||
| elfutils-libs \ | ||
| json-c \ | ||
| lz4 \ | ||
| pcre2 \ | ||
| systemd-libs \ | ||
| libtool-ltdl \ | ||
| libxml2 \ | ||
| bzip2-libs \ | ||
| xz-libs \ | ||
| libprelude \ | ||
| gnutls \ | ||
| nettle \ | ||
| && rpm2cpio clamav-0*.rpm | cpio -vimd \ | ||
| && rpm2cpio clamav-lib*.rpm | cpio -vimd \ | ||
| && rpm2cpio clamav-update*.rpm | cpio -vimd \ | ||
| && rpm2cpio json-c*.rpm | cpio -vimd \ | ||
| && rpm2cpio pcre*.rpm | cpio -vimd \ | ||
| && rpm2cpio libtool-ltdl*.rpm | cpio -vimd \ | ||
| && rpm2cpio libxml2*.rpm | cpio -vimd \ | ||
| && rpm2cpio bzip2-libs*.rpm | cpio -vimd \ | ||
| && rpm2cpio xz-libs*.rpm | cpio -vimd \ | ||
| && rpm2cpio libprelude*.rpm | cpio -vimd \ | ||
| && rpm2cpio gnutls*.rpm | cpio -vimd \ | ||
| && rpm2cpio nettle*.rpm | cpio -vimd \ | ||
| && rpm2cpio clamd-0*.rpm | cpio -idmv \ | ||
| && rpm2cpio elfutils-libs*.rpm | cpio -idmv \ | ||
| && rpm2cpio lz4*.rpm | cpio -idmv \ | ||
| && rpm2cpio systemd-libs*.rpm | cpio -idmv \ | ||
| && cp -r \ | ||
| /tmp/usr/bin/clamdscan \ | ||
| /tmp/usr/sbin/clamd \ | ||
| /tmp/usr/bin/freshclam \ | ||
| /tmp/usr/lib64/* \ | ||
| /usr/lib64/libpcre.so* \ | ||
| /opt/app/bin/ \ | ||
| && rm -rf /tmp/usr | ||
|
|
||
| # Fix the freshclam.conf settings | ||
| RUN echo "DatabaseMirror database.clamav.net" > /opt/app/bin/freshclam.conf | ||
| RUN echo "CompressLocalDatabase yes" >> /opt/app/bin/freshclam.conf | ||
| RUN echo "DatabaseMirror database.clamav.net" > /opt/app/bin/freshclam.conf \ | ||
| && echo "CompressLocalDatabase yes" >> /opt/app/bin/freshclam.conf \ | ||
| && echo "ScriptedUpdates no" >> /opt/app/bin/freshclam.conf \ | ||
| && echo "DatabaseDirectory /var/lib/clamav" >> /opt/app/bin/freshclam.conf | ||
| # clamd conf with hardened configs to avoid false positives | ||
| RUN echo "DatabaseDirectory /tmp/clamav_defs" > /opt/app/bin/scan.conf \ | ||
| && echo "PidFile /tmp/clamd.pid" >> /opt/app/bin/scan.conf \ | ||
| && echo "LogFile /tmp/clamd.log" >> /opt/app/bin/scan.conf \ | ||
| && echo "LocalSocket /tmp/clamd.sock" >> /opt/app/bin/scan.conf \ | ||
| && echo "FixStaleSocket yes" >> /opt/app/bin/scan.conf \ | ||
| && echo "DetectPUA yes" >> /opt/app/bin/scan.conf \ | ||
| && echo "ExcludePUA PUA.Win.Packer" >> /opt/app/bin/scan.conf \ | ||
| && echo "ExcludePUA PUA.Win.Trojan.Packed" >> /opt/app/bin/scan.conf \ | ||
| && echo "ExcludePUA PUA.Win.Trojan.Molebox" >> /opt/app/bin/scan.conf \ | ||
| && echo "ExcludePUA PUA.Win.Packer.Upx" >> /opt/app/bin/scan.conf \ | ||
| && echo "ExcludePUA PUA.Doc.Packed" >> /opt/app/bin/scan.conf | ||
|
|
||
| # Create the zip file | ||
| RUN groupadd clamav \ | ||
| && useradd -g clamav -s /bin/false -c "Clam Antivirus" clamav \ | ||
| && useradd -g clamav -s /bin/false -c "Clam Antivirus" clamupdate | ||
|
|
||
| ENV LD_LIBRARY_PATH=/opt/app/bin | ||
| RUN ldconfig | ||
|
|
||
| # Copy in the lambda source | ||
| WORKDIR /opt/app | ||
| RUN zip -r9 --exclude="*test*" /opt/app/build/lambda.zip *.py bin | ||
| COPY requirements.txt /opt/app/requirements.txt | ||
|
|
||
| WORKDIR /usr/local/lib/python3.7/site-packages | ||
| RUN zip -r9 /opt/app/build/lambda.zip * | ||
| # This had --no-cache-dir, tracing through multiple tickets led to a problem in wheel | ||
| RUN pip3 install --requirement requirements.txt --target /opt/app/python_deps \ | ||
| && rm -rf /root/.cache/pip | ||
|
|
||
| # Copy fangfrisch CLI from lambda image | ||
| COPY --from=cli_deps /opt/app/cli /opt/app/cli | ||
|
|
||
| # Create the zip file | ||
| COPY ./*.py /opt/app/ | ||
| COPY fangfrisch.conf /opt/app/fangfrisch.conf | ||
| RUN zip -r9 --exclude="*test*" /opt/app/build/lambda.zip *.py *.conf bin cli \ | ||
| && cd /opt/app/python_deps \ | ||
| && zip -r9 /opt/app/build/lambda.zip * | ||
|
|
||
| WORKDIR /opt/app | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
must copy all libpcre.so variants using wildcard, because libpcre.so.1 is a symbolic link sometimes