Replies: 17 comments
-
We actually do that... After checking for the syntax 😉
Can you extend/explain what you mean with that ?
PyFunceble already do that for us but yeah it's a way to gain some time 😉
Nope, autosaving with git and a few other things won't necessarily work! That's why on my list there is the test and implementation of other CI/CD infrastructure like the one from GitLab 😄 It's not for nothing that I put the milestone |
Beta Was this translation helpful? Give feedback.
-
Sure, If you take a look from: https://travis-ci.com/Import-External-Sources/google.tld/jobs/257187392#L414 and https://travis-ci.com/Import-External-Sources/pornhosts/jobs/257274300#L504 From my needs:
What is done.
For the above I haven't found a switch to NOT doing test 2. or 3. and only do the Other purpose of controlling test protocols....rCodeIf you are running a DNS server/Resolver that is meant to block with ex.:
Now you can explicit test if that is running as you think it should. where Now all failures will comes out as the above use case of RCODE in a http - ftpIf a user explicitly want to test against the http or ftp protocol you simply add this switch to the test string: Now PyFunceble is defaut only doing Reversed testSo how could PyFunceble be used in reversed test cases
On a local system I would of curse do the hosts/dns test in other ways ex. with diff on linux, but to do this by a CI, well it could be the lazy choice 😃
Hate to break it then, but it have been running on a assessor to https://gitlab.com/rpz-zones/google.tld 😃 and ok haven't tested it on Bitbucket at all. It will of curse make some nice sense if it only runs with confidence on travis end noteStopping the answer here, as I spent 1½ hours re-writing and yet again re-writing to make it better explained... So please drop a comment if an answer is unclear |
Beta Was this translation helpful? Give feedback.
-
|
From the test string from #61 add the guess what, there are no such records as Wouldn't it speed up thing if ALL tested domains was recorded to re-fetch eg. last tested as ex. Side note grep -i ".bid$" output/domains/INACTIVE/list | wc -l DB entries in Let's have another ex. where the test happens unexpected to me. INSERT INTO PyFunceble.pyfunceble_whois (subject,expiration_date,expiration_date_epoch,state,record,digest,created,modified) VALUES
('hdwkcfqzxhvx.com','05-may-2020',1588629600,'future',' Domain Name: HDWKCFQZXHVX.COM
Registry Domain ID: 2387572425_DOMAIN_COM-VRSN
Registrar WHOIS Server: whois.namecheap.com
Registrar URL: http://www.namecheap.com
Updated Date: 2019-05-05T01:15:23Z
Creation Date: 2019-05-05T01:15:18Z
Registry Expiry Date: 2020-05-05T01:15:18Z
Registrar: NameCheap, Inc.
Registrar IANA ID: 1068
Registrar Abuse Contact Email: [email protected]
Registrar Abuse Contact Phone: +1.6613102107
Domain Status: clientTransferProhibited https://icann.org/epp#clientTransferProhibited
Name Server: NS0.DNSMADEEASY.COM
Name Server: NS1.DNSMADEEASY.COM
Name Server: NS2.DNSMADEEASY.COM
Name Server: NS3.DNSMADEEASY.COM
Name Server: NS4.DNSMADEEASY.COM
DNSSEC: unsigned
URL of the ICANN Whois Inaccuracy Complaint Form: https://www.icann.org/wicf/
>>> Last update of whois database: 2019-11-17T23:02:29Z <<<
For more information on Whois status codes, please visit https://icann.org/epp
NOTICE: The expiration date displayed in this record is the date the
registrar''s sponsorship of the domain name registration in the registry is
currently set to expire. This date does not necessarily reflect the expiration
date of the domain name registrant''s agreement with the sponsoring
registrar. Users may consult the sponsoring registrar''s Whois database to
view the registrar''s reported date of expiration for this registration.
TERMS OF USE: You are not authorized to access or query our Whois
database through the use of electronic processes that are high-volume and
automated except as reasonably necessary to register domain names or
modify existing registrations; the Data in VeriSign Global Registry
Services'' ("VeriSign") Whois database is provided by VeriSign for
information purposes only, and to assist persons in obtaining information
about or related to a domain name registration record. VeriSign does not
guarantee its accuracy. By submitting a Whois query, you agree to abide
by the following terms of use: You agree that you may use this Data only
for lawful purposes and that under no circumstances will you use this Data
to: (1) allow, enable, or otherwise support the transmission of mass
unsolicited, commercial advertising or solicitations via e-mail, telephone,
or facsimile; or (2) enable high volume, automated, electronic processes
that apply to VeriSign (or its computer systems). The compilation,
repackaging, dissemination or other use of this Data is expressly
prohibited without the prior written consent of VeriSign. You agree not to
use electronic processes that are automated and high-volume to access or
query the Whois database except as reasonably necessary to register
domain names or modify existing registrations. VeriSign reserves the right
to restrict your access to the Whois database in its sole discretion to ensure
operational stability. VeriSign may restrict or terminate your access to the
Whois database for failure to abide by these terms of use. VeriSign
reserves the right to modify these terms at any time.
The Registry database contains ONLY .COM, .NET, .EDU domains and
Registrars.
','e96ec2e308f99637004d1f37eaaf0b5f9993f8c1d5617d2961bd82f13cfe7198','2019-11-18 00:02:37.0','2019-11-18 00:02:37.0')
;yet it have been tested on all 4 runs.... with-in a couple of hours |
Beta Was this translation helpful? Give feedback.
-
With
Patch on its way.
It's true, no data is stored. The patch is on its way. Stupid restructuration/improvement ideas I got weeks ago... |
Beta Was this translation helpful? Give feedback.
-
|
This tool is becoming bigger and bigger... I even get the nightmare of it 😂 I need to take a day off anything else and really refocus on the whole infrastructure before releasing it... The restructuration was successful but some of my optimization and improvement ideas went great or wrong in this case ... |
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
|
Nope, no
🆒 🤶 Is near by to comfort you 😄 Year, it getting bigger, but also better... we 🤞 for this. The god part of this is, you have already developed the most of it. PyFunceble have come from the crawling age to it walking stage... just wait to it becomes a teenager 🤕 |
Beta Was this translation helpful? Give feedback.
-
|
Hi, just done a simple math on this run.... https://travis-ci.com/github/Import-External-Sources/pornhosts/builds. First run is Build 1921. There is 9776 records in the file that needs to be tested, and at the current moment (build 1956) it have an average of testing 6,9 domains a minutes.... I thinks the reason for this is that every time a new Travis is started you Pyfunceble runs through the entire list from the beginning and not actually continues from where it left of. An initial thought on this could be to set(flip) a value in the DB of which was just tested, which was added and where do we continue. Just as examples, 0 = tested, 1 = Not tested, 2 = New, Not tested (added between runs). update to add a bit more logic to the code flow 😄 |
Beta Was this translation helpful? Give feedback.
-
|
Good point @spirillen, the problem: I want a solution for both: MariaDB/MySQL and the JSON one. I will think about this when I get a bit of a time 👍 |
Beta Was this translation helpful? Give feedback.
-
|
@AnonymousPoster, I removed it. The release is on its way out right now. (So will be available in the next 30-45 minutes if everything goes right) @spirillen, if you still need it, feel free to use the |
Beta Was this translation helpful? Give feedback.
-
|
We need to improve the way we scan/test a file 🤔 Any ideas @spirillen @AnonymousPoster ? I'm out of direct idea right now 🤔 https://travis-ci.com/github/dns-test/porn.host.rpz is an example of the whole process which may take days 🤔 |
Beta Was this translation helpful? Give feedback.
-
|
To continue our keybase talk in the open. Then here is a suggestion on how to change the DB architecture to use more caching and less UPDATE (delete - write) in the databases as it is destroying the cache on every update. The current design looks something like this:
What I think is we should distribute this over several databases as it would prevent a lot I/O for rebuilding cache So first we spread own information The whois information (somewhat more stable) In this we store all know WHOIS information, never delete(cleanup or so (of curse after 5 years or something)) In the next DB we store all domains ever tested as the PRIMARY key for everything The url is here a subkey to the domain_id. In case the domain is expired, we can drop testing the url This is the DB that requires a lot of resources This is where we build the And then a new one to meet the powerfull RPZ standard This is where we can find the source of a domain "distribution" and find similarities in the suckers DNS, as ex. These dumps is not fully completed, but thought as the base for the BIG idea |
Beta Was this translation helpful? Give feedback.
-
|
To follow up with a few comments The you currently store The same for the For the
For more information on Whois status codes, please visit https://icann.org/epp NOTICE: The expiration date displayed in this record is the date the TERMS OF USE: You are not authorized to access or query our Whois The Registry database contains ONLY .COM, .NET, .EDU domains and
For more information on Whois status codes, please visit https://icann.org/epp NOTICE: The expiration date displayed in this record is the date the TERMS OF USE: You are not authorized to access or query our Whois The Registry database contains ONLY .COM, .NET, .EDU domains and What it does is eating up resources. Extract the basic needs and have the script output the |
Beta Was this translation helpful? Give feedback.
-
|
Here is a doublet in The only way out of this is to convert the test key to the domain, this will leads to retesting the same Here is a couple of other examples, with same subject, different sources
And in sql it looks more like this select file_path, subject, count(*) as occurrences
from PyFunceble.pyfunceble_auto_continue
group by subject
having count(*) > 1;Top 10 results
You have the key so you can test it your self 😃 |
Beta Was this translation helpful? Give feedback.
-
|
For the @control-whois-storage Why is it you would like to keep the entire Examplesinfowhois azvjudwr.info
Domain Name: AZVJUDWR.INFO
Registry Domain ID: D503300000575448281-LRMS
Registrar WHOIS Server: whois.dynadot.com
Registrar URL: http://www.dynadot.com
Updated Date: 2020-02-05T13:57:43Z
Creation Date: 2019-02-19T11:16:34Z
Registry Expiry Date: 2021-02-19T11:16:34Z
Registrar Registration Expiration Date:
Registrar: Dynadot, LLC
Registrar IANA ID: 472
Registrar Abuse Contact Email: [email protected]
Registrar Abuse Contact Phone: +1.6502620100
Reseller:
Domain Status: clientTransferProhibited https://icann.org/epp#clientTransferProhibited
Registrant Organization:
Registrant State/Province: California
Registrant Country: US
Name Server: NS1.CSOF.NET
Name Server: NS2.CSOF.NET
Name Server: NS4.CSOF.NET
Name Server: NS3.CSOF.NET
Name Server: NS5.CSOF.NET
Name Server: NS6.CSOF.NET
Name Server: NS7.CSOF.NET
Name Server: NS8.CSOF.NET
DNSSEC: unsigned
URL of the ICANN Whois Inaccuracy Complaint Form is https://www.icann.org/wicf/
>>> Last update of WHOIS database: 2020-08-28T13:35:12Z <<<
For more information on Whois status codes, please visit https://icann.org/eppHere you should strip the This is waste of space. xyzAgain waste of space to store comAgain wasting space To optimize this you should extract the data, and add them to individual columns like:
The same goes on for the follow fields as they are most likely to be doubles in our situations and it would enhance the way of searching/extracting data ¹) |
Beta Was this translation helpful? Give feedback.
-
The only thing I can come up with, is that the I also posted a few ideas here https://www.mypdns.org/T1250#23313 for how to optimize the database structure to accomplice this, but again, that is only the SQL edition, not the JSON. And to add even more evil, then we have
Which both will be the pain for testing. Here the #115 comes to play, if we starting by reading in everything from any given source, then we can put up several slaves to test the next batch of records, but again, requires central SQL. Conclusion The way out of this is centralizing in any way(for the sake of speed) and the flip the URI/Domains into the keys for testing, and leave the source_tested=`-(u)f`
SELECT *
FROM PyFunceble.pyfunceble_status
where file_id LIKE '${source_tested}'
AND test_completed = '1'; |
Beta Was this translation helpful? Give feedback.
-
|
I'll bet this comment #343 (comment) would like to see the PyFunceble WHOIS DB to be published #259 |
Beta Was this translation helpful? Give feedback.
-
Intro
Brainstorm, to speedup the checks and maybe a few new thoughts...
If i take a peak in the SQL, there is a nice table called
pyfuncle_whoisand it's equlant in json iswhois_db.jsonTest flow
Now one of my thought on the test-flow is:
Why are domains not default verified against this data to see if we actually need to re-check the ACTIVE state? If expires is in the future, skip further testing. This should take milliseconds.
Let take a real ex. from a project I'm helping in it's start: https://travis-ci.com/Import-External-Sources/pornhosts/builds
Now add óne new domain to the https://github.com/Import-External-Sources/pornhosts/blob/master/source/hosts.txt and the CI/CD will be running 5 or 6 cycles ~ 1½ hour for testing(adding) 1 new submission. but testing the domain manually takes
0m0,572sWHOIS domain.tld
So what if a user would like actually do a specified dns or http testing? for
inactivethere's already the-dbr $days(Which i think should be-dbee for expired there should be something like:-dba : Recheck ACTIVE requires using one of ==>
-RcodeA(AAA)? : Validate A and AAAA
-httpv : Validate http(s)?
The thought about this is flexibility for the individual users setup and needs.
But for my need, I actually don't care whether a blocked domain is responding or not, because it can be responding 0,001 msec after my test
SQL
The next idea is based on how the DB is loaded on a CI startup... (note haven't checked the code on this)
But while running in a SQL environment the easiest is to handle a SQL file like
pyfunceble.sqlfir export/import on external DB's straight with the mysql-client(dump).CI/CD
This should be loaded before test to satisfy above suggestion, and dump after testing to do first statement in next run
Finishing note
These thought jump into my head on the jar 🚽 so forgive me if there are some thought on this in other issues, as I jumped to write this while it is fresh in mind.
Share
Please other users of PyFunceble, please share you comment on this, they are left as brainstorm/thoughts/idea
PS:
Using the option command
travisshould be renamed tociorcdas PyFunceble actually functions pretty well on other cd/cd than travis 😲Beta Was this translation helpful? Give feedback.
All reactions